How to extract data from web service and load into database

Overview

This Flow extracts data from a web service and loads it into a relational database.

Process

In Connections

Access Connections here.

Step 1. Create a source HTTP Connection.

Step 2. Create a destination Connection for the relational database and test it.

Step 3. Create a Format for the payload returned by the web service, most likely JSON or XML.

In Explorer

Access Explorer here.

Step 4. Link Connection and Format and test the response for the web service in the EXPLORER. 

Step 5. It is most likely that the response is a nested JSON or XML so learn how to flatten the nested dataset.

In Flows

Access Flows here.

Step 6. Create a new Web Service to database Flow.

Step 7. Add a new transformation, select FROM Connection, Format, and endpoint and TO Connection and table.

Step 8. Click Mappings and configure the flattening for the nested JSON or XML. Most likely it is going to be a Source SQL.

Step 9. Test the transformation. Make sure it returns the flat data set.

Step 10. Add per-field Mapping if needed.

Step 11. Configure MERGE (UPSERT) if needed.

Step 12. Save the Flow and execute it manually.

In Schedules

Step 13. Schedule the Flow to be executed periodically.