In Etlworks Integrator, it is possible to extract data from and push data into practicably any REST-based or SOAP-based web service. The integrator also includes a growing number of built-in connectors to the most commonly used APIs.
Connecting to web services
Built-in API connectors
- Google Analytics
- Google Sheets
- Google AdWords
- Google BigQuery
- Apache Hive
- Amazon MWS
- Social networks
- Amazon Athena
Other well-known APIs
Web services use different data exchange formats, which must be described in Integrator:
Parametrization and Dynamic URLs
Parameterization is a way to dynamically configure connections and transformations based on input parameters. The input parameters can be global variables or flow variables.
Testing and exploring API endpoints
Use the Explorer to test the connection to a web service and explore API endpoints.
Step 1. Create an HTTP connection to the web service.
Step 2. Create a format for the response. The most commonly used formats are JSON, XML and CSV.
Step 3. Open Explorer, select the connection created in Step 1 and link it to the format created in Step 2. Read more about how to link the format to a connection in Explorer.
Extracting, transforming and loading data in web services
Using copy file operation to copy the response to the file and send the payload to API
Sending payload to web service
Read how to send a payload to an HTTP endpoint.
Tips and tricks when working with web services
- Configuring auto-retry when working with web services
- Working with paginated APIs
- Creating a payload for HTTP call using a template with tokens
- How to use memory connection to parse the response from the web service
- Using templates to define fields
- Handling field names with special characters
- Handling columns with a blank name
- Working with duplicated column names
- Calculating field value
- Working with very large files