Getting started working with web services

In Etlworks Integrator, it is possible to extract data from and push data into practicably any REST-based or SOAP-based web service. The integrator also includes a growing number of built-in connectors to the most commonly used APIs.

Connecting to web services

• Read how to create a connection to any web service.
• Read how to configure authentication.

Built-in API connectors

• Google Analytics 
• Google Sheets 
• Google AdWords
• Google BigQuery
• Apache Hive
• Amazon MWS 
• Social networks 
• Salesforce
• Elasticsearch
• Amazon Athena

Other well-known APIs

• Connecting to well-known APIs if there is no built-in connector

Formats

Web services use different data exchange formats, which must be described in Integrator:

• CSV 
• Fixed Length Text 
• JSON 
• JSON dataset 
• XML 
• XML dataset 
• Avro
• Parquet
• HL7 2.x 
• HL7 FHIR 

Parametrization and Dynamic URLs

Parameterization is a way to dynamically configure connections and transformations based on input parameters. The input parameters can be global variables or flow variables.

• Read about parametrization of the connection.
• Read about dynamic URLs.

Testing and exploring API endpoints

Use the Explorer to test the connection to a web service and explore API endpoints.

Step 1. Create an HTTP connection to the web service.

Step 2. Create a format for the response. The most commonly used formats are JSON, XML and CSV.

Step 3. Open Explorer, select the connection created in Step 1 and link it to the format created in Step 2. Read more about how to link the format to a connection in Explorer.

Step 4. Explore the metadata (endpoints and fields), view data in a grid, and a raw format, query data and discover dependencies using SQL.

Extracting, transforming and loading data in web services

• A web service is a source
• A web service is a destination

Using copy file operation to copy the response to the file and send the payload to API

• Copy response from the HTTP request to a file 
• Send a file as a payload for an HTTP endpoint 

Sending payload to web service

Read how to send a payload to an HTTP endpoint.

Tips and tricks when working with web services

• Configuring auto-retry when working with web services 
• Working with paginated APIs 
• Creating a payload for HTTP call using a template with tokens
• How to use memory connection to parse the response from the web service
• Using templates to define fields 
• Handling field names with special characters 
• Handling columns with a blank name 
• Working with duplicated column names 
• Calculating field value 
• Working with very large files