The HTTP API Connector allows you to integrate with any HTTP-based API using GET, POST, PUT, PATCH, or DELETE requests. You can use it to read data from APIs or send data to APIs as part of an ETL flow. It supports multiple authentication methods, various data formats, pagination, scripting, and advanced response handling.

Types of API Connectors in Etlworks

Etlworks supports four types of API connectors:

Type	Description	Best for
Generic HTTP Connector	Fully customizable; supports any HTTP API, including REST, SOAP, OData, GraphQL and custom API	Any API integration requiring fine-grained control
Pre-configured Connectors	Based on the generic HTTP connector with built-in OAuth2 and presets. See Using Pre-Configured API Connectors for more details.	Popular APIs like Google, Microsoft, Salesforce, etc.
Dedicated Connectors for Connected Apps	Premium connectors that require annual subscription fee and expose API endpoints as virtual tables and views. To learn more, see connectors for Connected Apps.	Complex APIs; SQL-like querying
Connector for well-known APis	Dedicated connectors for a small group of popular APIs, including Google Analytics, Google Sheets, Google Ads, Facebook, and Twitter. These connectors differ from other connectors in that they are not generic or database-based, they use their own specialized connection type. Read more: ETL data from well-known APIs Get Started with Google Analytics Get Started with Google Sheets	Working with Google Analytics, Google Sheets, Google Ads, Facebook, and Twitter.

 

When to Use

Use the HTTP API Connector when you need to:

• Connect to HTTP-based APIs, including REST, SOAP, OData, GraphQL or custom APIs

• Read or write data over HTTP(S)

• Work with APIs that require authentication (API key, OAuth2, etc.)

• Use advanced features like retry, scripting, or response transformation

For usage examples and API integration patterns, see Get started with APIs.

Pre-configured API Connectors

Etlworks provides pre-configured HTTP connectors for commonly used APIs. These connectors simplify setup by pre-filling values such as base URL, authentication type (browser-based OAuth2), headers, and pagination parameters.

You can use them exactly like the generic HTTP connector, as a source or destination in any flow.

List of Available Pre-configured Connectors:

• Etlworks APIs Google APIs (Drive, Gmail, Sheets, Calendar, etc.)
• Microsoft APIs (Graph, Outlook, OneDrive, SharePoint)
• Salesforce
• HubSpot
• Quickbooks Online ZOHO (CRM, Creator, Inventory, Project)
• Jira
• Jira Service Management
• Confluence
• Amazon Marketplace

See Using Pre-Configured API Connectors for more details.

Creating a Connection

1. Go to Connections > Add connection.

2. Select HTTP from the list of connectors.

3. Configure Parameters:

Common Parameters

Note: The URL, as well as header keys and values, can include {tokens}. Tokens reference global variables and are resolved at runtime, allowing dynamic and environment-specific configurations.

• URL (required): Full connection URL. This is the base endpoint for all requests (e.g., https://api.example.com).

• Method: Default HTTP method to use (GET, POST, PUT, PATCH, or DELETE). 

• Content Type Header: Sets the Content-Type header (e.g., application/json, application/xml, or multipart/form-data).

• Headers: One or more custom HTTP headers to include in the request. Use key-value pairs (e.g., Authorization, X-API-Key).

• Payload: HTTP request body. Typically used for POST, PUT, or PATCH. Can contain raw JSON, XML, or other formats.

• File Name: Optional. Used to send a file with the request when using multipart or binary uploads.

• Maximum Asynchronous HTTP Calls: Number of concurrent HTTP requests. Set to 0 to run requests sequentially.

• Stream Payload to POST and PUT Endpoints: When enabled, streams the request body instead of buffering it in memory. Recommended for large payloads.

• Timeout: Optional request timeout in milliseconds. Defines how long to wait for a response before failing the request. The default HTTP timeout in Etlworks is 1 hour. 

Authentication

The HTTP connector supports multiple authentication methods, including:

• No authentication (anonymous)
• Basic authentication (username and password)
• Bearer tokens (static or dynamic)
• API keys (in headers or query parameters)
• OAuth2 (manual and browser-based)
• OAuth1
• AWS API
• Custom headers

To configure authentication:

1. Select the desired method in the Authentication parameters section of the connection settings.

2. Fill in the required credentials or tokens.

3. (Optional) Use variables or scripting to generate dynamic tokens.

For detailed guidance, see Authentication Methods for HTTP API Connector.

SSL Configuration

Configure these options if the API requires HTTPS and/or client certificate authentication:

• Trust Self-Signed and Expired Certificates: When enabled, the connector will trust SSL certificates that are self-signed or expired. Not recommended for production environments.

• Client SSL Certificate: Upload the client certificate file (e.g., .p12, .pem, or .crt) required for mutual TLS authentication.

• Client SSL Certificate Type: Select the format of the uploaded certificate (e.g., PKCS12, PEM, JKS).

• Client SSL Certificate Password: If the uploaded certificate is password-protected, enter the password here.

• Allowed TLS Versions: Optionally restrict which TLS versions are allowed (e.g., TLSv1.2, TLSv1.3). Leave empty to allow defaults.

Output and Formatting

Use these settings to define how the API response is interpreted and formatted.

• Data JSON Path: Optional path to the array or object in the response that contains the actual data. Required for most APIs that return nested responses or include metadata alongside the data.

  Use slash notation (e.g., /data/items) instead of dot notation.

  Examples:

  • /data/items for a response like:

{
    "data": {
       "items": [ {...}, {...} ]
    }
}

• Keep empty if the response itself is a plain array:

[ {...}, {...} ]

Note: This setting is often required for pagination to work. If your API response includes both data and a pagination token (e.g., nextToken), you must set Data JSON Path to isolate the array node that holds the data.

• Output as CSV: Converts the response into CSV format.

• Denormalize Nested Fields: Flattens nested structures into individual fields. Useful when exporting to a relational format.

• CSV Separator: Character used to separate fields in CSV output. Default is a comma (,), but can be changed to semicolon (;), tab (\\t), or another character.

Read about using output and formatting options when working with nested response.

Pagination

Many APIs return results in pages. The HTTP API Connector supports built-in pagination to automatically request additional pages and merge the results into a single dataset.

You can configure pagination using one of the following modes:

• Offset-based: Uses parameters like offset and limit.

• Page-based: Uses parameters like page and pageSize.

• Cursor/token-based: Uses a next token or URL returned in the response.

• Time-based: Uses a timestamp or date range to fetch records incrementally.

Important: If your API returns paginated data alongside metadata (such as a token or page count), make sure you set the Data JSON Path in the Output and Formatting section to correctly isolate the array of data items. This is required for pagination to work correctly.

For detailed setup instructions, examples, and tips for each mode, see Work with Paginated APIs using the HTTP Connector.

Handling Response

These settings control how HTTP responses are processed and logged.

• Success HTTP Codes: List of HTTP status codes considered successful (e.g., 200,201,202).

  Default values are provided, so you typically don’t need to configure this unless working with non-standard success codes.

• Exception HTTP Codes: List of HTTP status codes that should be treated as errors (e.g., 400,404,500).

  Defaults are also provided. Override only if your API uses custom error handling.

• Response File Name: Optional. If specified, the connector will write the full HTTP response to the given file path. You can use the {app.data} token to point to the root folder for user data, along with any other {tokens} representing global variables, to dynamically construct the file name and location.

• Save Response Only on Error: Saves the response body only when an error occurs.

• Log Response on Exception: Logs the response body into the flow log when the request fails. Helpful for debugging error payloads.

• Log URL to Debug Log: Logs the request URL to the flow log. Useful for troubleshooting requests with dynamic components.

• Save HTTP Code into Global Variable: Stores the HTTP status code in a global variable http_processor_last_http_code that can be used later in the flow.  

• Save Response Headers into Global Variable: Saves all HTTP response headers into a global variable http_processor_last_respons_headers for downstream use.

Read more about capturing HTTP status code and Response Headers.

Encoding

Use these settings to define how the request and response data should be encoded:

• Request Body Encoding: Sets the character encoding for the HTTP request body.

  Default is UTF-8. Change this only if the API explicitly requires a different encoding (e.g., ISO-8859-1).

• Encoding: Defines the expected character encoding for the response.

  Only configure this if the API returns a non-UTF-8 encoded response. If left blank, the connector will try to auto-detect the encoding based on the response headers.

Retries

Configure how the connector should retry failed requests:

• Number of Retries: Total number of retry attempts after a failure. Set to 0 or leave blank to disable retries.

• Delay Between Retries: Initial delay (in milliseconds) before the first retry. The connector uses exponential backoff — the delay doubles with each attempt until it reaches 1 minute. After that, all subsequent retries use a constant 1-minute delay.

Use this feature to handle temporary network issues, rate limits, or intermittent API errors.

Non-Retriable Errors

The following HTTP status codes are considered non-retriable and will immediately stop further attempts:

400, 404, 405, 406, 410, 411, 413, 414, 415, 422, 426, 428

Throttling

Use these settings to limit the rate of HTTP requests and avoid being rate-limited by the API.

• Wait After N Calls (ms): Specifies how long (in milliseconds) to wait after a defined number of calls.

  For example, if set to 1000 and Number of Calls is 5, the connector will pause for 1 second after every 5 requests.

• Number of Calls: The number of calls to make before pausing. Default is 1, meaning the connector will wait after each request.

• Endpoint Used for Throttling: Optional. If set, throttling will apply independently to this endpoint. This is useful when calling multiple endpoints with different rate limits.

Preprocessor

Use the Preprocessor field to programmatically modify the behavior of the HTTP connection before the request is sent. This script runs once per request and is written in JavaScript (Nashorn).

Common Use Cases for Preprocessor

Set Global Variables Used in URL, Headers, or Parameters

The most common use case is to dynamically compute and set global variables, which are referenced in the connection using {tokens}.

For example, you may need to:

• Generate a signature

• Build a dynamic path

• Set a timestamp or access token

var props = SystemConfig.instance().getProperties();
props.put("auth_token", "Bearer " + computeToken());
props.put("dynamic_path", "/files/" + Date.now());

In this case, the HTTP connection would include {auth_token} or {dynamic_path} in headers or the URL.

Note: Returning a value is not required in this scenario.

Override Connection Settings (Advanced)

Less commonly, you may need to fully override parts of the connection itself — such as the URL, username, password, or connection parameters.

To do this, your script must return an instance of com.toolsverse.etl.common.Alias.

var alias = new com.toolsverse.etl.common.Alias();
alias.setUrl("https://api.example.com/" + dynamicPath);
alias.setUserId("new-user");
alias.setPassword("new-password");
value = alias;

Variables Available in Preprocessor Code

Variable	Description
url	The request URL, with all {tokens} already resolved.
user	The username, resolved.
password	The password, resolved.
properties	java.util.Properties object containing all connection properties.
isAuthRequest	Boolean. true if this is an authentication request.
payload	The HTTP request body, if applicable.

Example: Digital Signature for CyberSource

This example sets global variables used in headers and URL:

var javaImports = new JavaImporter(
com.toolsverse.config,
javax.crypto.Mac,
javax.crypto.spec.SecretKeySpec,
java.util.Base64
);

with (javaImports) {
   var props = SystemConfig.instance().getProperties();
   var merchantId = properties.getProperty("header_v-c-merchant-id");
   var host = properties.getProperty("header_host");

   var date = new Date(Date.now()).toUTCString();
   var resource = "/reporting/v3/report-downloads?organizationId=" + merchantId + "&reportDate=2024-01-01";

   var signatureString = "host: " + host + "\\ndate: " + date +
      "\\n(request-target): get " + resource +
      "\\nv-c-merchant-id: " + merchantId;

   var secretKey = new SecretKeySpec(Base64.getDecoder().decode(password), "HmacSHA256");
   var mac = Mac.getInstance("HmacSHA256");
   mac.init(secretKey);
   mac.update(signatureString.getBytes());
   var signature = Base64.getEncoder().encodeToString(mac.doFinal());

   props.put("cs_auth_date", date);
   props.put("cs_auth_signature", 'keyid="' + user +
     '", algorithm="HmacSHA256", headers="host date (request-target) v-c-merchant-id", signature="' + signature + '"');
   props.put("cs_url", "https://" + host + resource);
}

Postprocessor

Common Use Case for Postprocessor

Use the Postprocessor to programmatically modify the HTTP response content before it is parsed or processed by the flow.

This is useful when:

• The response is not in a standard format

• You need to extract or transform content

• You want to filter or clean up the response before mapping

The script is written in JavaScript (Nashorn) and runs after the HTTP response is received but before any mapping or transformation is applied.

Variables Available in Postprocessor Code

Variable	Description
url	The request URL, with all {tokens} already resolved.
user	The connection’s username, resolved.
password	The connection’s password, resolved.
properties	A java.util.Properties object with all resolved connection properties.
isAuthRequest	Boolean. true if the current request is for authentication.
response	The original response content (string or object depending on format).

Return Values

Your script should assign a value to the value variable to define the post-processed result:

• value = response – return the original unmodified response.

• value = null – treat the response as empty.

• value = <new content> – return a modified version of the response.

Example: Clean up JSON response

// Remove a wrapper from the response
var obj = JSON.parse(response);
value = JSON.stringify(obj.data.items);

PGP Encryption

Use this section to automatically decrypt PGP-encrypted HTTP responses. This is commonly required when working with APIs that return sensitive data as encrypted files or text.

• PGP Private Key File: Upload your PGP private key file used to decrypt the response. Supported formats include .asc, .gpg, and others.

• PGP Password: Password (if any) for the private key file. Leave blank if the key is not password-protected.

Once configured, the connector will automatically detect and decrypt PGP-encrypted responses before further processing (e.g., mapping or parsing).

Expected Compression

The Expected Compression setting allows you to specify the compression format expected when reading individual files from a connection. Supported options include No Compression, Zip, and GZip.

If set to Zip or GZip, Etlworks will automatically decompress each resource as it’s read. This setting assumes that each compressed file contains a single resource; it does not support archives with multiple files. Use this setting with caution, as the system will always attempt to decompress the resource based on the selected format, regardless of its file extension or actual content.

Multi-part upload

Use multipart uploads to send files or a combination of different data types (e.g., file + JSON) in a single HTTP request. These requests are commonly used for form-based file uploads and APIs that require multipart/form-data encoding.

How to Configure the Connector for Multipart Upload

Step 1. Set Content Type Header to multipart/form-data.

If the endpoint (e.g., PHP-based backend) requires a custom boundary, leave the field blank or set it manually with the full value including boundary.

Step 2. Set File Name property

Use the File Name field to define one or more form fields as a key=value list, separated by &.

Use this format:

• file1={app.data}/report.pdf&description=Quarterly Report

• Each key represents a field in the multipart request.

• If the value points to a file, use an absolute path or a token like {app.data} to locate the file.

• Files must exist at the specified path and be accessible by Etlworks at runtime.

Note: You can use any accessible file path, not just {app.data}.