The HTTP API connector in Etlworks supports a wide range of authentication methods used across modern APIs, including basic auth, token-based schemes, AWS Signature, and user-defined headers. This guide explains each supported method and how to configure it.

Supported Authentication Types

• Basic Authentication

• Basic Preemptive Authentication

• API Key Authentication

• Token-based (OAuth2) Authentication

• Browser-based OAuth2 (Used in Pre-configured Connectors)

• OAuth1 (One-step OAuth)

• AWS Signature Authentication

• Header-based Authentication

• SOAP-style Authentication

• Custom Authentication using Preprocessor

Basic Authentication

• Select basic in the Authentication field.

• Enter the username and password.

Credentials will be passed in the Authorization header using standard base64 encoding.

Basic Preemptive Authentication

• Select basic preemptive in the Authentication field.

• Enter the username and password.

Preemptive mode sends the credentials without waiting for a 401 challenge.

API Key Authentication

• Select none in the Authentication field.

• Enter the API key in the Password field.

• Use {password} token in the URL or headers to insert the key.

Example:

https://api.example.com/data?apiKey={password}

OAuth2 (Two-step Token-based Authentication)

OAuth2 is a widely used standard for token-based authentication. It allows secure access to APIs without passing usernames and passwords with every request.

How OAuth2 Works

Step 1: Request an Access Token

Etlworks sends a request to the Authentication URL using the provided User (client ID) and Password (client secret). The token endpoint typically returns a JSON object that includes the access token.

Step 2: Use the Token

The token is extracted from the response and automatically added to the Authorization header (or a custom header) in all subsequent API calls.

Configuration Options

• Authentication: Select token or oauth2.

• Authentication URL: The token endpoint (e.g., https://example.com/oauth/token).

• User: Client ID.

• Password: Client Secret.

• HTTP Method for Token Authentication: Usually POST.

• Access Token Attribute: JSON node containing the token (e.g., access_token or result.accessToken).

• Access Token Prefix: Prefix added to the token in the header (e.g., Bearer). Use none to omit the prefix.

• Authentication Request Payload: Optional JSON body for the token request. Use {user}, {password}, or {refresh_token} tokens.

• Authentication Request Headers: Optional custom headers.

• Authentication Header Name: Header to receive the token (default: Authorization).

• Authentication Request Content Type: Usually application/json or application/x-www-form-urlencoded.

Browser-based OAuth2 (Used in Pre-configured Connectors)

Browser-based OAuth2 uses the authorization code flow, which requires a user to interactively log in and grant permission.

For browser-based OAuth2, use a Pre-configured API Connectors instead of configuring OAuth2 manually.

OAuth1 (One-step OAuth)

• Select oauth1 in the Authentication field.

• Fill in:

  • User → Consumer Key

  • Password → Consumer Secret

  • Access Token

  • Access Secret

AWS Signature Authentication (AWS v4)

• Select aws in the Authentication field.

• Enter Access Key in the User field.

• Enter Secret Key in the Password field.

• Optionally set:

  • AWS Region (default: us-east-1)

  • Service Name (e.g., s3, iam). If not set, Etlworks will attempt to extract it from the URL.

Header-based Authentication

Use this method when the API expects a non-standard authorization header.

• Select header in the Authentication field.

• Set:

  • Authentication Header Name: Custom header name (default: Authorization).

  • Content of the Authorization header: Token or header content (use {user}, {password} if needed).

• Use single quotes for string values: 'Bearer {password}' instead of double quotes.

SOAP-style Authentication

SOAP APIs often require signed payloads with tokens embedded in the XML.

The Preprocessor can be used to dynamically compute and set global variables, which are referenced in the SOAP envelope using {tokens}.

Step 1. Set a SOAP envelope with tokens in Payload

Here is an example:

<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" 
xmlns:ns1="http://www.vendor.com/soapendpoint/" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <SOAP-ENV:Header>
    <ns1:AuthenticationHeader>
      <vendorUserId>{requestUserId}</mktowsUserId>
      <vendorSignature>{requestSignature}</requestSignature>
      <vendorTimestamp>{requestTimestamp}</requestTimestamp>
    </ns1:AuthenticationHeader>
  </SOAP-ENV:Header>
  <SOAP-ENV:Body>
    <ns1:paramsGetMultipleCars>
      <leadSelector xsi:type="ns1:KeySelector">
        <keyType>MAKE</keyType>
        <keyValues>
          <stringItem>Ford</stringItem>
          <stringItem>Acura</stringItem>
        </keyValues>
      </leadSelector>
      <batchSize>100</batchSize>
    </ns1:paramsGetMultipleCars>
  </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

Step 2. Use Preprocessor to dynamically set values for tokens

Here is an example:

var javaImports = new JavaImporter(java.text, java.util, javax.crypto, 
javax.crypto.spec, org.apache.commons.codec.binary, com.toolsverse.config);

with (javaImports) {
   var props = SystemConfig.instance().getProperties();
   var vendorUserId = "user";
   var vendorSecretKey = "password";

   var df = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ssZ");
   var text = df.format(new java.util.Date());
   var requestTimestamp = text.substring(0, 22) + ":" + text.substring(22);

   var encryptString = requestTimestamp + vendorUserId;
   var secretKey = new SecretKeySpec(vendorSecretKey.getBytes(), "HmacSHA1");
   var mac = Mac.getInstance("HmacSHA1");
   mac.init(secretKey);
   var rawHmac = mac.doFinal(encryptString.getBytes());
   var signature = new java.lang.String(org.apache.commons.codec.binary.Hex.encodeHex(rawHmac));

   props.put("requestSignature", signature);
   props.put("requestTimestamp", requestTimestamp);
   props.put("requestUserId", vendorUserId);
}

Custom Authentication Using Preprocessor

Use the Preprocessor to dynamically generate tokens or modify connection parameters before the request is sent. 

The {tokens} can be referenced in URL and headers when configuring connection.