When to use this connector
In Integrator, you can create a connector to practically any HTTP-based web service. It is typically assumed that the web service is REST-based, but you can also connect to a SOAP or proprietary web service.
HTTP connector supports the GET, POST, PUT, PATCH, HEAD and DELETE methods.
Read how to work with web services in Etlworks.
Creating a connection
Start creating a connector to the web service by selecting the HTTP
Connection Type from the gallery:
Connection parameters
- URL - the endpoint URL. You can use an
{access_token}
as part of the URL. Example: -
https://host/rest/v1/lead/36938.json?access_token={access_token}
- Method - the HTTP method which will be used for that particular request. If
Default
is selected, the following methods will be used:GET
for getting data from the web service,POST
for pushing data to the web service, andDELETE
for deleting data. - Content Type Header - HTTP request content-type header.
- Headers - optional HTTP headers.
- Payload - actual payload for the POST or PUT request. Typically, the payload is automatically provided when flow transforms it from the source to the destination, but you can also hard-code the payload. You can use
{tokens}
as a part of the payload. The tokens will be automatically substituted during runtime on global variable values. - File Name - an optional file name, which contains a payload for the POST or PUT request. Typically, the payload is automatically provided when flow transforms the source to the destination, but you can also specify a name for the file which contains the payload.
- Maximum Asynchronous HTTP Calls - the maximum number of asynchronous HTTP calls for this connection. By default, HTTP calls are performed synchronously, but you can enable async calls by setting this property to a value greater than zero. Only the POST, PUT, and DELETE methods are supported. The global maximum number of asynchronous calls is
100
. - Stream payload to POST and PUT endpoints - if this option is selected and the
POST
orPUT
HTTP request contains a payload, the payload will be streamed to the endpoint. Streaming reduces memory footprint when submitting data and overall much faster, so it is enabled by default. In some rare cases streaming can cause an HTTP connection to "hang", hence the option to disable streaming. - Timeout - the request timeout in milliseconds. The default value is 1 hour. if the request is taking longer than configured or the default value, it will be automatically aborted.
- Below is the beginning of a section about the parameters used for authentication. If authentication is not required you can skip this section.
- Authentication - select
basic
orbasic preemptive
for username/password authentication,token
oroauth2
for OAUTH2 two-step authentication,oauth1
for one-step OAUTH1 authentication,aws
for AWS Signature authentication,header
for authentication when the Authorization header parameter is directly provided, andnone
when there is no authentication at all. Learn more about configuring authentication for HTTP connection. - User or AccessKey - the username or access key.
- Password or SecretKey - the password or secret key.
- Authentication URL - the endpoint URL for the authentication request. The endpoint must return an access token. Use it when the
token
oroauth2
Authentication type is selected. You can use{refresh_token}
as part of the URL. Example: -
https://host/oauth2/token?grant_type=refresh_token&refresh_token={refresh_token}
- HTTP Method for Token and OAuth2 Authentication - the HTTP method is used for obtaining an access token. Use it when the
token
oroauth2
Authentication type is selected. - Access Token Attribute - when the authentication endpoint is called, it typically returns an access token as a part of the response in the JSON format. Use this field to specify the name of the attribute in JSON, which contains an access token. For example: access_token. Use it when the
token
oroauth2
Authentication type is selected. - Access Token Prefix - prefix used together with the authentication token header parameter. The default is
Bearer
. - Authentication Request Payload - use this field to define a payload for the
POST
orPUT
authentication request. Use it when thetoken
oroauth2
Authentication type is selected and the HTTP Method for Token Authentication isPOST
. The following tokens are accessible as a part of the payload:{refresh_token}
,{user}
,{password}
. Example: -
{"login":"{user}","password":"{password}"}
- Authentication Request Content Type - the default is
application/json
. Use it when thetoken
oroauth2
Authentication type is selected and the HTTP Method for Token Authentication isPOST
. - Authentication Request Headers - HTTP headers for the authentication request.
- Refresh Token - an optional refresh token that can be used to obtain an access token. See the usage example above. Use it when the
token
oroauth2
Authentication type is selected. You can use tokens such as{user}
,{password}
and others. Use single quotes:'value'
instead of double quotes:"value"
(all single quotes will be substituted on double quotes). - AWS Region - an optional AWS Region used together with the
aws
authentication. The default value isus-east-1
. - Service Name - an optional AWS Service Name used together with the
aws
authentication. - Access Token - an optional access token used together with the
oauth1
authentication. - Access Secret - an optional access secret used together with the
oauth1
authentication. - Authentication Header Name - the name of the authentication header parameter. The default is
Authorization
. It used only when thetoken
,oauth2
orheader
authentication type is selected. - Content of the Authorization header - this parameter is required when
the header
authentication type is selected. - this is the end of the section on parameters used for authentication.
- Success Codes - a comma-separated list of HTTP codes which, when returned, are considered to be "success" codes. For example: 200,201. If this field is blank, the default "success" codes will be used for each particular request type (GET, POST, PUT, DELETE, HEADER).
- Exceptions from Error Codes - a comma-separated list of the HTTP codes which, when returned, are considered to be exceptions from the standard error codes. Use it when you still want to parse a response from the service even after it has already returned one of the standard error codes (4xxx, 5xx, etc.).
- Response File Name - the name of the file to dump the response from the request. If empty, the system will not create a response file.
- Save Response only on Error - if this option is selected, the response file will be created only if the HTTP request is executed with error.
- Add Response to Exception Log - if this option is selected, and the HTTP request is executed with an exception, the response content will be added to the exception log.
- Add URL to Debug Log - if this option is selected, the actual URL of the request will be added to the debug log.
- Read how to debug HTTP requests.
- Save HTTP Code into Global Variable - If this option is enabled, the HTTP code returned by the endpoint will be saved to the global variable
http_processor_last_http_code
. Read how to get access to this variable from JavaScript. - Save Response Headers into Global Variable - If this option is enabled, the headers in HTTP response returned by the endpoint will be saved to the global variable
http_processor_last_response_headers
, which can be accessed usingcom.toolsverse.config.SystemConfig.instance().getContextProperties().get('http_processor_last_response_headers')
. - Request Body Encoding - optional character encoding for the payload (request). Use it when you wish to re-encode the payload (for example, to
UT8
).No encoding
means there will be no additional encoding. - Encoding - optional character encoding for the response. Use it when you wish to re-encode the response (for example, to
UT8
).No encoding
means there will be no additional encoding. - Number of Retries - number of retries if the HTTP call was not successful. The maximum allowed number of retries is 10. The default is 0.
- Delay between retries - delay in milliseconds between retries if the HTTP call was not successful. The maximum allowed delay is 5 minutes (300000 milliseconds). The default is 0.
Configuring authentication for HTTP connection
HTTP connector supports the following types of authentication.
- Basic Authentication
- Basic Preemptive Authentication
- Basic Authentication with additional parameters
- API Key Authentication
- Two-step token-based authentication (OAuth2)
- One-step oauth authentication (OAuth1)
- AWS authentication (AWS Signature)
- SOAP-style Authentication
Basic Authentication
- When configuring an HTTP connection, select
basic
as the value for the Authentication field. - Enter the username and password into the User and Password fields.
Basic Preemptive Authentication
- When configuring an HTTP connection, select
basic preemptive
as the value for the Authentication field. - Enter the username and password into the User and Password fields.
Basic Authentication with additional parameters
In some cases, additional parameters (besides username and password) are required to connect to a web service.
- When configuring an HTTP connection, select
none
as the value for the Authentication field. - Make sure that
POST
orPUT
is selected as the value for the Method field. - Enter additional parameters in the JSON format into the Payload field. Use the tokens
{user}
and{password}
as substitutes for the username and password. For example:{"username":"{user}","password":"{password}","labName":"Demo"}
.
API Key Authentication
- When configuring an HTTP connection, select
none
as the value for the Authentication field. - When configuring the URL, use the token
{password}
as a substitute for the actual API key. For example:https://api.com/v1/metrics/timeseries?apiKey={password}
. - Enter an API key into the Password field.
Two-step token-based authentication (OAuth2)
- Select
token
oroauth2
as the value for the Authentication field. - Enter the API endpoint used to receive an access token in the Authentication URL field. For example:
https://777-XWC-041.mktorest.com/identity/oauth/token?grant_type=client_credentials&client_id={user}&client_secret={password}
- Select a value for the field HTTP Method for Token Authentication. Typically, it is
POST
. - Optionally define Authentication Request Headers.
- Enter a value for the optional Refresh Token field. A refresh token can be used to obtain an access token.
- Enter a value for the Access Token Attribute field. When the authentication endpoint is called, it typically returns an access token as part of a response in the JSON format. Use this field to specify the name of the attribute in JSON which contains an access token. For example:
access_token
. If the response for the authentication (token) endpoint is a nested JSON and the access token is a nested JSON node, for example,{"success":true, "result":{"accessToken":"AA-BB-CC-DD", "expires":1234567}}
use theowner.node
notation to get the value of the token. Example:result.accessToken
. - Enter a value for the Access Token Prefix field. A prefix is used for the authentication token header parameter. The default is
Bearer
. If "none" (actual wordnone
) is entered the system assumes that there is no prefix. - Enter an authentication payload into the Authentication Request Payload field. The following tokens are accessible as part of the payload:
{refresh_token}, {user}, {password}
. For example: - {"login":"{user}","password":"{password}"}
- Select a value for the Authentication Request Content Type field. The default is
application/json
. - Optionally enter the value in the Authentication Header Name field. The default value is
Authorization
. - Enter client id into the User field.
- Enter client credentials into the Password field.
One-step oauth authentication (OAuth1)
- Select
oauth1
as the value for the Authentication field. - Enter consumer key into the User field.
- Enter consumer secret into the Password field.
- Enter access token into the Access Token field.
- Enter access secret into the Access Secret field.
AWS authentication (AWS Signature)
- Select
aws
as the value for the Authentication field. - Enter the AccessKey and AccessSecret into the User and Password fields.
- Enter optional AWS Region into the AWS region field. The default region is
us-east-1
. - Enter optional Service Name into the Service Name field. If Service Name is not entered it will be extracted from the URL. For example, for the URL
https://iam.amazonaws.com/?Action=GetUser
the service name isiam
.
Authentication when the authorization header parameter is user-defined
- Select
header
as the value for the Authentication field. - Change the value for the optional Authentication Header Name field. The default value is
Authorization
. - Enter the required authorization header parameter into the Content of the Authorization header field. You can use tokens such as
{user}
,{password}
and others. Use single quotes:'value'
instead of double quotes:"value"
(always use single quotes in place of double quotes). - Enter an optional username into the User field.
- Enter an optional password into the Password field.
SOAP-style Authentication
When working with a SOAP web service, you are typically dealing with a payload that must be electronically signed in some fashion. There is no single standard way of doing this, so we will merely explain a basic approach.
Step 1. Let's assume that the payload (SOAP envelope) looks something like this (below):
<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:ns1="http://www.vendor.com/soapendpoint/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<SOAP-ENV:Header>
<ns1:AuthenticationHeader>
<vendorUserId>{requestUserId}</mktowsUserId>
<vendorSignature>{requestSignature}</requestSignature>
<vendorTimestamp>{requestTimestamp}</requestTimestamp>
</ns1:AuthenticationHeader>
</SOAP-ENV:Header>
<SOAP-ENV:Body>
<ns1:paramsGetMultipleCars>
<leadSelector xsi:type="ns1:KeySelector">
<keyType>MAKE</keyType>
<keyValues>
<stringItem>Ford</stringItem>
<stringItem>Acura</stringItem>
</keyValues>
</leadSelector>
<batchSize>100</batchSize>
</ns1:paramsGetMultipleCars>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>
Step 2. Notice the tokens {requestUserId},{requestSignature} and {requestTimestamp}
. They are part of the electronic signature.
Step 3. We will be using JavaScript to set the values of {requestUserId},{requestSignature} and {requestTimestamp}
.
Step 3.1. Create a flow which works with a SOAP web service.
Step 3.2. Create a JavaScript flow, which will be used to set tokens.
Below is an example of the JavaScript code:
importPackage(java.text);
importPackage(java.util);
importPackage(javax.crypto);
importPackage(javax.crypto.spec);
importPackage(org.apache.commons.codec.binary);
importPackage(com.toolsverse.config);
var props = SystemConfig.instance().getProperties();
var vendorUserId = "user";
var vendorSecretKey = "password";
var df = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ssZ");
var text = df.format(new java.util.Date());
var requestTimestamp = text.substring(0, 22) + ":" + text.substring(22);
var encryptString = requestTimestamp + vendorUserId;
var secretKey = new SecretKeySpec(new java.lang.String(vendorSecretKey).getBytes(), "HmacSHA1");
var mac = Mac.getInstance("HmacSHA1");
mac.init(secretKey);
var rawHmac = mac.doFinal(new java.lang.String(encryptString).getBytes());
var hexChars = Hex.encodeHex(rawHmac);
var signature = new java.lang.String(hexChars);
props.put('requestSignature', signature);
props.put('requestTimestamp', requestTimestamp);
props.put('requestUserId', vendorUserId);
Notice props.put(...)
at the end of the JavaScript code. This is where the actual values of the tokens are set: {requestUserId},{requestSignature} and {requestTimestamp}
.
Step 3.3. Create a nested flow, which includes the flows created in steps 3.1 and 3.2.
SSL
To allow an SSL handshake with the HTTP endpoint protected by self-signed or expired SSL certificates enable parameter Trust self-signed and expired certificates.
Using client-side SSL certificate with HTTP connection
Etlworks HTTP connector supports mutual authentication, which allows the user to present a certificate as well as the server.
To configure HTTP connection with the client-side SSL certificate (assuming that you already have a certificate):
Step 1. Use the field "Client SSL Certificate" to upload the certificate. The following formats are supported: jks, pfx, p12, p11.
Step 2. Select SSL certificate type in the "Client SSL Certificate Type" field.
Step 3. Enter Passphrase in the "Client SSL Password" field.
Configuring auto-retry when working with web services
To configure auto-retries use the Number of retries field when configuring the HTTP connection.
Available parameters
- Number of Retries - the number of retries allowed if the HTTP call was not successful. The maximum allowed number of retries is 10. The default is 0.
- Delay between retries - delay in milliseconds between retries if the HTTP call was not successful. The maximum allowed delay is 5 minutes (300,000 milliseconds). The default is 0.
Configuring anti-throttling parameters
Most public and private APIs implement various throttling algorithms, such as rate-limit throttling, to prevent API from being overwhelmed by too many requests.
In Etlworks you can set the artificial delay after the specified number of calls which will reduce the rate of calls to the given API endpoint.
Available parameters
- Wait after N calls (ms) - if this value is greater than 0 the system will pause and wait the configured number of milliseconds (less or equal to 10 minutes) before making another call to this endpoint. The endpoint is derived from the URI authority (https://myapi.com/data -> myapi.com). You can override the endpoint.
- Number of calls - this property works together with 'Wait after N calls (ms)' and defines the total number of calls to this endpoint the system can make before pausing for the configurable number of milliseconds.
- Endpoint - by default, the endpoint is derived from the URI authority (https://myapi.com/data -> myapi.com). Use this property to override the endpoint.
Comments
0 comments
Please sign in to leave a comment.