PUSH API: when API endpoint receives a payload it triggers execution of a data integration flow

About PUSH API

In Integrator, you can configure an API endpoint so that when it receives a payload from a third-party application, it will trigger the execution of a data integration flow.

PUSH API definition

PUSH API is a variation of the user-defined API.

Steps to building a REST API

Step 1. Create an API user, which will be used to run the flow.

Step 2. Create a listener. Most likely you will create an HTTP listener.

Step 3. Create a format for the payload. Most likely, you will use JSON, XML, HL7 2.x or HL7 FIHR, CSV, or Fixed length Text as a payload.

Step 4. Start creating a flow by selecting a flow type where the source is a web service.

Step 5. Continue by creating a transformation where the source connection (FROM) is a listener.

Only one listener can be used in a single flow.

Step 6. Add mapping and parameters just like you normally would.

Step 7. Since a flow with a listener cannot be manually executed, it must be scheduled.

Scheduled event-driven flows are executed when a payload arrives, as opposed to regular flows, which are executed at specific time intervals.

User-defined response

By default PUSH API endpoints return a pre-defined response. It is quite often required to return a specific response when calling a user-defined PUSH API endpoint.

Step 1. Configure HTTP listener with POST/PUT processing set to Sync and not empty POST/PUT Response File Name, for example, {app.data}/response.json.

Step 2. Create PUSH API.

Step 3. Create a flow that will create a response file. Note that the file name and location must be the same as configured in Step 1.

Step 4. Combine flows created in Steps 2-3 into the nested flow.

Step 5. Schedule flow created in Step 4.

The response file created in step 4 is automatically deleted when the API endpoint returns the response back to the caller.

Handling empty payload

It is common for the payload sent to the POST/PUT HTTP listener to be empty. For example, you might be using POST listener as a receiver for the webhook created in a third party application and the third-party application might be sending a POST request without the body. The payload is used as a source in the source-to-destination transformation so when it is empty, Eltworks Integrator will likely generate an exception because it will not be able to parse the payload.

To handle cases when the payload is empty, enter the "default" payload in the accepted format (for example, JSON) in the Default Payload field when configuring the HTTP listener.

Working with HTTP headers in Request and Response

In many cases, the important information is passed in the HTTP headers. In this section of the documentation, we will show you how to get the HTTP headers passed in the request and how to return them in the response.

Getting the HTTP headers passed in the request

To convert HTTP headers to the Flow Variables, simply enable the "Pass request headers to the flow as parameters" option when configuring the HTTP listener. 

Be aware that when HTTP headers are converted to Flow Variables, the variable name will change to ALL CAPS. Example Validation-Token->VALIDATION-TOKEN

All flow variables passed from the listener are available in the flow that has the listener and/or, if the flow is a part of the nested flow, in the condition for this flow. For example, assuming that the original header parameter is Validation-Token you can use the following code to access the value of the Validation-Token in the flow condition:

var token = scenario.getVariable('VALIDATION-TOKEN').getValue();

Returning the HTTP headers in the response

To add the header to the HTTP response, use the following code anywhere where you can execute the JavaScript or Python code:

etlConfig.addResponseHeader('header-name', header-value);

End-to-end example

Problem

The RingCentral allows configuring a webhook that will send a notification to the HTTP endpoint when someone subscribes to the service. The endpoint must be configured as a POST with no payload but with a Validation Token passed in the HTTP request. To complete the subscription, the endpoint must return the same Validation Token in the HTTP response. The actual name of the HTTP header is Validation-Token.

Solution

Step 1. Create HTTP listener to receive the notification from the webhook. Configure it as synchronous POST endpoint.

Step 2. Create a flow where the listener created in step 1 is a source.

Step 3. Create a nested flow and add the flow created in step 2.

Step 4. Click the edit button and configure the flow condition as following:

importPackage(com.toolsverse.config);

var token = scenario.getVariable('VALIDATION-TOKEN');

etlConfig.addResponseHeader('Validation-Token', token != null ? 
token.getValue() : '');

value = true;

Viewing inbound messages (a payload)

To view and query the inbound messages (a payload) sent to the PUSH API use the messaging app.

Testing an API endpoint

For a flow which is configured to be executed on-demand, you can create another flow, which will send an HTTP request from Integrator.

Let's assume that there is a scheduled on-demand flow which is triggered by sending a POST request to the https://app.etlworks.com/rest/v1/httplistener/patient endpoint, with a payload that contains a JSON document representing the patient.

For testing purposes, let's put patient.json in the folder on the server or cloud storage.

Step 1. Create an HTTP connection with the following parameters:

• URL: https://app.etlworks.com/rest/v1/httplistener/patient.
• Authentication: token.
• Authentication URL: https://app.etlworks.com/rest/v1/token-auth/issue.
• HTTP Method for Token Authentication: POST.
• Authentication Request Payload: {"login":"{user}","password":"{password}"}
• Authentication Request Content Type: application/json.
• User: any existing Integrator user who has permission to run flows.
• Password: password for the user.
• Method: POST.

Step 2. Create a flow where the source connection points to patient.json, the source format is JSON, the destination connection is the connection created in Step 1, and the destination format is the same as the source format.

Step 3. Manually execute the flow created in Step 2 and watch it trigger the execution of an on-demand flow.