Articles in this section

User-defined APIs

  • Updated
Follow
  • Startup
  • Business
  • Enterprise
  • On-Premise
  • Add-on

HTTP Listener

The core element of the infrastructure, which allows a developer to build REST APIs in Etlworks, is the HTTP Listener. The HTTP Listener is a special type of connector that is constantly listening to incoming HTTP requests, using a configurable PATH and METHOD.

Depending upon which side of the transformation (source or destination) the HTTP Listener sits, it can either push the payload to the Flow or return a response to the client.

Read more about event-driven Flows.

Related case study

NBCUniversal case study

Load metadata into MongoDB and create APIs for retrieving metadata for movies and shows.

NBCUniversal, a global leader in media and entertainment, needed a robust and scalable solution to integrate metadata for movies and shows from various sources into MongoDB and internal systems. With Etlworks, they implemented flexible pipelines capable of handling data from diverse formats and managing hundreds of thousands of API calls daily, all within a horizontally scalable cluster.

Transport for NSW case study

Integrate data from diverse sources to AWS for centralized management and analytics.

Local government agency Transport for NSW needed to create a REST API to aggregate data from a locally hosted SQL Server database, Google Analytics, Facebook, and third-party web services, providing enriched JSONs for Tableau visualizations. Traditional methods were too time-consuming and resource-intensive. With Etlworks, they built a working API in less than an hour and now add new data sources in minutes.

User-defined API

A user-defined API allows the developer to create custom HTTP endpoints, which can receive requests from third-party applications, triggering execution of the Etlworks's Flows.

  • PATH: /rest/v1/httplistener/{user-defined-path}.

EXAMPLE: https://app.etlworks.com/rest/v1/httplistener/hl7/patient, where /hl7/patient is a user-defined-path, defined when creating HTTP Listener.

  • METHOD: POST, PUT, GET, DELETE.
  • HEADER: Authorization:Bearer access-token.
  • REQUEST BODY: a payload in any of the supported Formats, such as CSV, XML, or JSON. The payload is not required.
  • REQUEST CONTENT TYPE: not required.
  • RESPONSE (GET): a response defined by the developer, typically a JSON document.
  • RESPONSE (PUSH and PUT): a pre-defined response or response defined by the developer, typically a JSON document.
  • RESPONSE (DELETE): a pre-defined response.
  • RESPONSE CODES: 200 for success, 401 and 403 for not authorized, and 500 for an internal error.

Pre-defined response (example)

{
 "auditId": 106358,
 "messageUuid": "caaa8347-063c-449b-a29b-12e68799c986",
 "flowId": 3219,
 "status": "success"
}

Examples

Below are the typical examples of user-defined APIs:

PULL: expose a data source as an API endpoint

  1. The Flow is configured and scheduled in Etlworks to receive GET requests, pull the data from any source, transform, and save it into the HTTP Listener (response).
  2. The third-party application, for example, Tableau makes a call to the API endpoint created in Etlworks to pull the enriched dataset.

Read about PULL API.

PUSH: when API endpoint receives a payload, it triggers the execution of a data integration Flow

  1. The third-party application updates its own database. 
  2. The update triggers an event, which sends a notification to Etlworks and includes updated record(s) as part of the payload.
  3. There is an HTTP Listener configured in Etlworks, which listens to the incoming HTTP requests.
  4. There is also a Flow, which has the Listener as the source (FROM) Connection. The Flow is scheduled and ready to receive a payload.
  5. Once the payload is received, Etlworks automatically executes the Flow and passes the payload as a source.
  6. The source data is transformed and placed into the destination. For example, information about an updated record(s) is sent to a data warehouse.

Read about PUSH API.

Related articles

Comments

0 comments

Please sign in to leave a comment.