Currently Running Tasks API

Overview

Our ETL engine records real-time metrics for all currently running tasks:

• Scrips, file operations, transformations
• Extract and Load
• CDC (change data capture)
• High Watermark

The tasks can be accessed from UI or using a built-in API.

GET tasks for the specific Flow API

This API returns running tasks for the specific Flow.

Authentication

Before making a call to API, the user must receive a JWT token from the authentication API.

Step 1. Use any user with the Administrator role to call Etlworks authentication endpoint and receive an access token.

Step 2. Use the access token received in Step 1 to call Etlworks API endpoints. The access token must be submitted as a header parameter, as in: Authorization:Bearer access-token.

Access tokens are short-lived and self-expiring. An access token gives you access to the APIs for approximately 10 minutes. It is recommended that you refresh the access token before each call to the API.

The API endpoint parameters

• PATH: /etl/rest/v1/tasks/{audit_id}/?type=type.

The type is an optional query parameter that can be set to one of the following:

• • stream: change data capture (CDC) stream
  • extract: extract
  • load: load
  • task: scrips, file operations, transformations
  • hwm: a query to get the current value of the High Watermark Field 

EXAMPLE: https://app.etlworks.com/etl/rest/v1/tasks/80610/?type=stream.

• METHOD: GET.
• HEADER: Authorization:Bearer access-token.
• REQUEST BODY: none.
• REQUEST CONTENT TYPE: none.

Audit ID

audit_id is a required URL parameter. To get the audit_id for the specific Flow, use the Flow status API. 

Response

The response is a JSON document in the following Format:

[{
   "requestId": "string",
   "owner": "string",
   "name": "string",
   "started": timestamp_in_milliseconds_since_epoch,
   "duration": number,
   "code": "the formatted CDC metrics",
   "type": "type",
   "tenant": "string",
   "language": null,
   "latency": number,
   "maxLatency": number,
   "minLatency": number,
   "avgLatency": number,
   "recordsProcessed": number,
   "recordsProcessedSinceLastCheck": number,
   "lastCheck": timestamp_in_milliseconds_since_epoch,
   "lastTimeRecordsReceivedDate": timestamp_in_milliseconds_since_epoch}
]

Response codes

• 200 for success
• 401 for not authorized
• 403 for forbidden
• 500 for an internal error

GET all currently running tasks API

This API returns all currently running tasks.

Authentication

Before making a call to API, the user must receive a JWT token from the authentication API.

Step 1. Use any user with the Administrator role to call Etlworks authentication endpoint and receive an access token.

Step 2. Use the access token received in Step 1 to call Etlworks API endpoints. The access token must be submitted as a header parameter, as in: Authorization:Bearer access-token, or query parameter, as in: ?Authorization=Bearer%20access-token.

Access tokens are short-lived and self-expiring. An access token gives you access to the APIs for approximately 10 minutes. It is recommended that you refresh the access token before each call to the API.

The API endpoint parameters

• PATH: /etl/rest/v1/tasks/?type=type

The type is an optional query parameter that can be set to one of the following:

• • stream: change data capture (CDC) stream
  • extract: extract
  • load: load
  • task: scrips, file operations, transformations
  • hwm: a query to get the current value of the High Watermark Field

EXAMPLE: https://app.etlworks.com/etl/rest/v1/tasks/?type=stream

• METHOD: GET
• HEADER: Authorization:Bearer access-token
• REQUEST BODY: none
• REQUEST CONTENT TYPE: none

Response

The response is a JSON document in the following Format:

[{
   "requestId": "string",
   "owner": "string",
   "name": "string",
   "started": timestamp_in_milliseconds_since_epoch,
   "duration": number,
   "code": "the formatted CDC metrics",
   "type": "type",
   "tenant": "string",
   "language": null,
   "latency": number,
   "maxLatency": number,
   "minLatency": number,
   "avgLatency": number,
   "recordsProcessed": number,
   "recordsProcessedSinceLastCheck": number,
   "lastCheck": timestamp_in_milliseconds_since_epoch,
   "lastTimeRecordsReceivedDate": timestamp_in_milliseconds_since_epoch}
]

Response codes

• 200 for success
• 401 for not authorized
• 403 for forbidden
• 500 for an internal error