Version

New

1. Webhooks

Webhooks allow you to build or set up integrations, which subscribe to certain events in Etlworks Integrator. When one of those events is triggered, we'll send a HTTP POST or GET request to the webhook's configured URL. Webhooks can be used to update your external system about events happening in the Integrator.

Read how to create and manage webhooks.

Read how to use API to manage webhooks.

2. The ability to export and import Connections, Formats and Listeners

Affected areas: managing connections, managing formats, managing listeners

Prior to this update, it was possible to export and import connections, formats and listeners only when exporting and importing flows. 

In this update, we introduced the ability to independently export and import connections, formats and listeners.

3. Improvements for HTTP connector

Affected areas: HTTP connector

GET and DELETE methods can now have a payload

The ability to save response HTTP headers into the global variable

When the option Save Response Headers into Global Variable is enabled, the headers in HTTP response returned by the endpoint will be saved to the global variable http_processor_last_response_headers, which can be accessed using com.toolsverse.config.SystemConfig.instance().getContextProperties().get('http_processor_last_response_headers').

SSL handshake with the HTTP endpoints protected by self-signed or expired SSL certificates

To allow an SSL handshake with the HTTP endpoint protected by self-signed or expired SSL certificates enable parameter Trust self-signed and expired certificates.

Anti-throttling

Most public and private APIs implement various throttling algorithms, such as rate-limit throttling, to prevent API from being overwhelmed by too many requests.

In Etlworks you can set the artificial delay after the specified number of calls which will reduce the rate of calls to the given API endpoint.

Read how to configure anti-throttling parameters.

HTTP connector now supports multi-part payload

4. Salesforce connector now supports OAuth2 authentication

Affected areas: Salesforce connector

Prior to this update, Etlworks Salesforce connector required a security token for authentification.

In this update, we added a Salesforce connector with OAuth2 authentication.

5. Improvements for database connectors

Affected areas: database connectors

In this update, we added a new configuration option for all database connectors - Current Database and Schema only. If this option is enabled the system populates tables and views for the current database and schema only. For databases with a lot of objects, it can greatly reduce the time needed to show the objects in Explorer and in the mapping.

6. New Elasticsearch connectors

Affected areas: Elasticsearch connectors

In this update, we added two new Elasticsearch connectors: Elasticsearch SQL and Elasticsearch Premium. 

The old Elasticsearch connector was renamed to Elasticsearch Legacy.

7. MongoDB connectors now support partial update 

Affected areas: MongoDB connectors

Prior to this update, the MongoDB connector was replacing all fields of the existing document.

In this update, we introduced the ability to update only the fields in the existing document that are passed in the payload and leave other fields unchanged. 

8. Improvements for outbound email connection and send email flow

Affected areas: Outbound email connector,  send email flow

In this update, we introduced the ability to configure the Header and the Footer which are added to the email message (the body). Header and the Footer can be configured for the specific Outbound email connection and independently for the Send email flow.

9. New options for parsing CSV formats

Affected areas: CSV format

In this update, we added two new options for parsing non-standard CSV files, when some of the data rows have a fewer or greater number of columns than a header row.

10. Improvements for log-based change replication (CDC)

Affected areas: CDC connectors

New configurable options for CSV serialization

• Escape double-quotes - is this option is enabled, a double-quote appearing inside a field will be escaped by preceding it with another double quote
• Preprocessor - the optional JavaScript that can be used to change the CDC event before saving it to the CSV file. The following variables are available in the preprocessor code:
  • event - the CDC event serialized as com.fasterxml.jackson.databind.JsonNode. You can change the event using the methods of the JsonNode class.
  •  key - the CDC key. This variable is immutable. 

New configurable option for pulling

• Retry N minutes before giving up - the number of minutes to retry before giving up if the poll returns no records.

11. Usability improvements

Tables and Views are now sorted alphabetically in Explorer

Listener editor shows and allows to copy the actual endpoint URL

The ability to export Macros and Schedules

Added confirmation dialog when manually executing schedules

Inbound messages UI now has a link to the flow which processed the message

Read more.

12. The ability to embed Etlworks UI

In this update, we introduced the ability to embed the Etlworks UI, specifically the connectors into the host web application that supports iframes. 

13. Improvements for File Loop

Affected areas: File Loop

Read about new global and flow variables set by the fille loop on each iteration. 

Fixes and performance improvements

1. Significantly improved performance of the nested flows with parameters and flows with wildcard loops

We fixed the issue which was causing a delay when executing nested flows with parameters and flows with wildcard loops.

2 Fixed opening nested flow from the direct link

Prior to this update, when the nested flow was opened from the direct link the changing tracking feature was disabled.

3. Fixed caching for Marketo connector

Prior to this update, the Marketo connector was caching the OAuth2 properties so it wasn't possible to use multiple connections to the different Marketo instances at the same time.

4. Automatic cleanup of the stuck maintenance tasks

In this update, we introduced the automatic cleanup of the stuck maintenance tasks.