Release Notes Through 2020-06-21

Version

New

1. Webhooks

Webhooks allow you to build or set up integrations, which subscribe to certain events in Etlworks Integrator. When one of those events is triggered, we'll send an HTTP POST or GET request to the webhook's configured URL. Webhooks can be used to update your external system about events happening in the Etlworks Integrator.

Read how to create and manage webhooks.

Read how to use API to manage webhooks.

2. The ability to export and import Connections, Formats, and Listeners

Affected areas: managing Connections, managing Formats, managing Listeners

Prior to this update, it was possible to export and import Connections, Formats, and Listeners only when exporting and importing Flows. 

In this update, we introduced the ability to independently export and import Connections, Formats, and Listeners.

3. Improvements for HTTP connector

Affected area: HTTP connector

3.1 GET and DELETE methods can now have a payload

The ability to save response HTTP headers into the global variable

When the option Save Response Headers into Global Variable is enabled, the headers in the HTTP response returned by the endpoint will be saved to the global variable http_processor_last_response_headers, which can be accessed using com.toolsverse.config.SystemConfig.instance().getContextProperties().get('http_processor_last_response_headers').

3.2 SSL handshake with the HTTP endpoints protected by self-signed or expired SSL certificates

To allow an SSL handshake with the HTTP endpoint protected by self-signed or expired SSL certificates, enable parameter Trust self-signed and expired certificates. 

3.3 Anti-throttling

Most public and private APIs implement various throttling algorithms, such as rate-limit throttling, to prevent API from being overwhelmed by too many requests.

In Etlworks Integrator, you can set the artificial delay after the specified number of calls which will reduce the rate of calls to the given API endpoint.

Read how to configure anti-throttling parameters.

HTTP connector now supports multi-part payload.

4. Salesforce connector now supports OAuth2 authentication

Affected area: Salesforce connector

Prior to this update, the Etlworks Salesforce connector required a security token for authentification.

In this update, we added a Salesforce connector with OAuth2 authentication.

5. Improvements for database connectors

Affected area: database connectors

In this update, we added a new configuration option for all database connectors: Current Database and Schema only. If this option is enabled, the system populates tables and views for the current database and schema only. For databases with a lot of objects, it can greatly reduce the time needed to show the objects in Explorer and in the mapping.

6. New Elasticsearch connectors

Affected area: Elasticsearch connectors

In this update, we added two new Elasticsearch connectors: Elasticsearch SQL and Elasticsearch Premium. 

The old Elasticsearch connector was renamed Elasticsearch Legacy.

7. MongoDB connectors now support partial update 

Affected area: MongoDB connectors

Prior to this update, the MongoDB connector was replacing all fields of the existing document.

In this update, we introduced the ability to update only the fields in the existing document that are passed in the payload and leave other fields unchanged. 

8. Improvements for outbound email Connection and send email Flow

Affected areas: Outbound email connector, send email Flow

In this update, we introduced the ability to configure the Header and the Footer which are added to the email message (the body). Header and the Footer can be configured for the specific Outbound email Connection and independently for the Send email Flow.

9. New options for parsing CSV Formats

Affected area: CSV Format

In this update, we added two new options for parsing non-standard CSV files when some of the data rows have a fewer or greater number of columns than a header row.

10. Improvements for log-based change replication (CDC)

Affected area: CDC connectors

New configurable options for CSV serialization:

• Escape double-quotes: is this option is enabled, a double-quote appearing inside a field will be escaped by preceding it with another double quote.
• Preprocessor: the optional JavaScript that can be used to change the CDC event before saving it to the CSV file. The following variables are available in the preprocessor code:
  • event: the CDC event serialized as com..fasterxml.jackson.databind.JsonNode. You can change the event using the methods of the JsonNode class.
  • key: the CDC key. This variable is immutable. 

New configurable option for pulling:

• Retry N minutes before giving up: the number of minutes to retry before giving up if the poll returns no records.

11. Usability improvements

11.1 Tables and Views are now sorted alphabetically in Etlworks Explorer.

11.2 Listener editor shows and allows to copy the actual Endpoint URL.

11.3 The ability to export Macros and Schedules.

11.4 Added confirmation dialog when manually executing schedules.

11.5 Inbound messages UI now has a link to the Flow, which processed the message

Read more.

12. The ability to embed Etlworks UI

In this update, we introduced the ability to embed the Etlworks UI, specifically the connectors, into the host web application that supports iframes. 

13. Improvements for File Loop

Affected area: File Loop

Read about new global and Flow variables set by the fille loop on each iteration.

Fixes and performance improvements

1. Significantly improved performance of the nested Flows with parameters and Flows with wildcard loops

We fixed the issue which was causing a delay when executing nested Flows with parameters and Flows with wildcard loops.

2 Fixed opening nested Flow from the direct link

Prior to this update, when the nested Flow was opened from the direct link, the changing tracking feature was disabled.

3. Fixed caching for Marketo connector

Prior to this update, the Marketo connector was caching the OAuth2 properties, so it wasn't possible to use multiple Connections to the different Marketo instances at the same time.

4. Automatic cleanup of the stuck maintenance tasks

In this update, we introduced the automatic cleanup of the stuck maintenance tasks.