Release Notes Through 2020-10-05

This is a major release that, in addition to the new features, includes a lot of improvements, fixes, and changes under the hood. Self-hosted customers are encouraged to upgrade their instances. 

Version

New

1. Improvements for log-based change replication (CDC)

1.1 Upgraded Debezium to 1.3

Affected areas: CDC Flows, CDC Connections

We upgraded the change data capture engine (Debezium) to the latest stable version 1.3. Previously Etlworks Integrator was using Debezium 1.2.

1.2 Oracle CDC connector now supports native LogMiner adapter

Affected areas: CDC Flows, Oracle CDC connector

Debezium ingests change events from Oracle using the XStream API or directly by LogMiner.

To use the XStream API, you need to have a license for the GoldenGate product (though it is not required that GoldenGate itself is installed).

LogMiner is a built-in Oracle replication API that doesn't require an additional license to use.

1.3 New option to stop the long-running CDC Flow

Affected areas: CDC Flows, CDC connectors

Prior to this release, it was possible to configure the CDC connector to automatically stop pulling the data if there are no new CDC events after a configurable number of retries or a configurable number of minutes. 

In this release, we added a new option to stop the CDC Flow after the configurable number of minutes, even if new CDC events are coming from the database.

This option can be used to shut down a long-running CDC Flow unconditionally. 

2. Support for SQL Server Change Data Tracking (CT)

Affected areas: data replication, change data tracking

SQL Server Change Tracking, also known as CT, is a lightweight tracking mechanism introduced for the first time in SQL Server 2008. It can be used to track the DML changes performed in SQL Server database tables. SQL Change Tracking can be configured in all SQL Server editions, including the free Express edition.

In this update, we added the ability to use change data tracking for data replication.

3. The ability to use JavaScript to calculate high watermark value (HWM)

Affected areas: data replication, change replication using high watermark

Prior to this update, it was possible to use SQL to calculate the HWM value.

In this update, we added the ability to use JavaScript as well. 

4. Flow parameters when executing Flow manually

Affected area: manually executing Flows

Run flow dialog now allows adding additional parameters (Flow and global variables) for on-demand executions. Additional parameters are remembered per Flow, so you don't have to type them in every time.

5. Improvements for the scheduler

5.1 Flow Parameters

Affected area: scheduling Flows

User suggestion: Allow setting Flow variables in scheduler

It is now possible to configure the Flow parameters that will be passed to the Flow at run time. The parameters will be added as global and Flow variables.

5.2 The ability to configure Flows that shouldn't be running at the time of schedule getting triggered

Affected area: scheduling Flows

User suggestion: Scheduler queue

Non-listener Schedules now have No running flows list, which allows selecting a list of Flows that shouldn't be running at the time of that schedule getting triggered.

5.3 The ability to stop Flow automatically if it is running longer than expected

Affected area: scheduling Flows

User suggestion: Max runtime value for scheduled job

If the Flow is running longer than expected, it is now possible to automatically stop it after a configurable time.  

5.4 The ability to disable and enable the schedule using API

Affected area: scheduling Flows, schedules management API

• Enable schedule API
• Disable schedule API

6. The ability to search parameters and exceptions in the audit-trail log

Affected area: audit-trail

User suggestion: Add the ability to search within audit log JSON

We added two new fields to the Audit Trail filter: Parameters and Exeption.

7. HubSpot connector now supports OAuth2 authentication

Affected area: HubSpot connector

We added the ability to Sign in with Hubspot which simplified the authentication for the HubSpot connector. 

Usability and UX improvements

Flow statistics dashboard

Icons and labels

Flows

Listener Flows now clearly marked with listener icon as in other places.

Nested Flows now have a dropdown that shows all child Flows and icons that show if that Flow has a defined condition or loop.

User suggestion: Add a nested Flows icon to the Flow list

Schedule icon changed from bright green to filled in with checkmark to reduce attention and differentiate from successful execution icon and listener icon.

Edit (pencil icon) has been removed (on all other screens as well). Run Flow icon will not be shown for Listener Flows (instead of being disabled).

Flows export and import icons were put into a single dropdown to be consistent with Connections.

Flows export moved to its own modal dialog, same as in Connections for performance reasons and consistency.

Schedules

Edit (pencil icon) and statistics removed from Schedules list controls since they are available from the list itself.

Listener Schedules in the list now show the Listener icon and next execution is set to LIVE instead of just saying Listener and Listening.

Statistics

Statistics daily executions list now shows proper names of the trigger (Started by) and those are direct links as well

and can be extended to full screen.

Audit-trail

Audit trail records that had HTTP exception with no other information now will include that HTTP error response body.

The duration column value was changed from long unreadable milliseconds to hh:mm:ss.mil format. Example: 853354 ms > 00:14:13.354  

Other UX improvements

Bottom controls in all editors now have a top border to clearly separate them from the rest of the screen.

Rare log in screen infinite loop blinking has been fixed.

Breadcrumbs on all screens were updated. Each part of the breadcrumbs is a proper link now, so it is possible to use right-click and open on a new tab.