Version

New

1. The ability to generate and use the API key with user-defined API endpoints

Affected areas: JWT authentication, user management 

Prior to this update, it was required to make a separate authentication call to the /token-auth/issue API endpoint which returns a short-lived JWT token that can be later used to make a call to the user-defined API.

In this update, we introduced the ability to generate the not-expirable API key and use it as an access token to authenticate calls to the user-defined APIs.

Read about the API key.

Read about the JWT authentication when calling the user-defined API endpoints.

2. Improvements when extracting data using wildcard source name

Affected areas: working with databases, working with files, loading data in Snowflake, loading data in Redshift

The ability to include objects

By default, when the wildcard source name is used in the source-to-destination transformation the system extracts data from all database objects (tables and views) or files that match the wildcard name.

Prior to this update, it was possible to exclude objects when extracting data by a wildcard source (FROM) name.

In this update, we introduced the ability to Include only specific database objects or files. Use comma (,) to include multiple objects. The wildcard object names are supported in the inclusion list.

Note that you must use the fully qualified database object name in the inclusion list.

The ability to exclude all tables or all views

To exclude all views enter all views in the Exclude objects field.

To exclude all tables enter all tables in the Exclude objects field.

The ability to use token {table} in the Source Query

It is now possible to use token {table} (or {TABLE}) in the Source query when extracting data by a wildcard source name.

It can be useful you want to extract only the specific field(s) from the source. Example:

select ID from {table}

3. The ability to execute JavaScript or Python program before executing flows 

Affected areas:  Send an email, Execute SQL, Merge data with the template, Send payload to the HTTP endpoint, Execute shell (command line) script

It is now possible to execute the JavaScript or Python program before executing any of the flows above.

Use cases:

• Setting the global variables which can be used to parameterize transformations and connections and flow variables that can be used to parameterize the Source query.
• Executing the flow conditionally 

4. The ability to use relative path in FROM and TO when reading and creating files

Affected areas: reading and writing files from/to specific folder using the same connection

It is now possible to use the same connection to read files from (and writes files to) the specific subfolder under the same folder.

5. The ability to configure how input data should be interpreted when writing to the Google Sheet

Affected areas: Google Sheets connector.

In this update, we added the ability to configure how input data should be interpreted when writing to the Google Sheet.

There are two available options:

• RAW - the values the user has entered will not be parsed and will be stored as-is.
• USER_ENTERED - the values will be parsed as if the user typed them into the UI. Numbers will stay as numbers, but strings may be converted to numbers, dates, etc. following the same rules that are applied when entering text into a cell via the Google Sheets UI.

6. Reporting API

Affected areas: built-in APIs

In this update, we added the new API endpoint that allows users to create reports displaying the aggregated per-flow metrics available in the flows executions dashboard in the Statistics window.

Read about reporting API. 

7. Global variables are created when passing URL parameters to the user-defined APIs

Affected areas: user-defined APIs

Prior to this update when the user-defined API endpoint was called with the URL or query parameters the system was automatically extracting the parameters from the URL creating the flow variables with the same name. Flow variables can be used to parametrize the Source and Destination queries but cannot be used to parametrize the connections and transformations.

In this update, we changed this behavior. Now, when the user-defined API endpoint is called with the URL or query parameters the system creates global variables, as well as flow variables.

8. UX/UI improvements

Affected areas: Etlworks Integrator UI

Direct links to the editors

It is now possible to access the specific editor (flow, scheduler, agent, user, tenant, etc.) right from the grid, without clicking the dedicated Edit button. The Edit button will be removed in the next update.

A confirmation dialog when running selected flows from the scheduler

We added a confirmation dialog box displayed when the user wants to run the selected flows from the Scheduler.

The previous behavior was to run flows immediately and without the prompt. 

Changes

1. The maximum number of threads is now set to 5 by default

Affected areas: limiting maximum number of threads when running extract and load in parallel

In this update, we set the maximum number of threads to 5 by default. The previous behavior was to allow the system to choose the number of threads which was causing the performance issues in some cases. Note that the max number of threads is not a required field so you can still enable the previous behavior if you want to. 

2. The process all files is now enabled by default

Affected areas: processing files in a folder by a wildcard filename

In this update, we enabled the Process all files flag by default. Prior to this update, it was disabled by default even when the FROM was configured with a wildcard. 

3. The flows stats dashboard now displays pairs of sources and targets as a single transformation when processing objects by a wildcard

Affected areas: working with databases

Prior to this update, the pairs of sources and targets were displayed as two separate rows, which was causing the perception that they displayed twice. 

4. All database connectors now set the JDBC Fetch Size to 50000 by default

Affected areas: database connectors

The JDBC fetch size is an optional field that can be used to set the maximum number of records which will be retrieved in one database call, thus limiting the memory consumption within the JVM. You can set the Fetch Size at the connection level, or override it at the transformation level.

In this update, we defaulted this field to 50000. Prior to this update, it was not set by default which was causing the performance issues when extracting data from large tables with millions of rows. 

Fixed

1. Fixed a bug causing the Google Sheets connector to not clear the worksheet

Affected areas: Google Sheets connector.

In this update, we fixed the bug which was causing the Google Sheets connector to not fully clear the worksheet when the number of rows to load is fewer than the current number of rows in a worksheet. 

2. Fixed a bug causing the transformations to be executed out of order if the flow has a mix of the wildcard and non-wildcard transformations

Affected areas: working with databases, working with files, loading data in Snowflake, loading data in Redshift

In this update, we fixed the bug which was causing the wildcard transformations to be executed first, regardless of the order of transformations in the flow.

3. Fixed a bug in Snowflake and Redshift flows when extracting data using a wildcard source and using high-watermark change replication

Affected areas: loading data in Snowflake, loading data in Redshift

In this update, we fixed the bug which was causing the Snowflake and Redshift flows to ignore the {table} token in the HWM query when extracting data using a wildcard source. Prior to this fix, the flow was automatically switching the full load, regardless of HWM settings.