Version
New
1. The ability to generate and use the API key with user-defined API endpoints
Affected areas: JWT authentication, user management
Prior to this update, it was required to make a separate authentication call to the /token-auth/issue API endpoint, which returns a short-lived JWT token that can be later used to make a call to the user-defined API.
In this update, we introduced the ability to generate the not-expirable API key and use it as an access token to authenticate calls to the user-defined APIs.
Read about the API key.
Read about the JWT authentication when calling the user-defined API endpoints.
2. Improvements when extracting data using wildcard source name
Affected areas: working with databases, working with files, loading data in Snowflake, loading data in Redshift
2.1 The ability to include objects
By default, when the wildcard source name is used in the source-to-destination transformation, the system extracts data from all database objects (tables and views) or files that match the wildcard name.
Prior to this update, it was possible to exclude objects when extracting data by a wildcard source (FROM) name.
In this update, we introduced the ability to include only specific database objects or files. Use comma (,) to include multiple objects. The wildcard object names are supported in the inclusion list.
Note that you must use the fully qualified database object name in the inclusion list.
2.2 The ability to exclude all tables or all views
To exclude all views enter all views in the Exclude objects field.
To exclude all tables enter all tables in the Exclude objects field.
2.3 The ability to use token {table} in the Source Query
It is now possible to use token {table} (or {TABLE}) in the Source query when extracting data by a wildcard source name.
It can be useful you want to extract only the specific field(s) from the source. Example:
select ID from {table}
3. The ability to execute JavaScript or Python program before executing Flows
Affected areas: Send an email, Execute SQL, Merge data with the template, Send payload to the HTTP endpoint, Execute shell (command line) script
It is now possible to execute the JavaScript or Python program before executing any of the Flows above.
Use cases:
- Setting the global variables which can be used to parameterize transformations and Connections and Flow variables that can be used to parameterize the
Source query. - Executing the Flow conditionally.
4. The ability to use relative path in FROM and TO when reading and creating files
Affected area: reading and writing files from/to specific folder using the same connection
It is now possible to use the same Connection to read files from (and writes files to) the specific subfolder under the same folder.
5. The ability to configure how input data should be interpreted when writing to the Google Sheet
Affected area: Google Sheets connector
In this update, we added the ability to configure how input data should be interpreted when writing to the Google Sheet.
There are two available options:
- RAW: the values the user has entered will not be parsed and will be stored as-is.
- USER_ENTERED: the values will be parsed as if the user typed them into the UI. Numbers will stay as numbers, but strings may be converted to numbers, dates, etc., following the same rules applied when entering text into a cell via the Google Sheets UI.
6. Reporting API
Affected area: built-in APIs
In this update, we added the new API endpoint that allows users to create reports displaying the aggregated per-flow metrics available in the Flows executions dashboard in the Statistics window.
Read about reporting API.
7. Global variables are created when passing URL parameters to the user-defined APIs
Affected area: user-defined APIs
Prior to this update, when the user-defined API endpoint was called with the URL or query parameters, the system automatically extracted the parameters from the URL, creating the Flow variables with the same name. Flow variables can be used to parametrize the Source and Destination queries but cannot be used to parametrize the Connections and transformations.
In this update, we changed this behavior. Now, when the user-defined API endpoint is called with the URL or query parameters, the system creates global variables, as well as Flow variables.
8. UX/UI improvements
Affected area: Etlworks Integrator UI
8.1 Direct links to the editors
It is now possible to access the specific editor (Flow, scheduler, agent, user, tenant, etc.) right from the grid, without clicking Edit. Edit will be removed in the next update.
8.2 A confirmation dialog when running selected Flows from the scheduler
We added a confirmation dialog box displayed when you want to run the selected Flows from the Scheduler.
The previous behavior was to run Flows immediately and without the prompt.
Changes
1. The maximum number of threads is now set to 5 by default
Affected area: limiting the maximum number of threads when running extract and load in parallel
In this update, we set the maximum number of threads to 5 by default. The previous behavior was to allow the system to choose the number of threads that were causing the performance issues in some cases. Note that the max number of threads is not a required field, so you can still enable the previous behavior if you want to.
2. The process all files is now enabled by default
Affected area: processing files in a folder by a wildcard filename
In this update, we enabled the Process all files flag by default. Prior to this update, it was disabled by default even when the FROM was configured with a wildcard.
3. The Flows stats dashboard now displays pairs of sources and targets as a single transformation when processing objects by a wildcard
Affected area: working with databases
Prior to this update, the pairs of sources and targets were displayed as two separate rows, causing the perception that they were displayed twice.
4. All database connectors now set the JDBC Fetch Size to 50000 by default
Affected area: database connectors
The JDBC fetch size is an optional field that can be used to set the maximum number of records that will be retrieved in one database call, thus limiting the memory consumption within the JVM. You can set the Fetch Size at the Connection level or override it at the transformation level.
In this update, we defaulted this field to 50000. Prior to this update, it was not set by default, causing performance issues when extracting data from large tables with millions of rows.
Fixed
1. Fixed a bug causing the Google Sheets connector not to clear the worksheet
Affected area: Google Sheets connector
In this update, we fixed the bug which was causing the Google Sheets connector to not fully clear the worksheet when the number of rows to load is fewer than the current number of rows in a worksheet.
2. Fixed a bug causing the transformations to be executed out of order if the Flow has a mix of the wildcard and non-wildcard transformations
Affected areas: working with databases, working with files, loading data in Snowflake, loading data in Redshift
In this update, we fixed the bug which was causing the wildcard transformations to be executed first, regardless of the order of transformations in the Flow.
3. Fixed a bug in Snowflake and Redshift Flows when extracting data using a wildcard source and using high-watermark change replication
Affected areas: loading data in Snowflake, loading data in Redshift
In this update, we fixed the bug which was causing the Snowflake and Redshift Flows to ignore the {table} token in the HWM query when extracting data using a wildcard source. Prior to this fix, the Flow was automatically switching the full load, regardless of HWM settings.
Comments
0 comments
Please sign in to leave a comment.