Etlworks Integrator 2.2 has been released!
1. Added ability to process database tables by wildcard name
Previously, to extract and load data from/to multiple database tables you would have to create separate pairs of source-to-destination transformations.
In this update, we introduced an ability to process database objects (tables and views) by wildcard names.
- How to move tables matching a wildcard name from one schema or database to another
- Loading multiple table into Snowflake by wildcard name
- Loading multiple table into Amazon Redshift by wildcard name
2. Added CLOB format which can be used to easily transform the text messages
The most powerful transformation is in Etlworks source-to-destination. In almost all cases it hides the complexity of working with specific data formats and allows ETL developers to use high-level instruments, such as mapping editor.
There are cases, however, when you just want to make a few changes in the source text document and save it to the same or different location.
3. New UI/UX
Affected areas: using Etlworks Integrator
In this update, we introduced a two-panel interface in the Connections window. It is now extremely easy to navigate and update connections, listeners, and formats on the fly, without switching between the grid and the editor.
Statistics and Audit
To improve performance and usablity we split the Statistics into two independent windows:
We introduced a two-panel interface in Messages.
We also made tons of small usability and CSS changes across all UI elements to make Integrator look and behave better than before.
3. Find Connection, Listener, Format Usage
Prior to this update, you would have to manually check flows, use tags, or specific naming conventions to figure out which connection/format/listener belongs to which flow(s).
In this update, we introduced the ability to discover how connections, formats, and listeners are used in the flows.
4. Added named connections to Script flow
Affected areas: script flow
5. Option to add UUID suffix to the file name when loading files in Redshift and Snowflake
Previously, when loading files directly into Snowflake or Redshift (without any transformations) the system was simply copying files as-is from the source to the destination and executing direct load command (COPY INTO or COPY). In a typical scenario, it was not causing any issues. However, when the direct load flow is executed using UPLOAD API it is possible that multiple instances of the same flow will be running in parallel. This could create a scenario when the file with the same name is getting created by multiple instances of the flow at the same time.
To fix this problem we added an option to add a configurable unique suffix to the filename when copying/moving files to the stage area and before executing the direct load command. The default suffix is UUID.
6. Any to Any ETL flow
Affected areas: source-to-destination transformation
Any to Any ETL flow is a flavor of the flow type Extract data from source, transform, and load in destination.
Unlike all other flows in this group, Any to Any ETL flow allows developers to include different types of source-to-destination transformations within the same flow.
For example, you can pipeline the transformation which extracts data from a web service and loads it into the temporary database with the transformation which loads data from the temp database into multiple targets: databases, files, etc.
Read more about any to any ETL flow.
Affected areas: Excel format
1. Fixed parsing XSL spreadsheets with columns formatted using scientific notation
Previously, when reading XSL spreadsheets with columns formatted using scientific (exponential) notation the system was converting them to the long: 1.23E+10 -> 12345678901.
In this update, we fixed it so the original formatting is preserved.