Etlworks Integrator 2.2 has been released!
New
1. Added ability to process database tables by wildcard name
Affected areas: working with databases, loading data in Snowflake, loading data in Redshift
Previously, to extract and load data from/to multiple database tables you would have to create separate pairs of source-to-destination transformations.
In this update, we introduced an ability to process database objects (tables and views) by wildcard names.
- How to move tables matching a wildcard name from one schema or database to another
- Loading multiple table into Snowflake by wildcard name
- Loading multiple table into Amazon Redshift by wildcard name
2. Added CLOB format which can be used to easily transform the text messages
Affected areas: source-to-destination transformation, working with files
The most powerful transformation is in Etlworks source-to-destination. In almost all cases it hides the complexity of working with specific data formats and allows ETL developers to use high-level instruments, such as mapping editor.
There are cases, however, when you just want to make a few changes in the source text document and save it to the same or different location.
In this update, we introduced the CLOB format which can be used to easily transform text messages without engaging more complex transformations.
3. New UI/UX
Affected areas: using Etlworks Integrator
Connections
In this update, we introduced a two-panel interface in the Connections window. It is now extremely easy to navigate and update connections, listeners, and formats on the fly, without switching between the grid and the editor.
- Formats
Statistics and Audit
To improve performance and usablity we split the Statistics into two independent windows:
- Audit
Inbound Messages
We introduced a two-panel interface in Messages.
All-around
We also made tons of small usability and CSS changes across all UI elements to make Integrator look and behave better than ever before.
3. Find Connection, Listener, Format Usage
Affected areas: Managing Connections, Managing Formats, Managing Listeners
Prior to this update, you would have to manually check flows, use tags, or specific naming convention to figure out which connection/format/listener belongs to which flow(s).
In this update, we introduced an ability to discover how connections, formats, and listeners are used in the flows.
4. Added named connections to Script flow
Affected areas: script flow
Previously, if JavaScript or Python flow needed a connection (for example to iterate through the files in a folder) the nested flow would need to be created, then the connection and the JavaScript or Python flow would need to be added to the nested flow.
In this update, we introduced an ability to add named connections directly to the JavaScript/Python flows.
5. Option to add UUID suffix to the file name when loading files in Redshift and Snowflake
Affected areas: direct load into Snowflake, direct load into Redshift
Previously, when loading files directly into Snowflake or Redshift (without any transformations) the system was simply copying files as-is from the source to the destination and executing direct load command (COPY INTO or COPY). In a typical scenario, it was not causing any issues. However, when the direct load flow is executed using UPLOAD API it is possible that multiple instances of the same flow will be running in parallel. Which could create a scenario when the file with the same name is getting created by multiple instances of the flow at the same time.
To fix this problem we added an option to add a configurable unique suffix to the filename when copying/moving files to the stage area and before executing the direct load command. The default suffix is UUID.
6. Any to Any ETL flow
Affected areas: source-to-destination transformation
Any to Any ETL flow is a flavor of the flow type Extract data from source, transform, and load in destination.
Unlike all other flows in this group, Any to Any ETL flow allows developers to include different types of source-to-destination transformations within the same flow.
For example, you can pipeline the transformation which extracts data from a web service and loads it into the temporary database with the transformation which loads data from the temp database into multiple targets: databases, files, etc.
Read more about any to any ETL flow.
Fixed
Affected areas: Excel format
1. Fixed parsing XSL spreadsheets with columns formatted using scientific notation
Previously, when reading XSL spreadsheets with columns formatted using scientific (exponential) notation the system was converting them to the long: 1.23E+10 -> 12345678901.
In this update, we fixed it so the original formatting is preserved.
Comments
0 comments
Please sign in to leave a comment.