Version

New

1. Version control

Affected areas: version control.

In this update, we implemented the most requested feature: version control.

Read about version control in Etlworks.

Version control is supported for the following artifacts: flows, connections, formats, listeners, schedules, macros.

Using built-in version control you can:

• View the history of changes - who-changed-what-and-when.
• Compare any two versions.
• Revert to any previous version.
• Add comments to the commit when saving the artifacts.

2. New connectors

Affected areas:  working with messages queues, change data capture, working with Microsoft Exchange

In this update, we added two new connectors:

• Azure Event Hubs
• Microsoft Exchange

3. Improved usability when configuring a database connection

Affected areas: database connectors

Prior to this update, it was required to enter the JDBC URL to configure the database connection.

After this update, you can still use the URL if you want to but you only need to enter database-specific parameters, such as host, port, database name, etc. 

4. New change data capture (CDC) techniques

Affected areas:  change data capture

Prior to this update, the following CDC techniques were available to developers:

• CDC with Kafka and Debezium
• Point-to-point CDC

In this update, we introduced two new CDC techniques:

• CDC when events are sent to the message queue
• CDC when events are stored as CSV files

5. Ability to schedule flows to run continuously 

Affected areas:  scheduling flows

Prior to his update, the following schedule types were available:

• Scheduling periodically executed flows
• Scheduling real-time flows
• Scheduling event-driven flows

In this update, we introduced a new schedule type: scheduling flow to run continuously. 

6. New scripting transformation, executed before extract

Affected areas: scripting transformations

Prior to this update, the following scripting transformations were available to developers:

• For each row
• After extract

In this update, we introduced before extract transformation which can be used to set global and flow variables on the fly.

7. Web/HTML scrapping using JavaScript and Python

Affected areas: Web/HTML scraping

In this update, we introduced the ability to use JavaScript or Python for Web/HTML scrapping.

8. HL7 connector now supports HL7 version 2.7 and 2.8 

Affected areas: working with HL7 messages

Prior to this update, our connector supported the following HL7 versions: 2.1-2.6. Support for 2.7 was limited.

In this update, we added support for HL7 versions 2.7 and 2.8.

9. Added the ability to use parallel threads when processing sources by a wildcard

Affected areas: source-to-destination transformation

In this update, we introduced the ability to use Parallel Threads when processing sources by a wildcard.

10. Added the ability to delete multiple files in Explorer

Affected areas: deleting files in Explorer

Prior to this update, it was only possible to delete a single file in Explorer.

In this update, we introduced the ability to delete multiple selected files.

11. New options for Snowflake COPY INTO

Affected areas: extract, transform and load data in Snowflake

In this update, we introduced new options for configuring format for Snowflake COPY INTO.

• Replace Invalid characters - if enabled, Snowflake replaces invalid UTF-8 characters with the Unicode replacement character.
• Error on Column Mismatch - this option specifies whether to generate a parsing error if the number of delimited columns (i.e. fields) in an input data file does not match the number of columns in the corresponding table. If disabled, an error is not generated and the load continues. If the file is successfully loaded: If the input file contains records with more fields than columns in the table, the matching fields are loaded in order of occurrence in the file and the remaining fields are not loaded. If the input file contains records with fewer fields than columns in the table, the non-matching columns in the table are loaded with NULL values.
• Trim Space - this option specifies whether to remove white space from fields.
• String used to convert to and from SQL NULL - When loading data, Snowflake replaces these strings in the data load source with SQL NULL. To specify more than one string, enclose the list of strings in parentheses and use commas to separate each value. For example: NULL_IF = ('N', 'NULL', 'NUL', '')

12. Limiting the maximum number of threads when loading data in parallel from multiple sources

Affected areas: source-to-destination transformation

Prior to this update, it was required to use a flow variable MAX_THREADS to limit the maximum number of threads when loading data in parallel from multiple sources.

In this update, we introduced a new configuration parameter that can be set in the nested flow.

13. The built-in Python scripting engine now supports Python 2.7

Affected areas: writing scripts using Python

Prior to this update,  the built-in Python scripting engine only supported Python 2.4.

In this update, we upgraded the Python engine to the latest and greatest, which supports Python 2.7. 

14. Ability to filter out errors and warnings when sending notifications about ignored errors

Affected areas: handling exceptions in the flows and in the transformations

In this update, we introduced the ability to filter out warnings and exceptions based on a reason when sending notifications about ignored errors.

15. Ability to capture response headers from the last HTTP call

Affected areas: working with web services

In this update, we introduced the ability to capture the response headers from the last HTTP call.

Fixed

1. Bug with the decimal numbers when loading data from Excel worksheets into the new database table

Affected areas: Excel format, source-to-destination transformation when the destination is a database.

Condition: the source Excel worksheet has a column with decimal numbers.  

Flow: the flow loads data into the database and creates a database table on the fly by sampling the source Excel worksheet.

Previous behavior: the column in a new database table is created as NUMBER, hence ignoring the decimal part.

New behavior: the column in a new database table is created as NUMBER(max, 5).

2. Browsing metadata for Microsoft Access databases in Explorer

Affected areas: Explorer, MS Access connector

Prior to this update, there was an error "This feature is not supported" when attempting to expand the Microsoft Access connection in Explorer in order to see tables and views.

In this update, we fixed the error. It is now possible to browse MS Access tables and views in the Explorer.