Version
New
1. Version control
Affected area: version control
In this update, we implemented the most requested feature: version control.
Read about version control in Etlworks.
Version control is supported for the following artifacts: Flows, Connections, Formats, Listeners, Schedules, and Macros.
Using built-in version control, you can:
- View the history of changes - who-changed-what-and-when.
- Compare any two versions.
- Revert to any previous version.
- Add comments to the commit when saving the artifacts.
2. New connectors
Affected areas: working with messages queues, change data capture, working with Microsoft Exchange
In this update, we added two new connectors:
3. Improved usability when configuring a database Connection
Affected area: database connectors
Prior to this update, it was required to enter the JDBC URL to configure the database Connection.
After this update, you can still use the URL if you want to, but you only need to enter database-specific parameters, such as host, port, database name, etc.
4. New change data capture (CDC) techniques
Affected area: change data capture
Prior to this update, the following CDC techniques were available to developers:
- CDC with Kafka and Debezium
- Point-to-point CDC
In this update, we introduced two new CDC techniques:
5. Ability to schedule Flows to run continuously
Affected area: scheduling Flows
Prior to his update, the following Schedule types were available:
In this update, we introduced a new Schedule type: scheduling Flow to run continuously.
6. New scripting transformation, executed before extract
Affected area: scripting transformations
Prior to this update, the following scripting transformations were available to developers:
In this update, we introduced before extract transformation, which can be used to set global and Flow variables on the fly.
7. Web/HTML scrapping using JavaScript and Python
Affected area: Web/HTML scraping.
In this update, we introduced the ability to use JavaScript or Python for Web/HTML scrapping.
8. HL7 connector now supports HL7 version 2.7 and 2.8
Affected area: working with HL7 messages.
Prior to this update, our connector supported the following HL7 versions: 2.1-2.6. Support for 2.7 was limited.
In this update, we added support for HL7 versions 2.7 and 2.8.
9. Added the ability to use parallel threads when processing sources by a wildcard
Affected areas: source-to-destination transformation
In this update, we introduced the ability to use Parallel Threads when processing sources by a wildcard.
10. Added the ability to delete multiple files in Explorer
Affected area: deleting files in Explorer
Prior to this update, it was only possible to delete a single file in Etlworks Explorer.
In this update, we introduced the ability to delete multiple selected files.
11. New options for Snowflake COPY INTO
Affected areas: extract, transform and load data in Snowflake
In this update, we introduced new options for configuring Format for Snowflake COPY INTO:
- Replace Invalid characters: if enabled, Snowflake replaces invalid UTF-8 characters with the Unicode replacement character.
- Error on Column Mismatch: this option specifies whether to generate a parsing error if the number of delimited columns (i.e., fields) in an input data file does not match the number of columns in the corresponding table. If disabled, an error is not generated, and the load continues. If the file is successfully loaded: If the input file contains records with more fields than columns in the table, the matching fields are loaded in order of occurrence in the file, and the remaining fields are not loaded. If the input file contains records with fewer fields than columns in the table, the non-matching columns in the table are loaded with
NULLvalues. - Trim Space: this option specifies whether to remove white space from fields.
- String used to convert to and from SQL NULL: When loading data, Snowflake replaces these strings in the data load source with SQL NULL. To specify more than one string, enclose the list of strings in parentheses and use commas to separate each value. For example:
NULL_IF = ('N', 'NULL', 'NUL', '')
12. Limiting the maximum number of threads when loading data in parallel from multiple sources
Affected area: source-to-destination transformation
Prior to this update, it was required to use a Flow variable MAX_THREADS to limit the maximum number of threads when loading data in parallel from multiple sources.
In this update, we introduced a new configuration parameter that can be set in the nested Flow.
13. The built-in Python scripting engine now supports Python 2.7
Affected area: writing scripts using Python
Prior to this update, the built-in Python scripting engine only supported Python 2.4.
In this update, we upgraded the Python engine to the latest and greatest, supporting Python 2.7.
14. Ability to filter out errors and warnings when sending notifications about ignored errors
Affected areas: handling exceptions in the Flows and the transformations
In this update, we introduced the ability to filter out warnings and exceptions based on a reason when sending notifications about ignored errors.
15. Ability to capture response headers from the last HTTP call
Affected area: working with web services
In this update, we introduced the ability to capture the response headers from the last HTTP call.
Fixed
1. Bug with the decimal numbers when loading data from Excel worksheets into the new database table
Affected areas: Excel Format, source-to-destination transformation when the destination is a database
Condition: the source Excel worksheet has a column with decimal numbers.
Flow: the Flow loads data into the database and creates a database table on the fly by sampling the source Excel worksheet.
Previous behavior: the column in a new database table is created as NUMBER, hence ignoring the decimal part.
New behavior: the column in a new database table is created as NUMBER(max, 5).
2. Browsing metadata for Microsoft Access databases in Explorer
Affected areas: Explorer, MS Access connector
Prior to this update, there was an error This feature is not supported when attempting to expand the Microsoft Access Connection in Etlworks Explorer to see tables and views.
In this update, we fixed the error. It is now possible to browse MS Access tables and views in Etlworks Explorer.
Comments
0 comments
Please sign in to leave a comment.