Etlworks Integrator 2.3 has been released!

New

1. Added nine new connectors

Affected areas: connectors

• Microsoft Sharepoint connector
• OneDrive for business connector
• Google Drive connector
• Box connector
• Dropbox connector
• Smartsheet connector
• Magento connector
• OData connector
• Amazon Marketplace connector

2. New account usage notifications and account dashboard

Affected areas: notifications, account dashboard

Account usage notifications

We added the following new notifications to help customers stay within the limits set for the specific subscription plan.

• Etlworks Integrator daily overage notice (day of the month) - sent to the account owner when the number of records processed during the previous day exceeded the maximum number allowed for the subscription plan.
• Etlworks Integrator monthly overage notice (month) - sent to the account owner when the number of records processed during the previous month exceeded the maximum number allowed for the subscription plan.

New account dashboard

We added more important information to the account dashboard but kept it clean and easy to navigate.

• New look&feel
• New data points: the total number of records for the day, the current date in the timezone, configured for the account, the total number of the records for the selected month, the limits set for the account
• Ability to see stats for the specific month 
• Ability to refresh real-time stats
• The dashboard now displays the overage information, see examples below:

Exceeding the daily maximum

Potential monthly overage

Exceeding the monthly maximum

3. Improved handling of the schema changes when loading data in Snowflake and Redshift

Affected areas: loading data in Snowflake and Redshift 

The following options are available for both databases (two new):

1. Ignore fields in the source that do not exist in the target table - if this option is enabled and the source has more fields than the target table, the system will automatically exclude extra fields. Options 1, 2 and 3 are mutually exclusive. 
2. NEW. Alter target table if the source has columns that the target table doesn't have - if this option is enabled and the source has different fields than the target table, the system will add extra fields to the target table. Options 1, 2 and 3 are mutually exclusive. 
3. NEW. Recreate target table if the source has columns that the target table doesn't have - if this option is enabled and the source has different fields than the target table, the system will automatically drop and recreate the target table. This option is not available when HWM change replication is enabled. Options 1, 2 and 3 are mutually exclusive. 
4. Insert null into the fields in the target table that do not exist in the source - if this option is enabled and the target table has different fields than the source, the system will automatically insert NULL values into these fields in the target.

4. Added support for PGP encryption

Affected areas: scripting, PGP encryption

In this update, we added the ability to encrypt and decrypt files and messages using PGP (Pretty Good Privacy) algorithms. 

Read more.

5. Added the ability to use a client-side SSL certificate with HTTP connection 

Affected areas: HTTP connector

Etlworks HTTP connector now supports mutual authentication, which allows the user to present a certificate as well as the server. 

Read more.

6. Change Data Capture (CDC) flows now support wildcard table names

Affected areas: CDC flows, loading data in Snowflake and Redshift 

Prior to this update, it was required to configure individual source-to-destination transformations for each pair of source-target tables.

In this update, we introduced the ability to use wildcard table names. If the wildcard table name is used the system will perform a CDC change replication for all tables in the source with a name that matches the wildcard pattern (example: test.dbo.prod_*)

7. Improved handling of the errors in CDC flows

Affected areas: CDC flows

In this update, we introduced new option for the CDC connections. If any error occurred and the option Switch to Snapshot if Error is enabled, one the next run the system will automatically switch to the snapshot mode and attempt to reload the whole table. 

8. Added new flow type for merging data with a template

Affected areas: merging data with template

In this update, we introduced a new flow type: Merge data with a template. 

Use it when you want to merge global variables with the template and send the resulting payload to any of the following destination connections: 

• File-based connections
• HTTP
• Social networks
• Outbound email
• NoSQL databases

Read more.

9. Added the ability to configure output template for the CLOB format

Affected areas: CLOB format

If the Output Template field is not empty, the {tokens} in the template will be merged with global variables to produce the output. 

10. Added the ability to parse comments in the XML document

Affected areas: XML parser, XML Format

In this release, we introduced the ability to parse comments in the XML document.

Fixed

1. Fixed FTPS connector

Affected areas: FTPS connector

Prior to this update, in some rare cases, the FTPS connector was returning an empty list even when there were files in the folder. We fixed it by introducing a new configuration option "Execute PBSZ and PROT commands After Login", which is enabled by default.

2. Updated Salesforce connector

Affected areas: Salesforce connector

Salesforce updated some of the APIs so we updated our Salesforce connector.

3. Fixed issue with email attachments

Affected areas: sending emails

Prior to this release, the system was always sending an attachment even when the outbound email connection wasn't configured to have one. We fixed it in this release.

4. Fixed HTTP authentication when the authorization header parameter is user-defined

Affected areas: HTTP authentication when the authorization header parameter is user-defined

Previously, it wasn't possible to use tokens {user} and {password} in the user-defined authorization header. We fixed it in this release.

5. Fixed parsing of the x-www-form-urlencoded payload by the HTTP Listener

Affected areas: user-defined PUSH API, HTTP listener

Previously, if the payload was sent using x-www-form-urlencoded content type the HTTP listener wasn't able to process it. We fixed it in this release.