Version

New

1. Permissions-driven UI

Affected areas: roles and permissions

Prior to this update, we were relying on the backend to notify the user that certain functions are not available. For example, if a user with a "viewer" role was trying to update the flow the system was generating an error.

In this update, we introduced the permission-driven UI which adjusts itself based on the roles and permissions given to the specific user.  Using the example above the Save button is always disabled for the user with a "viewer" role.

We also renamed the "Executor" role to a more logical "Operator".

2. Working with on-premise data

Affected areas: working with on-premise data

In this update, we made our remote Integration Agent available to the general audience. Agents can be used to work with the on-premise data. 

Prior to this update, this technology was available to the selected customers only.

Integration Agent is a zero-maintenance, easy-to-configure, fully autonomous ETL engine, running as background service behind the company’s firewall.  

It can be installed on Windows and Linux.

Remote Integration Agents allow you to run flows that use on-premise applications and databases. Outbound communication between Etlworks cloud instance and Remote Integration Agent is fully secured as data is not staged. 

Note that this is the first public release of the Integration Agent. In the next couple of releases, we are planning to dramatically improve integration between remote Agents and the cloud instance. Please stay tuned. 

Read about using remote Integration Agents.

3. The ability to use SQL to view data in the specific Excel or Google worksheet in Explorer

Affected areas: working with Excel, working with Google Sheets

Prior to this update, it was required to use a specific Excel format or Google Sheets connection to view the data in the specific worksheet in Explorer.

In this update, we extended our built-in SQL engine with the ability to select data from the specific worksheet.

• Selecting data from the specific Excel worksheet in Explorer 
• Selecting data from the specific Google Sheets worksheet in Explorer

4. The ability to read from and write into different worksheets using the same Google Sheets connection

Affected areas: working with Google Sheets

In Etlworks the name or index of the Google Sheets worksheet is configured in the Google Sheets connection.

Prior to this update, it was required to create a new connection to read/write from/into different worksheets.

In this update, we introduced the ability to read from and write into different worksheets using the same Google Sheets connection.

This functionality is similar to the one that we introduced early for the Excel.

5. The ability to delete source files after loading into the destination

Affected areas: source-to-destination transformation when the source is a file.

One of the typical uses-cases is when you want to extract/load the file and then delete it. 

In this update, we added the ability to automatically delete the source file(s) after loading data into the destination.

The transformation can be configured to delete the source file(s) on success, on error or both.

6. The ability to execute JavaScript or Python code after the load

Affected areas: scripting transformations

In addition to the existing Before Extract, For Each Row, and After Extract, we added After Load scripting transformation.

As the name says it is executed in a last step of the extract-transform-load after the data was successfully loaded into the destination. The transformation can be used to set global and flow variables, logging, clean up or for any other reason.

You can use JavaScipt (default) or Python in the scripting transformations.

7. The ability to set number of threads when copying files to the internal Snowflake stage

Affected areas: loading data in Snowflake

When configuring a flow to load data in Snowflake it was always possible to use the internal named Snowflake stage. All you need to do is create a named stage and set the destination in the Snowflake source-to-destination transformation to the Server storage. It can save you some money since most likely you are already paying for the internal Snowflake storage.

Now, the flow needs to copy files to load from the local hard drive (the volume associated with the Server storage connection) to the internal Snowflake stage.  We are using the Snowflake PUT command to copy files into the internal storage. 

One of the parameters supported by PUT command is PARALLEL=number of threads. The default number of threads is 4 but in this update, we added the ability to change it. It can be anything from 1 (no parallel load) to 99.

Read more.

8. The ability to limit the number of files processed by COPY/MOVE/DELETE/RENAME/ZIP/UNZIP flows

Affected areas: file management flows

In this update, we introduced the ability to limit the number of files that can be processed by the COPY/MOVE/DELETE/RENAME/ZIP/UNZIP flows.

If the value of the property Maximum Number of Files to Process is greater than 0 the flow will stop processing the files after the number of processed files will reach the configured threshold. 

Use it if the number of files is extremely large and you want to process them in chunks.