Create and manage Flows

About Data Integration Flows

Data Integration Flow is an ETL or automation scenario which can be executed manually, on a schedule, or triggered by calling an API endpoint.

Read about available Flows.

Create a Flow

In most cases, before you can start creating Flows, you need to create all the required Connections, Formats, and, in some cases, Listeners.

Step 1. To create a new Flow, go to the Flows page, click +, and then click the corresponding Flow type in the opened box.

To quickly find the correct type, you can enter all or a part of the name in the search box. After this, only the types which include the specified string in their names will be displayed.

To filter Flows by category, you can click the corresponding Flow category from the list to the left of Flow types.

Step 2. After a Flow type has been selected, continue by defining the Flow's transformations. To add a new transformation, click +.

Transformations can be modified, deleted, and rearranged using drag-and-drop.

Most of the transformations require FROM and TO Connections and Formats, as well as the names of the FROM and TO. Objects could be file names, database tables, etc.

Step 3. For each transformation, you will need to define the Mapping and parameters.

Start defining the Mapping by clicking MAPPING on the grid which contains the transformation.

Continue in the opened box by creating a per-field Mapping, either automatically (by clicking Create Mapping), or manually (by adding fields one-by-one using +). You can modify Mappings at any time. You can also rearrange fields using drag-and-drop.

Step 4. After the Flow has been created, click Save.

Manage Flows

All saved Flows are displayed in a grid on the Flows page and can be quickly filtered by name, type, and tag. Flows can be modified, scheduled, duplicated, or deleted directly from the grid.

Export and import Flows

In the Etlworks Integrator, you can export any Flow to the file saved locally, and later import it from that same file. When you export the Flow, the Etlworks Integrator also exports all related Connections and Formats.

Export/Import is especially useful when you wish to migrate Flows from one instance of the Etlworks Integrator to another, or copy Flows between different accounts.

Export a Flow

Go to the Flows page, click Export/Import Flow, and select Export. After which, select the Flow by name in the opened box and click Export. The file with the extension .intflow will be downloaded to your local drive.

Import a Flow

Go to the Flows page, click Export / Import Flow, and select Import. After which, click Choose file and select the file with the extension .intflow that you wish to import. A new copy of the Flow and all corresponding Connections and Formats will be created in the Etlworks Integrator.

Find where the Flow is included in the nested Flows 

In the Etlworks Integrator, any Flow can be added as a step to the nested Flow. To find all nested Flows where this Flow is included, click Find Usage at the bottom of the Flow editor screen:

The system will display a popup with a list of nested Flows which include this specific Flow. You can click on a link to open the nested Flow editor. 

If the Flow is included in the nested Flow, we display an icon next to the Flow name which indicates that the Flow has a parent nested Flow. Click on the icon to open the nested Flow editor.

Disable manual flow execution

If the Flow is a part of the nested Flow and is not meant to be executed independently you can disable the manual flow execution for that specific Flow under the Parameters tab.