Create and manage Flows

About Data Integration Flows

Data Integration Flow is an ETL or automation scenario which can be executed manually, on a schedule, or triggered by calling an API endpoint.

Read about available Flows.

Create a Flow

In most cases, before you can start creating Flows, you need to create all the required Connections, Formats, and, in some cases, Listeners.

Step 1. To create a new Flow, go to the Flows page, click +, and then click the corresponding Flow type in the opened box.

To quickly find the correct type, you can enter all or a part of the name in the search box. After this, only the types which include the specified string in their names will be displayed.

To filter Flows by category, you can click the corresponding Flow category from the list to the left of Flow types.

Step 2. After a Flow type has been selected, continue by defining the Flow's transformations. To add a new transformation, click +.

Transformations can be modified, deleted, and rearranged using drag-and-drop.

Most of the transformations require FROM and TO Connections and Formats, as well as the names of the FROM and TO. Objects could be file names, database tables, etc.

Step 3. For each transformation, you will need to define the Mapping and parameters.

Start defining the Mapping by clicking MAPPING on the grid which contains the transformation.

Continue in the opened box by creating a per-field Mapping, either automatically (by clicking Create Mapping), or manually (by adding fields one-by-one using +). You can modify Mappings at any time. You can also rearrange fields using drag-and-drop.

Step 4. After the Flow has been created, click Save.

Manage Flows

All saved Flows are displayed in a grid on the Flows page and can be quickly filtered by name, type, and tag. Flows can be modified, scheduled, duplicated, or deleted directly from the grid.

Export and import Flows

In Etlworks, you can export any Flow to the file saved locally, and later import it from that same file. When you export the Flow, Etlworks also exports all related Connections and Formats.

Export/Import is especially useful when you wish to migrate Flows from one instance of Etlworks to another, or copy Flows between different accounts.

Export a Flow

Go to the Flows page, click Export/Import Flow, and select Export. After which, select the Flow by name in the opened box and click Export. The file with the extension .intflow will be downloaded to your local drive.

Import a Flow

Go to the Flows page, click Export / Import Flow, and select Import. After which, click Choose file and select the file with the extension .intflow that you wish to import. A new copy of the Flow and all corresponding Connections and Formats will be created in Etlworks.

Find where the Flow is included in the nested Flows 

In Etlworks, any Flow can be added as a step to the nested Flow. To find all nested Flows where this Flow is included, click Find Usage at the bottom of the Flow editor screen:

The system will display a popup with a list of nested Flows which include this specific Flow. You can click on a link to open the nested Flow editor. 

If the Flow is included in the nested Flow, we display an icon next to the Flow name which indicates that the Flow has a parent nested Flow. Click on the icon to open the nested Flow editor.

Disable manual flow execution

If the Flow is a part of the nested Flow and is not meant to be executed independently you can disable the manual flow execution for that specific Flow under the Parameters tab.

Daemon fllows

Once started, daemon flow runs in the infinite loop until there is an exception or the flow is manually interrupted. 

For all intents and purposes, it is the equivalent of scheduling the flow to run every second but without the scheduling and without the step which initializes the resources (for example opens the connections) on each iteration. 

When to use daemon flows

The main use case for Daemon Flows is when you want to run integrations in micro-batches without scheduling the flow. The most common example is ETL from a message queue. The daemon ETL flow, where the source is a message queue, reads the fixed number of messages from the queue, sleeps for a few moments, and restarts from the last known position. 

The other common use case is when you want to actively monitor the resource, for example, a folder in the file storage, and trigger the flow when the resource is not empty, for example, there are files in the folder. 

Configure a flow to run as a daemon

Any flow can be a daemon flow but if there is a nested flow only the main nested flow can start as a daemon.

To set the flow as a Daemon simply navigate to the Parameters tab and enable the Daemon flag.