About Data Integration Flows
Data Integration Flow is an ETL or automation scenario which can be executed manually, on a schedule, or triggered by calling an API endpoint.
Read about available Flows.
Create a Flow
In most cases, before you can start creating Flows, you need to create all the required Connections, Formats, and, in some cases, Listeners.
Step 1. To create a new Flow, go to the Flows
page, click +
, and then click the corresponding Flow type in the opened box.
To quickly find the correct type, you can enter all or a part of the name in the search box. After this, only the types which include the specified string in their names will be displayed.
To filter Flows by category, you can click the corresponding Flow category from the list to the left of Flow types.
Step 2. After a Flow type has been selected, continue by defining the Flow's transformations. To add a new transformation, click +
.
Transformations can be modified, deleted, and rearranged using drag-and-drop.
Most of the transformations require FROM
and TO
Connections and Formats, as well as the names of the FROM
and TO
. Objects could be file names, database tables, etc.
Step 3. For each transformation, you will need to define the Mapping and parameters.
Start defining the Mapping by clicking MAPPING
on the grid which contains the transformation.
Continue in the opened box by creating a per-field Mapping, either automatically (by clicking Create Mapping
), or manually (by adding fields one-by-one using +
). You can modify Mappings at any time. You can also rearrange fields using drag-and-drop.
Step 4. After the Flow has been created, click Save
.
Manage Flows
All saved Flows are displayed in a grid on the Flows
page and can be quickly filtered by name, type, and tag. Flows can be modified, scheduled, duplicated, or deleted directly from the grid.
Export and import Flows
In the Etlworks Integrator, you can export any Flow to the file saved locally, and later import it from that same file. When you export the Flow, the Etlworks Integrator also exports all related Connections and Formats.
Export/Import
is especially useful when you wish to migrate Flows from one instance of the Etlworks Integrator to another, or copy Flows between different accounts.
Export a Flow
Go to the Flows
page, click Export/Import Flow
, and select Export
. After which, select the Flow by name in the opened box and click Export
. The file with the extension .intflow
will be downloaded to your local drive.
Import a Flow
Go to the Flows
page, click Export / Import Flow
, and select Import
. After which, click Choose file
and select the file with the extension .intflow
that you wish to import. A new copy of the Flow and all corresponding Connections and Formats will be created in the Etlworks Integrator.
Find where the Flow is included in the nested Flows
In the Etlworks Integrator, any Flow can be added as a step to the nested Flow. To find all nested Flows where this Flow is included, click Find Usage
at the bottom of the Flow editor screen:
The system will display a popup with a list of nested Flows which include this specific Flow. You can click on a link to open the nested Flow editor.
If the Flow is included in the nested Flow, we display an icon next to the Flow name which indicates that the Flow has a parent nested Flow. Click on the icon to open the nested Flow editor.
Disable manual flow execution
If the Flow is a part of the nested Flow and is not meant to be executed independently you can disable the manual flow execution for that specific Flow under the Parameters
tab.
Comments
0 comments
Please sign in to leave a comment.