Complex workflow (nested Flow)

When to use this Flow type

In the Etlworks Integrator, it is possible to create Flows of different types:

Flows working with databases.
Flows working with files.
Flows working with web services.
Flows executing SQL and JavaScript.
Flows sending emails.
Specialized Flows, like Enterprise API connectors.
Many other Flow types.

There is an easy way to combine Flows of different types into a single nested Flow (pipeline), add conditions, loops, and parallel execution.

Using this technique, you can create very complex data integration scenarios, where one Flow is used as input for another, where Flow is executed multiple times based on conditions, or where Flow is not executed at all when conditions are not met.

Create nested Flow

Assuming that you have already created Flows that you want to combine into a single nested Flow:

Step 1. Start creating a nested Flow by opening the Flows window, clicking Add flow, and typing Nested Flow into the Select Flow Type box:

Step 2. Continue by adding Flows to the nested Flow by clicking Add flow.

Remove Flows from the pipeline

Disable Flows

Flow parameters

For each Flow in a pipeline (inner Flow), you can specify parameters by clicking the pen icon.

Available parameters

Parallel: if enabled (disabled by default), the inner Flow will be executed in parallel, in its own thread. Disable it if you want to chain inner Flows together.
Condition: a JavaScript condition, which, if false is returned, disables execution of the inner Flow.
Loop type: the available loop types are: JavaScript (the default, means to execute while JavaScript returns a not null value), SQL (execute SQL, executes the inner Flow for each row of the result set), and Files by wildcard (executes for all files matching a wildcard).
Connection: an optional field, which is available when SQL or Files by wildcard is selected for the Loop type.
Loop Script: a JavaScript, SQL, or wildcard loop condition.
Maximum number of iterations: this parameter is designed to prevent the system from creating infinite loops when the loop conditions are not properly defined. The default value is 100000. By setting the value to <= 0 (for example, to -1) you can disable the prevention mechanism altogether, so be careful.
Loop Threads: a maximum allowed number of parallel loop executions.

Execute Flow conditionally

You can create a nested Flow where some of the steps (inner Flows) are configured to be executed conditionally.

Read how to execute steps in the nested Flow conditionally.

Alternatively, some Flows, such as the Flows below, can be executed conditionally even when they are not part of the nested Flow.

Read more about executing Flows conditionally.

Execute Flow in a loop

You can create a nested Flow where some of the steps (inner Flows) are configured to be executed in a loop.

Typical use cases for executing Flows in a loop:

Extract data from the HTTP endpoints with parameters. The parameters can be provided at runtime by using the loop.
Configure the transformations and Connections by reading configuration parameters from the database.
Dynamically setting up the credentials for the Connections.
Executing Flow the fixed number of times.

Read how to execute Flow in a loop.

Flow Variables

When creating a nested Flow, you can add Flow Variables, KEY/ VALUE pairs, which can be referenced in SQL and JavaScript code as a {VARIABLE}. For example, there is a Flow variable NAME, defined for the nested Flow:

Use Flow variables

A Flow variable can be referenced in the Source query as demonstrated below:

select * from connection
where lower(name) like '%{NAME}%'

A Flow variable can also be accessed within the JavaScript.

Named Connections

Typically, a nested Flow inherits all the Connections from the inner Flows. You can add additional named Connections under the Connections tab.

Use named Connections

You can then use the named Connections in inner Flows, for example, in JavaScript.

Articles in this section

When to use this Flow type