Complex workflow (nested flow)

When to use this flow type

In Integrator, it is possible to create flows of different types:

Flows working with databases
Flows working with files
Flows working with web services
Flows executing SQL and JavaScript
Flows sending emails
Specialized flows, like Enterprise API connectors
Many other flow types

There is an easy way to combine flows of different types into a single nested flow (pipeline), add conditions, loops, and parallel execution.

Using this technique, you can create very complex data integration scenarios, where one flow is used as input for another, where flows are executed multiple times based on conditions, or where flows are not executed at all when conditions are not met.

Creating nested flow

Assuming that you have already created flows that you want to combine into a single nested flow:

Step 1. Start creating a nested flow by opening the Flows window, clicking the Add flow button and typing Nested flow into the "Select Flow Type" box:

Step 2. Continue by adding flows to the nested flow by clicking the Add flow button.

Removing flows from the pipeline

Disabling flows

Flow parameters

For each flow in a pipiline (inner flow), you can specify parameters by clicking the pen button.

Available parameters

Parallel - if enabled (disabled by default) the inner flow will be executed in parallel, in its own thread. Disable it if you want to "chain" inner flows together.
Condition - a JavaScript condition, which, if false is returned, disables execution of the inner flow.
Loop type - the available loop types are: JavaScript (the default, means to execute while JavaScript returns a not null value), SQL (execute SQL, executes the inner flow for each row of the result set), and Files by wildcard (executes for all files matching a wildcard).
Connection - an optional field, which is available when SQL or Files by wildcard is selected for the Loop type.
Loop Script - a JavaScript, SQL or wildcard loop condition.
Maximum number of iterations - this parameter is designed to prevent the system from creating infinite loops when the loop conditions are not properly defined. The default value is 100000. By setting the value to <= 0 (for example, to -1) you can disable the prevention mechanism altogether, so be careful.
Loop Threads - a maximum allowed number of parallel loop executions.

Execute flow conditionally

You can create a nested flow where some of the steps (inner flows) are configured to be executed conditionally.

Read how to execute steps in the nested flow conditionally.

Alternatively, some flows, such as the flows below can be executed conditionally even when they are not part of the nested flow.

Read more about executing flows conditionally.

Execute flow in a loop

You can create a nested flow where some of the steps (inner flows) are configured to be executed in a loop.

Typical use cases for executing flows in a loop:

Extract data from the HTTP endpoints with parameters. The parameters can be provided at runtime by using the loop.
Configure the transformations and connections by reading configuration parameters from the database.
Dynamically setting up the credentials for the connections.
Executing flow the fixed number of times.

Read how to execute flow in a loop.

Flow Variables

When creating a nested flow, you can add Flow Variables - key/value pairs, which can be referenced in SQL and JavaScript code as a {VARIABLE}. For example, there is a flow variable NAME, defined for the nested flow:

Using flow variables

A flow variable can be referenced in the Source Query as demonstrated below:

select * from connection
where lower(name) like '%{NAME}%'

A flow variable can also be accessed within the JavaScript.

Named Connections

Typically, a nested flow inherits all the connections from the inner flows. You can add additional named connections under the Connections tab.

Using named connections

You can then use the named connections in inner flows, for example in JavaScript.

Articles in this section

When to use this flow type