Create a pipeline (nested flow)

Nested flow basics

In Integrator it is possible to create flows of different types:

• Flows working with databases
• Flows working with files
• Flows working with web services
• Flows executing SQL and JavaScript
• Flows sending emails
• Specialized flows, like Enterprise API connectors
• Many other flow types

There is an easy way to combine flows of different types into a single nested flow (pipeline), add conditions, loops, and parallel execution.

Using this technique, you can create very complex data integration scenarios, where one flow is used as input for another, where flows are executed multiple times based on conditions, or where flows are not executed at all when conditions are not met.

Creating nested flow

Assuming that you have already created flows that you want to combine into a single nested flow:

Step 1. Start creating a nested flow by opening the Flows window, clicking the Add flow button and typing Nested flow into the "Select Flow Type" box:

Step 2. Continue by adding flows to the nested flow by clicking the Add flow button.

Removing flows from the pipeline

Disabling flows

Flow parameters

For each flow in a pipiline (inner flow) you can specify parameters by clicking the pen button.

Available parameters

• Parallel - if enabled (disabled by default) the inner flow will be executed in parallel, in its own thread. Disable it if you want to "chain" inner flows together.
• Condition - a JavaScript condition, which, if false is returned, disables execution of the inner flow.
• Loop type - the available loop types are: JavaScript (the default, means to execute while JavaScript returns a not null value), SQL (execute SQL, executes the inner flow for each row of the result set), and Files by wildcard (executes for all files matching a wildcard).
• Connection - an optional field, which is available when SQL or Files by wildcard is selected for the Loop type.
• Loop Script - a JavaScript, SQL or wildcard loop condition.
• Maximum number of iterations - this parameter is designed to prevent the system from creating infinite loops when the loop conditions are not properly defined. The default value is 100000. By setting the value to <= 0 (for example, to -1) you can disable the prevention mechanism altogether, so be careful.
• Loop Threads - a maximum allowed number of parallel loop executions.

Execute flow conditionally

Read how to write a JavaScript to execute flow conditionally.

Execute flow in a loop

Read how to execute flow in a loop.

Flow Variables

When creating a nested flow, you can add Flow Variables - key/value pairs, which can be referenced in SQL and JavaScript code as a {VARIABLE}. For example, there is a flow variable NAME, defined for the nested flow:

Using flow variables

A flow variable can be referenced in the Source Query as demonstrated below:

select * from connection
where lower(name) like '%{NAME}%'

A flow variable can also be accessed within the JavaScript.

Named Connections

Typically, a nested flow inherits all the connections from the inner flows. You can add additional named connections under the Connections tab.

Using named connections

You can then use the named connections in inner flows, for example in JavaScript.