When to use this flow type
In Integrator, it is possible to create flows of different types:
- Flows working with databases
- Flows working with files
- Flows working with web services
- Flows executing SQL and JavaScript
- Flows sending emails
- Specialized flows, like Enterprise API connectors
- Many other flow types
There is an easy way to combine flows of different types into a single nested flow (pipeline), add conditions, loops, and parallel execution.
Using this technique, you can create very complex data integration scenarios, where one flow is used as input for another, where flows are executed multiple times based on conditions, or where flows are not executed at all when conditions are not met.
Creating nested flow
Assuming that you have already created flows that you want to combine into a single nested flow:
Step 1. Start creating a nested flow by opening the Flows window, clicking the Add flow
button and typing Nested flow
into the "Select Flow Type" box:
Step 2. Continue by adding flows to the nested flow by clicking the Add flow
button.
Removing flows from the pipeline
Disabling flows
Flow parameters
For each flow in a pipiline (inner flow), you can specify parameters by clicking the pen
button.
Available parameters
- Parallel - if enabled (disabled by default) the inner flow will be executed in parallel, in its own thread. Disable it if you want to "chain" inner flows together.
- Condition - a JavaScript condition, which, if
false
is returned, disables execution of the inner flow. - Loop type - the available loop types are:
JavaScript
(the default, means to execute while JavaScript returns anot null
value),SQL
(execute SQL, executes the inner flow for each row of the result set), andFiles by wildcard
(executes for all files matching a wildcard). - Connection - an optional field, which is available when
SQL
orFiles by wildcard
is selected for the Loop type. - Loop Script - a JavaScript, SQL or wildcard loop condition.
- Maximum number of iterations - this parameter is designed to prevent the system from creating infinite loops when the loop conditions are not properly defined. The default value is
100000
. By setting the value to<= 0
(for example, to-1
) you can disable the prevention mechanism altogether, so be careful. - Loop Threads - a maximum allowed number of parallel loop executions.
Execute flow conditionally
Read how to write a JavaScript to execute flow conditionally.
Execute flow in a loop
Read how to execute flow in a loop.
Flow Variables
When creating a nested flow, you can add Flow Variables - key/value pairs, which can be referenced in SQL and JavaScript code as a {VARIABLE}
. For example, there is a flow variable NAME
, defined for the nested flow:
Using flow variables
A flow variable can be referenced in the Source Query as demonstrated below:
select * from connection
where lower(name) like '%{NAME}%'
A flow variable can also be accessed within the JavaScript.
Named Connections
Typically, a nested flow inherits all the connections from the inner flows. You can add additional named connections under the Connections tab.
Using named connections
You can then use the named connections in inner flows, for example in JavaScript.
Comments
0 comments
Please sign in to leave a comment.