Overview
In Etlworks, you can configure any workflow step (child flow) to execute in parallel with other steps.
Parallel execution is useful when steps are independent and can run at the same time.
Use parallel execution only when steps do not depend on each other’s output or side effects.
Configure Parallel Execution
To enable parallel execution for a step:
- Select the step in the workflow.
- Open Flow parameters for the step.
- Enable Run this flow in parallel with other child flows.
When enabled, Etlworks will execute this step concurrently with other steps that are eligible to run in parallel.
When to Use Parallel Steps
Parallel execution is appropriate when steps are independent, for example:
- Calling multiple HTTP endpoints where each call uses different parameters and does not depend on results from other calls
- Running independent data preparation flows in the same workflow
- Performing separate exports or loads that do not touch the same destination objects
Avoid parallel execution when steps:
- Write to the same tables or files
- Depend on shared state that can be modified concurrently
- Must run in a strict order
Parallel Steps vs Parallel Loop Iterations
Parallel execution can be configured in two different places depending on what you are trying to run concurrently.
Parallel Steps (Child Flows)
When Run this flow in parallel with other child flows is enabled, Etlworks executes that step concurrently with other steps in the same workflow.
This applies to different steps that are part of the workflow structure.
Parallel Loop Iterations
If a step is configured to run in a loop, the Loop Threads parameter controls whether iterations of that loop run in parallel.
This applies to repeated executions of the same step within a loop.