Overview
It is possible to execute multiple source-to-destination transformations (extract-load) in parallel threads.
By default, Etlworks limits the number of threads to 5 but you can change this limit.
Process
Here are the steps to limit the maximum number of threads:
Step 1. Create a Flow with parallel transformations.
Step 2. Create a nested Flow and add a Flow you created in step 1.
Step 3. Set the parameter Maximum Number of Parallel Threads to the positive integer > 1.
Alternatively, the max number of threads can be set for the flow with multiple source-to-destination transformations:
Thread-count behavior starting with 10.0.0
Configured values above 10 now take effect. In earlier versions, parallel execution pools were silently limited to 10 concurrent threads even when a higher value was configured. Starting with 10.0.0, the configured value is used, from 1 up to 100.
Important: flows that were saved with high thread counts will now actually run at that level. Before upgrading, review parallel settings and confirm that the source and destination systems can handle the additional concurrent load, then adjust the values if necessary.
Per-execution worker budget (10.0.0+)
A new property, etl.concurrency.max.workers (default 100, set in the application configuration), limits the total number of worker threads a single flow execution can use across scenarios, nested flows, loops, source extraction, and destination loads. When the limit is reached, nested work runs inline on the calling thread instead of waiting for a free worker, which prevents deadlocks in deeply nested parallel flows.
Notes:
- The budget applies per execution, not per JVM or per node.
- Heavily nested parallel flows that hit the budget may run partly in sequence and take longer; raise the value only when the host has the CPU and memory for it.
- Waiting for parallel tasks is now event-driven, and a configured timeout applies to the whole group of tasks rather than restarting per task.