Overview

Workflow steps can be configured to execute in a loop.

Looping allows a step to run repeatedly based on a defined condition or dataset. This is useful when a workflow must process multiple items, such as files, database records, or dynamically generated values.

A loop is configured at the workflow step level.

The step executes repeatedly until the loop condition completes or reaches the configured iteration limit.

Loop Execution Behavior

When a step is configured with a loop:

• The workflow reaches the step. 
• The loop generates iterations. 
• The step executes once per iteration.

Each iteration runs the same flow with the loop context applied.

If nested steps exist inside the looped step, they execute within each iteration.

Configure Loop Execution

To configure looping for a step:

1. Select the workflow step. 
2. Open the Child Flow parameters panel. 
3. Configure the Loop settings.

Loop behavior is controlled using the loop configuration fields.

Loop Types

Etlworks supports several loop types that determine how iterations are generated.

Script Loop

A script loop allows you to define iteration logic using JavaScript or Python.

The script controls the loop behavior and determines how many times the step executes.

Script loops are useful when iteration logic must be calculated dynamically.

Example use cases include:

• API pagination 
• Retry logic 
• Iterating through generated values 
• Executing a flow a fixed number of times

Configure Script Loop

1. Set Loop Type = Script. 
2. Enter JavaScript or Python code in Loop Script.

Script Loop Behavior

The step continues executing while the script returns a non-null value.

Example:

// this flow will be executed while the value of NEXT_MARKER is not null
var nextCursorMark = com.toolsverse.config.SystemConfig
    .instance()
    .getProperties()
    .get("NEXT_MARKER");
value = !com.toolsverse.util.Utils.isNothing(nextCursorMark) ? nextCursorMark : null;

When value becomes null, the loop stops.

Execute Flow a Fixed Number of Times

A Script Loop can also run a step a fixed number of times.

Example:

// this flow will be executed 40 times
value = evals <= 40 ? evals : null;

Variables Available in Script Loop

The following variables can be used inside the loop script.

Programmatically Set Variables for Each Iteration

You can set global and flow variables dynamically using loopKeyValues.

Example:

loopKeyValues.put('VAR_NAME', 'var_value');

These variables become available during the iteration.

Database Loop

Database loop executes the step once for each row returned by a SQL query.

The query is executed before the loop begins. Each row becomes a separate iteration.

This is commonly used when a workflow needs to process database-driven tasks such as:

• Processing a list of records 
• Executing operations for each tenant or customer 
• Driving workflow execution from database metadata

Configure Database Loop

1. Set Loop Type = SQL. 
2. Select the database Connection used for the loop query. 
3. Enter the SQL query in Loop Script.

If the query is actually a script (for example PL/SQL or T-SQL), enable Execute as Script.

The script must return a cursor.

Variables Set by Database Loop

Each column returned by the query automatically added as a global and flow variable in each iteration of the file loop.

Example Database Loop

Loop Script

select roleid, name from role

The step executes once for each returned record.

Automatically set variables

roleid
name

These variables can be referenced as {roleid} and {name} and used in:

• transformation 
• fields connection properties 
• source queries 
• destination queries

Transformation

Debugging Database Loops

In a Database Loop, each column returned by the SQL query becomes a variable that can be referencedusing {variable} syntax.

If a column value is null or empty, Etlworks cannot replace the corresponding {variable} token. This may cause the step to fail.

When this happens, Etlworks writes a warning to the workflow log:

WARNING: scenario 'scenario name' will be executed with an empty value for the variable {variable name}

If a database loop fails unexpectedly, check the workflow log to verify that the SQL query returns valid values for all variables used in {tokens}.

File Wildcard Loop

A wildcard loop iterates over files that match a specified file pattern.

Each matching file triggers a separate execution of the step.

Typical use cases include:

• Processing files in a directory 
• Processing files uploaded during the day 
• Batch processing of data feeds

Configure File Loop

1. Set Loop Type = Files by wildcard. 
2. Select the file Connection. 
3. Enter the wildcard pattern in File path.

File Path Modifiers

The File Path parameter supports modifiers for filtering and sorting files.

File Selection Modifiers

Exclude specific files

*.csv;-test1.csv,test2.csv

Include only specific files

*.csv;+test1.csv,test2.csv

Process folders matching a wildcard

<*test>

Skip previously processed files

To avoid reprocessing the same files, use the [path] modifier.

The system tracks processed files using filename and modification timestamp.

Optional parameters:

[path,cache]
[path,ttl]
[path,cache,ttl]

Example:

[*.csv,my_cache,86400000]

This configuration skips files processed during the last 24 hours.

Sorting Modifiers

Sorting modifiers override connection-level sorting.

;oldest
;newest
;ascending
;descending
;largest
;smallest

Variables Set by File Loop

The following global and Flow variables are automatically set in each iteration of the file loop:

• loop_file_name: the name of the file processed by the loop. The name includes the extension but does not include the path.
• loop_base_file_name: the base name of the file processed by the loop. The name does not include the extension and the path.
• loop_file_id: the id of the file. Currently file id is only set by Google Drive connector. If there is no file id loop_file_id =loop_file_name. 
• evals: 0-based number of loop iteration.

Example File Loop

File path:

*.csv

Transformation

File Loop with Filter

File loops can also be implemented using a Script Loop that filters files programmatically.

This allows advanced scenarios such as:

• processing files created today 
• processing only large files 
• splitting files into buckets

Step 1. Create a new script flow.

Step 2. Add a named source connection to the flow. Use the connection that contains the source files. The name that you assign to the connection is going to be used in the next step. Assign any format, it doesn't matter which one but format is a required field.

Step 3. Add the JavaScript or Python code to filter or split files in buckets.

Available file attributes:

• file.getName - the name of the file with an extension but without the folder.
• file.getSize - the size of the file in bytes.
• file.getPath - the full path.
• file.getLastModified - the timestamp in Unix epoch when the file was modified.

Filter examples

Here are several examples:

Files created or updated today

importPackage(java.time);

// Get the list of the files in the folder matching the wildcard
// For the connection, use the same name as you assigned 
// to the named connection is step 2
var list = com.toolsverse.etl.core.task.common.FileManagerTask.list(etlConfig, 
'source', '*.*');

// create buckets
var newFiles = new java.util.ArrayList();

// split files in buckets
if (list != null) { 
    for each (var file in list) {
        if (LocalDate.now().equals(Instant.ofEpochMilli(file.getLastModified()).atZone(ZoneId.systemDefault()).toLocalDate())) {
            newFiles.add(file);
         } 
    } 
}

// add named buckets to the common object storage
etlConfig.setValue('new_files', newFiles);

Large files

// Get the list of the files in the folder matching the wildcard
// For the connection, use the same name as you assigned 
// to the named connection is step 2
var list = com.toolsverse.etl.core.task.common.FileManagerTask.list(etlConfig, 
'source', '*.*');

// create buckets
var largeFiles = new java.util.ArrayList();

// split files in buckets
if (list != null) { 
    for each (var file in list) {
         if (file.getSize()>=10000) {
              largeFiles.add(file);
         }
    }
}

etlConfig.setValue('large_files', largeFiles);

Step 4. Create a new nested flow

Step 5. Add flows created in step 1 as a first step.

Step 6. Add flow which will be processing the files as a step 2.

Step 7. Configure scripting loop for the flow which will be processing the files in a bucket:

Click the editor icon:

Configure the Loop script

Loop script for the File Loop with Filter

Here is a template. If you have one bucket the only parameter that you need to modify is the bucket name:

importPackage(com.toolsverse.util);

// gets the files in bucket generated after applying the filter
var files = etlConfig.getValue('same bucket name as in step 3');

if (files != null && !files.isEmpty()) {
   // get the file name and id
   var fileName = FilenameUtils.getName(files.get(0).getName());
   var baseName = FilenameUtils.getBaseName(fileName);
   var fileId = files.get(0).getId();
   fileId = fileId == null ? fileName : fileId;

   // remove the file from the bucket
   files.remove(0);

   // set loop variables
   loopKeyValues.put('loop_file_name', fileName);
   loopKeyValues.put('loop_base_file_name', baseName);
   loopKeyValues.put('loop_file_id', fileId);
   loopKeyValues.put('evals', Utils.makeString(evals));

   // continue the loop if the files in the bucket after removing the first file
   value = 'true';
} else {
   // stop the loop
   value = null;
}

Global and Flow variables set by the file loop with filter

The following global and Flow variables are automatically set in each iteration of the file loop:

• loop_file_name: the name of the file processed by the loop. The name includes the extension but does not include the path.
• loop_base_file_name: the base name of the file processed by the loop. The name does not include the extension and the path.
• loop_file_id: the id of the file. Currently file id is only set by Google Drive connector. If there is no file id loop_file_id =loop_file_name. 
• evals: 0-based number of loop iteration.

Example of the flow executed by File Loop with Filter

This flow converts all.csv files in a source folder after applying a filter into the.json files with the same name: 

Parallel Loop Execution

Loop iterations can run in parallel threads.

Parallel loops allow multiple iterations of the same step to execute concurrently, which can significantly improve performance when iterations are independent.

To enable parallel execution, set the Loop Threads parameter to a value greater than 1.

This parameter defines the maximum number of threads that can execute loop iterations simultaneously.

When to Use Parallel Loops

Parallel loops are useful when each iteration performs an independent operation.

Typical use cases include:

• making a large number of HTTP requests
• loading multiple files into a database 
• performing repeatable data processing tasks across many records

Parallel loops should only be used when iterations do not depend on each other.

How Parallel Loops Work

When a loop executes, each iteration becomes a task.

If Loop Threads is greater than 1, Etlworks creates a thread pool with the configured maximum number of threads.

Example:

• The loop has 100 iterations 
• Loop Threads = 10

Execution behavior:

1. The system creates a pool of 10 threads. 
2. Each iteration is submitted as a task. 
3. If a thread is available, the iteration starts immediately. 
4. If all threads are busy, remaining iterations wait until a thread becomes available.

This continues until all iterations complete.

Variables in Parallel Loops

When using loops such as Database Loops, the system automatically sets variables for each iteration.

These variables are thread-safe, meaning multiple parallel threads can use them safely.

For example, a parallel database loop 

select user_id from users 

might return:

user_id 

The step can use the value:

https://host.api/user/{user_id}

Each parallel iteration receives its own value of {user_id}.

Setting Variables in Parallel Loops

If you set variables manually in JavaScript during loop execution, you should use the thread-safe container for global variables.

com.toolsverse.config.SystemConfig.instance().addPropertyInCurrentEtlThread(key, value);

This ensures that variables set in one thread do not interfere with other parallel iterations.

Retry Loop Iterations

Loops support automatic retries when an iteration fails.

Configuration parameters:

• Loop retries – number of retries per iteration 
• Retries delay – delay between retries (milliseconds)

Stop the loop after the fixed number of iterations

You can configure any loop to stop after the fixed number of iterations by setting the parameter Maximum number of iterations. The default is 100000.

Stop the loop if it exceeds a certain time limit

You can set a loop to stop running if it exceeds a certain time limit by using the Loop timeout parameter (in milliseconds). It is possible to use {global variable} to set the value of this parameter.

You can choose to stop the loop or raise an exception in case of timeout by using the On loop timeout parameter. The default setting is Stop loop. 

If the loop is configured to run iterations in parallel and the Loop timeout is set, the flow will always raise an exception if the loop runs longer than expected, regardless of the On loop timeout setting. 

If the maximum number of iterations and the loop timeout are both set, then the maximum number of iterations is checked first, followed by the loop timeout.

Stop the loop programmatically

A loop continues running as long as:

• There are records in the result set returned by an SQL loop query.

• There are files to process in a file-based loop.

• The JavaScript or Python condition returns a non-null value.

In some cases, you may need to stop the loop early based on custom conditions.

Using setRequestStop(true) to Stop a Loop

You can programmatically stop a loop from within JavaScript or Python code by calling setRequestStop(true). This method is thread-safe and ensures the loop stops cleanly.

When etlConfig is Available

Variable etlConfig is directly accessible within:

• Mapping transformations

• Scripting transformations

• The dedicated Scripting Flow

To stop a specific loop:

etlConfig.setRequestStop(true, 'flow name'); 

Where:

• flow name is the name of the Flow executing inside the loop.

• If flow name is null or this parameter is not provided, this command stops all active loops, including parallel and nested loops.

If etlConfig is not available

Preprocessors (e.g., Format Preprocessor, HTTP Connection Preprocessor, etc.) do not have access to etlConfig. In these cases, use:

com.toolsverse.config.SystemConfig.instance().getEtlThreadContext().getData().setRequestStop(true, 'flow name');

• flow name specifies the Flow to stop.

• If flow name is null or this parameter is not provided, all loops will be stopped.

Summary

Looping allows workflows to execute a step multiple times based on defined logic.

Key points:

• Looping is configured at the workflow step level. 
• Multiple loop types are supported. 
• Each iteration runs the same step. 
• Loop limits and thread controls help manage execution behavior.

Looping enables workflows to process dynamic sets of data and automate repetitive tasks.

Read Next

• Parallel Execution in Workflows