Overview
Workflows often interact with external systems such as APIs, databases, and file storage. These systems may occasionally fail due to temporary conditions such as network interruptions, rate limits, or database locks.
To make workflows resilient to these transient failures, Etlworks allows you to configure automatic retry logic for workflow steps.
Retry behavior is configured using the loop retry settings.
Retries apply to any loop iteration. If a step runs inside a loop and an iteration fails, that iteration can be retried automatically according to the configured retry parameters.
For steps that are not already executed in a loop, the retry behavior can be enabled by configuring a single-iteration loop.
How Retry Works
Retries are part of the loop execution configuration.
When retry parameters are configured:
- If a step fails during execution, the system retries the current loop iteration.
- The retry occurs according to the configured retry count and delay.
- If all retries fail, the step fails normally and the workflow continues according to its failure handling configuration.
This behavior applies to:
- steps inside loops with multiple iterations
- steps configured with a single-iteration loop for retry purposes
Retry in Multi-Iteration Loops
When a step executes inside a loop with multiple iterations, retry logic applies to each iteration independently.
If an error occurs during an iteration:
- The iteration fails.
- Etlworks retries that same iteration according to the retry configuration.
- If the retry succeeds, the loop continues to the next iteration.
Example use cases include:
- retrying API calls during pagination
- retrying file processing steps
- retrying database operations driven by SQL loops
Configure Retry for a Loop
- Select the step in the workflow.
- Open Child Flow parameters. Scroll to the Loop section.
- Configure the following settings.
- Loop retries – number of retry attempts
- Retries delay – delay between retries (milliseconds)
Retry for Regular Workflow Steps
Steps that are not normally executed in a loop can still use retry logic.
This is implemented using a single-iteration loop.
The loop runs only once under normal conditions, but the retry configuration allows the step to be re-executed if it fails.
Configure Retry for a Step
To configure retry for a regular workflow step which not running in a loop:
- Select the step in the workflow.
- Open Child Flow parameters. Scroll to the Loop section.
- Configure the following settings.
Loop Script
trueThis creates an unconditional loop.
Maximum Number of Iterations
Maximum number of iterations = 1Retry Parameters
Configure retry behavior:
- Loop retries – number of retry attempts
- Retries delay – delay between retries (milliseconds)
Example configuration:
Loop retries = 3
Retries delay = 5000
Execution behavior:
- The step executes once.
- If the step fails, Etlworks retries it up to 3 additional times.
- Each retry waits 5 seconds before re-executing the step.
Designing Resilient Workflows
Retries should be combined with other workflow mechanisms such as:
- conditional execution
- loop timeouts
- error pipelines
- exception logging
Best practices:
- use retries only for transient failures
- keep retry counts reasonable
- ensure retry operations are safe to repeat
Proper retry configuration improves reliability while keeping workflows predictable and maintainable.
Summary
Retries in Etlworks are configured using loop retry settings.
Key points:
- retry settings apply to each loop iteration iterations
- can be retried automatically if they fail regular steps
- can use retry by configuring a single-iteration loop
- retry behavior is controlled by Loop retries and Retries delay
This mechanism allows workflows to automatically recover from temporary failures while maintaining a simple and predictable execution model.