Startup
Business
Enterprise
On-Premise
Add-on

Overview

The Integration Agent can run Flows created in Etlworks. The Flows can be executed manually, scheduled to run automatically, and stopped.

Process

Flows can be added, removed, and scheduled when the specific Integration Agent is running. The communication session is always initiated by the Agent as we assume that Etlworks cannot access the Agent which is running behind the firewall.

Here's how you can add Flows to run in the remote Integration Agent:

Step 1. In Etlworks, open the Agents window and select the already created Integration Agent.

Step 2. Add previously created Flow(s) to the Agent.

Step 3. Select one of the available schedule types.

Step 4. Configure the schedule if the Schedule type is not Manual

Execute flow every n seconds

Execute flow periodically using a cron-based schedule

Configure Auto-Retry on Failure

When creating or modifying a Schedule, it is possible to configure auto-retries. When the auto-retry is enabled and the Flow execution fails, the scheduler will attempt to execute the Flow again until the Flow is successfully executed or the configurable number of times, whichever comes first.

Step 1. Enable Retry of failure.

Step 2. Select the number of retries (5 is the maximum).

Step 3. Select Delay between retries (1-10 minutes).

Handling Specific Exceptions in Auto-Retry

In some cases, you may want to retry only on specific errors or exclude certain exceptions from being retried. You can configure this using the following fields:

• Retry Exception Mask – Define a list of exception messages or patterns that should trigger a retry. If this field is populated, the flow will only retry when the error message matches one of the specified values. Use semicolon (;) as a separator.

• Retry Exception Mask (Exclude) – Define exceptions that should not trigger a retry. If an error matches this list, the auto-retry mechanism will skip retrying for that failure. Use semicolon (;) as a separator.

Optionally configure automatic activation or deactivation of the schedule

If you want to auto activate flow schedule, make sure that flow is deactivated before saving (deselect checkbox on the left side of the flow). When setting "Deactivate on date" make sure that flow is active before saving.

Note: Auto activation and deactivation of agent flow schedule happens on the server side, not the individual agent.

The activation and deactivation occurs at midnight in the server’s timezone on the specified date via Auto enable/disable schedules maintenance task execution.
The server timezone is determined by the host operating system running Etlworks. On Linux systems, it is typically UTC unless manually changed.
Maintenance task schedule can be adjusted in the maintenance task management UI, however date matching during the task execution will be calculated in the server timezone, make sure to adjust with time difference in mind.

Step 5. Optionally provide a description for the flow.

The description is an additional attribute available when adding flows to the Agent. It can be used as an informational field only but it is required if you wish to schedule the same flow multiple times with different flow variables. Read about parametrization.

Step 6. Optionally add flow parameters.

Click the Editor icon next to the flow status link.

then add the Flow parameters that will be passed to the Flow at run time. The parameters will be added as global and Flow variables.

Read about parametrization.

Auto-population for parameter keys and values

You can configure auto-population for parameter keys and values using global settings managed by a super admin. This feature enables dropdown suggestions based on predefined or previously entered parameters, making it easier to maintain consistency across flows.

Learn more about configuring auto-suggestions for flow parameters

Entering Empty String Values

When entering parameters for a flow in the format key=value, the UI does not accept an empty string ("") as a valid value. To pass an empty string as the value of a specific parameter, you must enter it using the Unicode representation for an empty character: \u200B. This allows the system to recognize and process the parameter with an empty string as its value.

Entering Secure Parameters

Pass secure parameters such as passwords in theSecure parametersbox:

Step 7. Optionally configure email notifications.

When creating or modifying a Schedule, it is possible to configure the email notifications.

Click Email button

Configure Notifications

The following notifications are available:

Send email on success

This notification is sent when the Flow is successfully executed.

Step 1. Enable On success.

Step 2. Use Emails to configure the email address(es) that will be receiving a notification.

Send email on failure

This notification is sent when the Flow is either failed or canceled.

Step 1. Enable On failure.

Step 2. Use Emails to configure the email address(es) that will be receiving a notification.

Truncate Exception Details in Error Notifications

By default, when a scheduled flow fails, the full exception stack trace is included in the email notification. To make these notifications easier to read and reduce clutter, you can enable the global setting “Truncate exception in failed flow email notification” under Settings → Flows. When this option is enabled, only the error description and message will be included in the email, without the full stack trace. This setting is disabled by default and must be manually turned on. Once enabled, it applies to all on-error notifications sent by scheduled flows running on both the host instance and Integration Agents.

Send an email if Flow execution is taking longer than expected

This notification is sent when the Flow execution is taking longer than expected. The notification will be sent one time when the actual run time exceeds the configured threshold.

Step 1. Use Expected flow execution time to specify the expected Flow execution time in minutes.

Step 2. Use Emails to configure the email address(es) that will be receiving a notification.

Send an email and stop the flow if Flow execution is taking longer than maximum allowed time in minutes

This notification is sent and the flow is stopped when the Flow execution is taking longer than maximum allowed time in minutes. The notification will be sent one time when the actual run time exceeds the configured threshold.

Step 1. Use Stop flow after time to specify the maximum allowed Flow execution time in minutes.

Step 2. Use Emails to configure the email address(es) that will be receiving a notification.

Note: The stop-flow signal will be send to the agent but it is not guaranteed that the flow will be actually stopped. There are cases when the flow is stuck when executing not cancellable IO operation (for example, opening the database connection with no configured timeout). In this cases the flow will continue running until it is finally able to stop or the agent is rebooted.

Include metrics in email

If enabled, the notification email will include detailed execution metrics captured by the flow. These are the same metrics shown in the Flow Execution Dashboard, such as:

Start and end time
Duration
Number of records processed
Number of files processed
Status and errors (if any)

The metrics are included in the email as an HTML table for better readability.

This option is helpful when you want:

A quick summary of execution results without logging into the dashboard
To log metrics externally (e.g., in a ticketing system or notification archive)
To troubleshoot failed or inconsistent runs by inspecting flow parameters used during execution

Include Flow Parameters in Email

If enabled, the parameters defined for the schedule will be included in the email as an HTML table.

This option is helpful when you want:

A quick summary of execution results without logging into the dashboard
To log parameters externally (e.g., in a ticketing system or notification archive)
To troubleshoot failed or inconsistent runs by inspecting flow parameters used during execution

Filter flows by name, description, and tags

It is possible to filter flows, associated with the agent by

1. Tags.

2. Name and Description

Navigate to Flow from Agent

User can quickly navigate to the flow from the Agent dashboard by clicking the link icon.

Clicking the icon takes you directly to the Flow editor for that specific flow.
You can return to the Agent view at any time using the browser’s Back button.

Run the Flow manually

You can request to run any flow added to the Etlworks Agent. If the Flow is not already running it will be started by the Agent.

The Run button is disabled if the Agent is offline.

Running flow scheduled on Agent on the host instance

When manually triggering a flow in Etlworks, the system automatically detects if the flow is configured to execute on an Integration Agent rather than the main instance. If a flow is scheduled to run on an agent, Etlworks provides a warning, as shown in the screenshot, to prevent accidental execution on the main instance.

Automatic Detection:

• Etlworks checks the flow configuration to determine if it is assigned to one or more Integration Agents.

Warning Message:

• If the flow is configured to run on an agent, a message will appear, advising against running the flow directly on the main instance. The warning includes the names of the assigned agents (e.g., “Dev agent”).

User Options:

• Cancel: Stops the flow from executing, allowing the user to switch to the correct agent for execution.

• Run anyway: Allows the flow to execute on the main instance despite the recommendation.

Stop the running Flow

You can request to stop any running flow added to the Etlworks Agent. If the Flow is running it will be stopped by the Agent.

The Stop button is disabled if the Agent is offline.

Parametrization

It is possible to add flow variables to the flow executed by the Integration Agent. Flow variables can be referenced as {VARNAME} anywhere in the code.

Examples:

Parameterization of the source and destination queries.
Parametrization of the command-line (shell) scripts.

Flow variables can be converted to global variables.

Read how to use global variables.

Read how to add flow variables to the flow executed by Integration Agent.

Duplicating Flows Attached to Agents

The platform provides an easy way to duplicate existing flows that are attached to agents, allowing you to quickly replicate and reuse configurations without manually recreating them. As shown in the screenshot, the Duplicate button (represented by two overlapping squares) is available next to each flow.

Duplicate Flow: When clicked, the system creates a copy of the selected flow with all its parameters, schedules, and settings intact. This is particularly useful when you need to set up similar ETL flows or modify an existing flow for a new use case without starting from scratch. The duplicated flow can then be edited and customized independently of the original.

Monitor the Flow

Flow Statistics dashboard

Each time the Flow is executed by the Integration Agent (manually or automatically by the scheduler) it reports stats, metrics, and optionally console log for that specific run back to Etlworks.

Important: Unlike flows running in the cloud Etlworks, the tasks and the log for the flows executed by the Agent are not updated in real-time.

To access the history of runs as well as starts, metrics, and log for each run click the status link next to the flow added to the Integration Agent.

The standard Etlworks Flow Statistics dashboard will be displayed.

Flow Execution Log

Important: Read how to enable and configure log upload.

When the log is available it can be viewed and downloaded from the Flow Statistics dashboard.

Articles in this section

Schedule and Run Flows in the Integration Agent

Overview

Process

Step 1. In Etlworks, open the Agents window and select the already created Integration Agent.

Step 2. Add previously created Flow(s) to the Agent.

Step 3. Select one of the available schedule types.