Articles in this section

Run and Control Flows

  • Updated
Follow

Overview

Flows in Etlworks can be executed in several ways depending on your use case.

The most common execution methods include:

  • Manually from the UI (for testing and ad-hoc runs) 
  • Automatically using schedules 
  • Triggered by API calls 
  • Triggered by CLI commands 
  • Triggered by listeners (HTTP endpoints)
  • Executed by Integration Agent

This article focuses on how to manually run, stop, suspend, and resume Flows when you need direct control.

For other execution methods, refer to:

Parallel execution and multiple Flow instances

Starting with version 9.3.1, Etlworks supports running multiple instances of the same Flow in parallel, depending on how the Flow is triggered.

This behavior varies by execution type.

Manual execution

When a Flow is triggered manually (from the UI):

  • Only one instance of the Flow can run at a time 
  • This includes executions started by schedules

If an instance is already running, a new execution will not start.

Scheduler execution

When a Flow is triggered by the Scheduler:

  • Multiple instances of the same Flow can run in parallel 
  • Each execution is independent
  • You can monitor and control each instance separately in the Flow Statistics dashboard

This is useful for high-frequency schedules or long-running Flows.

API and CLI execution

When a Flow is triggered via API or CLI:

  • By default, only one instance can run at a time 
  • You can enable parallel execution by passing a parameter in the request

Additional capabilities:

  • Fire and forget execution (asynchronous) 
  • Wait until finished (synchronous)

This allows external systems to control concurrency and execution behavior.

Listener-based execution (HTTP endpoints)

When a Flow is triggered by a listener (GET, POST, PUT, DELETE):

  • Multiple instances can run in parallel

Additional behavior for POST and PUT listeners:

  • Support synchronous execution (wait for response) 
  • Support throttling 
  • Support queued execution

This makes listener-based Flows suitable for real-time and high-throughput use cases.

Agent-based execution

When Flows are executed on an Integration Agent:

  • The same Flow can be scheduled and run in parallel 
  • Each instance must have a different description 
  • Each execution is treated as a separate instance and can be monitored and controlled independently

This allows running multiple agent-based jobs from the same Flow definition without conflicts.

Parallel Execution Notes

  • Each execution is treated as a separate instance 
  • You can stop, monitor, and analyze each instance independently
  • Parallel execution may increase load on the instance or Integration Agent

Manually run a Flow

You can run one or multiple Flows at any time from the UI.

Run from the Flows screen

Run a single Flow

  1. Open the Flows screen. 
  2. Locate the Flow. 
  3. Click Run flow. 
  4. Confirm execution.

Run multiple Flows

  1. Open the Flows screen. 
  2. Select one or more Flows. 
  3. Right click to open Context Menu
  4. Click Run selected flows. 
  5. Confirm execution.

Notes

  • When multiple Flows are selected, they are executed independently.
  •  Each Flow runs using its own configuration (including assigned Integration Agent, if any). 
  • A confirmation dialog is displayed before execution to prevent accidental runs.

Run from the Flow Builder

  1. Open the Flow in the Flow Builder. 
  2. Click Run. 
  3. If the Flow was modified, confirm saving. 
  4. Confirm execution.

Restrictions

You cannot manually run a Flow if:

  • The Flow is already running 
  • The Flow is event-driven (triggered by an API or listener)

Confirmation dialog

When you click Run, Etlworks displays a confirmation dialog to prevent accidental execution.

If the Flow is configured to run on an Integration Agent, a warning is shown to avoid running it on the main instance.

Options:

  • Cancel: abort execution 
  • Run anyway: execute immediately

Run options

Capture console log

Enabled by default. Displays execution logs in real time and is required for debugging.

mceclip2.png

Extra Log Step

Adds additional log checkpoints after a specified number of processed records.

  • Use for large datasets (typically > 1M records) 
  • May impact performance 
  • Requires console log enabled
mceclip3.png

Flow parameters

  • You can pass parameters at runtime. 
  • These are available as global and Flow variables.

Recommendations:

  • Use uppercase keys 
  • Avoid spaces
mceclip5.png

Secure parameters

Use the Secure parameters field for sensitive values such as passwords.

mceclip0.png

Parameter auto-suggestions

You can configure auto-population for parameter keys and values using global settings managed by a super admin. 

This feature enables dropdown suggestions based on predefined or previously entered parameters, making it easier to maintain consistency across flows.

Empty values

To pass an empty string, use:

\u200B

Execution on Integration Agent

If a Flow is configured to run on an Integration Agent:

  • Etlworks detects it automatically 
  • A warning is displayed before execution 
  • You can choose to cancel or run anyway

Notifications and status

After execution starts:

mceclip6.png

When a flow is executed directly from the Flow Builder, you are automatically redirected to the Flow Statistics dashboard after clicking Run flow.

Stop a running Flow

You can request to stop a Flow at any time.

Important: Stopping a Flow is not always immediate. If the Flow is executing a non-cancellable operation (for example, a database call without timeout), it may continue running until the operation completes or the instance is restarted.

Stop from the Flows screen

  1. Open Flows 
  2. Click Stop next to the running Flow

Notes:

  • Only available when Flow status is Running 
  • If multiple instances are running, only the latest instance is stopped

Stop from Flow Statistics

  • Open the Flow Statistics dashboard 
  • Select the running instance 
  • Click Stop

This method allows stopping a specific instance when multiple executions are running.

Flow status after stop

Status changes from Running → Canceled 

Final state: 

  • Error (regular Flows) 
  • Success (streaming Flows) 

Important: Complete stop may take up to a few minutes.

Suspend and resume Flow execution

You can temporarily prevent all Flows from starting.

This is useful during maintenance, upgrades, or controlled downtime.

How it works

  • All schedules remain active 
  • No new Flows will start 
  • Applies to: 
    • Scheduled execution 
    • Manual execution 
    • API-triggered execution 

Suspend from the UI

1. Go to Settings → Flows 

2. Click Suspend flow executions 

3. Optionally enable: Stop currently running flows 

A Suspended label will appear in the UI.

Resume from the UI

Click Resume flow executions to restore normal operation.

Behavior

The suspend flag is cleared automatically after restart In multi-node environments, it applies to all nodes

APIs and CLI

All operations described above can also be performed programmatically.

Available capabilities include:

  • Run a Flow 
  • Stop a Flow 
  • Suspend execution 
  • Resume execution

Use:

  • REST API endpoints 
  • Etlworks CLI

Refer to:

Summary

  • Use manual execution for testing and ad-hoc runs 
  • Use stop to terminate running Flows (when possible) 
  • Use suspend/resume to control execution at the system level 
  • Use APIs and CLI for automation and integration

Related articles