Articles in this section

Monitor Flow Execution

  • Updated
Follow

Overview

Etlworks provides multiple ways to monitor Flow execution in real time and review past runs.

You can:

  • Monitor running Flows 
  • Review execution history and results 
  • Inspect logs, errors, and metrics 
  • View running tasks and scripts 
  • Troubleshoot issues using SQL and debug information

All monitoring starts from the Flow Statistics dashboard.

Flow status

Each Flow execution has a status displayed in the Flows screen:

  • Running – Flow is currently executing 
  • Success – completed successfully 
  • Error – failed 
  • Warning – completed with issues 
  • Canceled – stopped manually

Click the Status link to open detailed execution information.

Flow Statistics dashboard

The Flow Statistics dashboard is the central place for monitoring and troubleshooting.

You can access it from:

1. Flows screen (Status column) 

2. Schedules screen 

3. Flow Builder 

4. Agents screen

Execution details

  • Status 
  • Duration 
  • Execution ID 
  • Trigger source (user, schedule, agent, listener)

Execution history

  • Displays a list of past executions:
  • Last 50 runs by default (can load more) 
  • Shows who started the Flow and when 
  • Click any entry to view full details

mceclip2.png

Direct links to executions

Each execution has a unique Execution ID.

You can open or share a specific execution using:

/#/app/flows/{flowId}/statistics/{executionId}

or

/#/app/schedules/{scheduleId}/statistics/{executionId}

Visual indicator for retries

Flow history visually indicates retries triggered by the scheduler or integration agent, including a parent process ID. This helps differentiate retries from regular executions.

Filtering execution history

You can filter Flow execution history using the filter panel. To open it, click the small filter icon next to the History label.

Filtering Options

Once the filter panel is open, you can narrow down the Flow execution history using the following options:

Date range (From / To)

  • From date: Shows executions starting from 00:00 (midnight) of the selected date, in the Admin or tenant timezone.

  • To date: Shows executions up to the end of the selected date, in the Admin or tenant timezone.

    Together, these restrict results to a calendar date range.

Time range (From time / To time)

  • From time: Shows executions that happened at or after the specified time of day, in the Admin or tenant timezone. If more than one day of executions is displayed, this filter applies to each day individually.
  • To time: Shows executions that happened before the specified time of day, in the Admin or tenant timezone. If more than one day of executions is displayed, this filter applies to each day individually.
  • Both filters are independent of the From date / To date filters.

Status

Filter executions by status:

  • running
  • success
  • warning
  • error
  • canceled

Trigger type

Filter by how the execution was triggered:

  • schedule
  • on-demand
  • listener
  • agent

Started by

Identifies who or what initiated the execution:

  • If started manually → user login.
  • If started by schedule → schedule name.
  • If executed on an agent → agent name.

Description

Filter by the description associated with a Flow executed on an agent.

Metrics

Show executions whose Metrics or File Metrics contain the provided text.

Execution ID

Filter by a unique execution ID associated with a specific Flow run.

Quick filtering from charts

You can also click directly on a slice of the pie chart (right-hand panel) to quickly filter executions by status (e.g., show only Errors).

Metrics

Metrics provide insight into how data is processed.

Transformation status

success Success The transformation was successfully executed.
mceclip5.png Warning The transformation was executed with a warning or not executed at all. Check the Warning column for more details.
error Error The transformation was executed with an error. Click the error icon for more details. 

Records Metrics

Shows:

  • Start and End timestamps
  • Records extracted 
  • Records loaded 
  • Transformation status: Success, Warning, Error
  • Last Recorded High Watermartk
  • Exception

File Metrics

  • Shows file operations such as:
  • Copy 
  • Move 
  • Delete 
  • Rename

Nested Flows

Metrics are grouped by Flow when using nested Flows.

nested-metrics.png

Advanced metrics options

Recording all steps when executing nested flow

By default the flow only reports Records Metrics (metrics for processed source-to-destination ETL transformations) and File Metrics (proceeded files and folders). If you want the nested flow to report all executed steps, for example scripts, calling HTTP endpoints, etc., enable Track metrics for all types of flows for the main nested flow.

When this option is enabled the Records Metrics tab includes all flows, executed as a part of the main nested flow.

Independently track metrics for each loop iteration

By default the flow aggregates Records Metrics across all loop iterations. To track metrics for each loop iteration independently enable Track metrics for loops.

When this option is enabled the dashboard displays metrics for each loop iteration:

Filtering and Exporting Metrics to CSV

The Flow Stats Dashboard allows you to filter both Flow Metrics and File Metrics and export metrics to a CSV file.

Metrics can be filtered using the search field at the top of metrics panel. Filtering applies to all visible attributes, such as flow name, step name, file name, status, timestamps, duration, and record counts. The filter is applied immediately and updates the displayed results in real time.

Metrics can be exported with or without an active filter. To export metrics, click the Download CSV button in the top-right corner of the dashboard.

If a filter is applied, the exported CSV file contains only the metrics that match the current filter. If no filter is applied, the CSV file contains all available metrics currently displayed in the selected tab.

The exported file name depends on the active tab:

  • From the Metrics tab, the file is named metrics-<auditId>.csv
  • From the File Metrics tab, the file is named file-metrics-<auditId>.csv

Monitor running Flows (real-time)

To monitor a Flow while it is running:

  1. Open the Flows screen 
  2. Click the Running status 
  3. Open the Flow Statistics dashboard

From there you can:

  • View live metrics 
  • Open logs 
  • Monitor tasks

Flow logs

Flow logs provide detailed execution output.

View logs

  1. Open the Flow Statistics dashboard 
  2. Click Console View

This opens a live (tail) view of the log.

Download logs

  1. Open the Flow Statistics dashboard 
  2. Click Console Download

Read how to download Flow log using API.

Notes

  • Logs are available for running and completed Flows 
  • Logs are stored in the file system 
  • Requires Capture console log enabled

Monitor running tasks and scripts

Etlworks tracks currently executing operations such as:

  • SQL scripts JavaScript and Python code 
  • Data extraction and loading 
  • File operations
  • CDC and Streaming

Monitor tasks for a specific Flow

  1. Open the Flow Statistics dashboard 
  2. Click View Running Tasks 
  3. Optionally enable Auto-refresh

Monitor tasks across all Flows

  1. Open the Statistics screen 
  2. Expand Running flow tasks

Notes

  • Tasks are ordered by start time and duration 
  • Easier to navigate than logs 
  • Inline transformations (filtering/validation) are not tracked

View last executed SQL

When a Flow fails while writing to a database, you can inspect the last executed SQL.

Steps

  1. Open the Flow Statistics dashboard 
  2. Click View Last Executed SQL

Debug information

You can add custom debug information programmatically using JavaScript:

This information is available in:

  • Flow logs 
  • Monitoring views

Summary

  • Use the Flow Statistics dashboard as the main entry point 
  • Monitor running Flows for real-time visibility 
  • Use execution history to review past runs 
  • Use logs, tasks, and SQL for troubleshooting 
  • Use metrics to understand data processing behavior

Related articles