Overview

A Data Integration Flow is an ETL or automation scenario that can be executed manually, on a schedule, or triggered by an API call.

Read about available Flow types.

Create a Flow

Step 1. Select a Flow Type

To create a new Flow:

• Navigate to the Flows page.

• Click Add flow to open the Flow Type gallery, where you can select any supported Flow type, such as: "database to database", "file to file", "web service to database", and others

Find the right Flow type

Filter By Category

Filter Flow types by category using the list on the left.

Filter By Name or Use Case

You can quickly look up a Flow by entering a name or pattern like database to file, web service to database, etc. Only Flow types matching the string will be displayed.

Oy you can use prompt describing what you want to achieve (e.g., “I want to send emails with attachments” or “Stream data from PostgreSQL to MongoDB”).

The system will intelligently suggest the most appropriate Flow type(s).

Read about AI-Augumented Search.

Other Options

Alternatively, click the dropdown arrow next to Add flow to access additional options:

• Add workflow (nested flow) to create a Workflow container for orchestrating multiple steps. See: Complex Workflow

• Create flow from template to start from one of 4,000+ ready-to-use, preconfigured point-to-point templates. See: Integration and Automation Templates

Step 2. Add and Configure Transformations

After selecting the Flow type:

• Define one or more Transformations by clicking +.

• Each Transformation typically requires:

  • FROM Connection, Format, and object name (e.g., file or table)

  • TO Connection, Format, and object name

Transformations can be rearranged, modified, or deleted using drag-and-drop.

Step 3. Define Mapping and Parameters

For each Transformation:

• Click CONFIGURE to open the Mapping Editor.

• Create per-field mapping:

  • Use Create Mapping to auto-generate field pairs

  • Or add fields manually using +

You can update and rearrange mappings at any time.

Step 4. Save the Flow

Click Save to store the Flow.

Manage Flows

All Flows are listed on the Flows page and can be filtered by name, type, or tag.

You can:

• Open and edit Flows

• Run flows manually

• Duplicate, delete, or schedule them

• Export or import between environments

Flow Views: Flat and Hierarchical

The Flows page supports two complementary ways to browse and manage Flows: Flat view and Hierarchical view. These views control how nested Flows are displayed and accessed.

Switching Between Views

You can toggle between Flat and Hierarchical views using the view toggle button on the Flows page.

• Use Flat view for fast filtering, searching, and bulk management.

• Use Hierarchical view to understand execution structure and Flow relationships.

Both views operate on the same Flows and do not affect execution, scheduling, or configuration.

Flat View (default)

In Flat view, all Flows are displayed in a single, flat grid regardless of hierarchy.

• Every Flow appears as an individual row in the list.

• Nested Flows are indicated by a special icon next to the Flow name.

• Clicking the nested-flow icon opens an expandable popup that shows the full nested structure.

• From this popup, you can navigate to any parent or child Flow.

Flat view is optimized for searching, filtering, and working with large numbers of Flows.

Hierarchical View

In Hierarchical view, Flows are displayed inline as a tree starting from top-level Flows.

• Parent and child Flows are shown directly in the grid with indentation.

• Nested Flows can be expanded or collapsed inline.

• This view provides a clear visual representation of execution hierarchy.

Interaction with Tags View

When Hierarchical view is enabled together with Tags view:

• Only top-level (main) Flows are shown in the grid.

• Nested Flows are still accessible, but via the expandable popup instead of inline expansion.

• This preserves tag-based grouping while preventing deeply nested trees inside tag sections.

When Tags view is disabled, Hierarchical view displays the full inline tree with all nested levels.

Table and Card Views

The Flows page supports two visual layouts for browsing Flows: Table view and Card view.

You can switch between views using the view selector in the toolbar.

Table View

Table view displays Flows in a compact grid format.

Each row shows key Flow information such as:

• Flow name

• Flow type

• Last execution status

• Scheduling indicators

• Quick action icons

Table view is optimized for:

• Managing large numbers of Flows

• Sorting and filtering

• Performing bulk operations

Card View

Card view displays each Flow as a visual card.

Each card shows:

• Flow name

• Flow type

• Execution status

• Action buttons such as run, duplicate, and delete

Card view is useful when browsing templates or visually scanning Flows, especially when working with a smaller number of Flows.

Both views operate on the same Flows and switching between them does not affect Flow configuration, execution, or scheduling.

Flow Context Menu (Right-Click Menu)

You can quickly access common Flow actions using the context menu.

To open the menu:

1. Navigate to the Flows page.

2. Right-click a Flow in the list.

The menu provides actions for the selected Flow, including:

• Open flow – opens the Flow editor.

• Show schedules – displays all schedules associated with the Flow.

• View flow XML – shows the underlying Flow definition.

• Duplicate flow – creates a copy of the Flow.

• Delete flow – removes the Flow.

When multiple Flows are selected, the context menu also includes actions for the selected group, such as:

• Run selected flows

• Advanced options

• Delete selected flows

The context menu provides a fast way to manage Flows without navigating through multiple screens.

Bulk Operations on Flows

The Flows page supports bulk operations that allow you to update multiple Flows at once. This is useful when managing large environments with many Flows.

To perform bulk actions:

1. Navigate to the Flows page.

2. Select one or more Flows using the checkboxes on the left side of the grid.

3. Open Flow Context menu.

4. Choose one of the available bulk operations.

Available bulk actions include:

• Run selected flows – executes all selected Flows immediately.

• Advanced options – opens the Bulk Actions dialog with additional configuration options.

• Delete selected flows – deletes all selected Flows.

Advanced Bulk Actions

The Advanced bulk actions dialog allows you to update multiple Flow properties in a single operation.

Supported bulk updates include:

• Name prefix – add a prefix to the names of selected Flows.

• Name suffix – add a suffix to the names of selected Flows.

• Name regex replace – rename Flows using a regular expression pattern.

• Add tags – add one or more tags to all selected Flows.

• Remove tags – remove specific tags from selected Flows.

• Set tags – replace all existing tags with a new set.

These operations help maintain consistent naming conventions and organize Flows using tags across large projects.

Export and import Flows

You can export any Flow to a local.etld file and later import it into another Etlworks instance or account.

All related Connections and Formats are exported along with the Flow.

Export a Flow

1. Go to the Flows page.

2. Click Export / Import Flow → Export.

3. Select the Flow and click Export.

4. A.etld file will be downloaded.

Read more about Exporting Flows.

Import a Flow

1. Go to the Flows page.

2. Click Export / Import Flow → Import.

3. Choose a.etld file.

4. The system will create a new Flow and all necessary components.

Read more about Importing Flows.

Advanced Configuration

Generate Flow Documentation

You can generate detailed documentation for any flow directly from the Etlworks UI.

To generate documentation, first open the Flow Editor for the desired flow. This applies to both regular and nested flows.

There are two ways to open the documentation popup:

1. Top right link in the flow editor

2. Documentation icon at the bottom right

Both options open the same popup window with documentation, which includes markdown-formatted structured documentation and Visual HTML/SVG diagram which represents the structure of the flow.

Read about Generating Flow Documentation.

Collapsed and Expanded Flow Header

The Flow Editor header can be displayed in either a collapsed or expanded mode, depending on how much context you want visible while working on a Flow.

To expand or collapse the header, use the expand/collapse control on the right side of the Flow header.

The collapsed header keeps the editor clean and focused, making it easier to concentrate on building transformations and nested logic without visual distractions. This mode is well suited for experienced users or for editing deeply nested Flows where vertical space matters.

The expanded header exposes additional context such as Flow type, description, and quick access actions. This is especially useful when designing or reviewing complex workflows, onboarding new users, or working with large nested pipelines where understanding intent and metadata upfront helps prevent mistakes.

You can control which mode is used by default in User Preferences. Enable or disable “Open flow header expanded by default” (disabled by default) to match your working style. The header can still be expanded or collapsed manually at any time, regardless of the default setting.

Find where the Flow is included in the nested Flows 

In Etlworks, a Flow can be reused as a step inside a nested Flow.

To check where a Flow is used:

• Click Find Usage at the bottom of the Flow Editor.

• A popup will show all parent Flows that include this Flow.

• Click on any Flow name to open it.

If a Flow is included in a nested Flow, an icon appears next to its name. Click the icon to jump to the parent Flow.

Also, if the Flow is included in the nested Flow, we display an icon next to the Flow name which indicates that the Flow has a parent nested Flow. Click on the icon to open the nested Flow editor.

Disable manual flow execution

If a Flow is used only inside a nested Flow and should not be run directly:

• Open the Flow Editor.

• Go to the Parameters tab.

• Disable Manual execution.

Daemon flows

A Daemon Flow runs in an infinite loop until stopped or interrupted.

It is effectively a Flow that runs every second—without scheduling and without reinitializing resources each time.

When to Use Daemon Flows

Use Daemon Flows for:

• Micro-batch processing (e.g., reading from a message queue)

• Polling for files or changes (e.g., watching a folder)

Instead of relying on timers or schedulers, the Daemon Flow wakes up, performs a task, then sleeps briefly before repeating.

How to Enable Daemon Mode

• Open the Flow Editor.

• Go to the Parameters tab.

• Enable the Daemon flag.

Organize flows with tags

Tags are the primary way to organize flows, connections, formats, listeners, and schedules in Etlworks. A tag view groups artifacts in the list by the tags you've assigned; switching the tag view off shows them as a flat list.

See Tags and Tag Views for full details on creating, assigning, and filtering by tags.