Selecting Files and Folders in Flows

Overview

Every flow that reads or writes files — ETL flows with a file source or destination, file-management flows, file loops, and bulk-load flows — selects those files the same way: a connection defines where to look, and the FROM (or TO) value defines what to select, using relative paths, wildcards, and optional modifiers.

This article is the reference for that selection model: the connection Directory as the root, relative paths, wildcards, the File Path Modifiers syntax and dialog, processing files in subfolders, and how Copy/Move/Rename place files in the destination. Flow-type specifics live in the linked articles.

The connection Directory is the root

Every file-based and cloud storage connection has a Directory setting. It defines the root folder for all operations performed through that connection. If no Directory is set, the storage system's default root is used (for example, the SFTP user's home folder or the root of a cloud container).

All paths in FROM and TO are resolved relative to the Directory, and flows never operate above it. For example, with Directory demo/data and FROM sales/*.csv, the flow selects files under demo/data/sales.

Relative paths and wildcards

The FROM value can be:

• a single file name — customers.csv;
• a relative path — inbound/customers.csv;
• a wildcard — *.csv, orders_2026*.json;
• a relative path with a wildcard — inbound/*.csv;
• any of the above followed by File Path Modifiers.

Wildcards match against the file's name (its basename). {tokens} for global variables can be used anywhere in the path — see Parameterization of file operations.

Include files in subfolders

By default a wildcard selects files only in the configured folder. Enable Include files in subfolders to search below it: the wildcard is matched against the name of every file in every descendant folder, and matches keep their path relative to the connection Directory so the flow can open the exact file.

FROM: *.csv    (Include files in subfolders enabled)

matches:
customer.csv
inbound/customer.csv
inbound/2026/customer.csv

The option is available wherever a flow selects source files:

• ETL flows with a file-based source (including Any-to-Any ETL when the selected source is a file connection);
• file-management flows that select files — Copy, Move, Rename, Delete, archive and extract variants, Check/Count files, and the generic File Management transformation;
• Files by wildcard loops in nested flows and Composer;
• applicable bulk-load flows.

Points to understand:

• Disabled by default. Existing flows behave exactly as before; nothing changes unless you enable it.
• The connection Directory remains the boundary — recursion never escapes it.
• Include/exclude rules, processed-file tracking, sorting, and maximum-file limits all continue to apply. Results from all folders are combined first; sorting and the maximum-file limit are then applied globally, not per folder.
• Include and exclude patterns can use folder-qualified paths, for example *.csv;-archive/*.*,failed/*.* or *.csv;+inbound/*.csv,priority/*.csv.
• An explicit sorting modifier in the File path overrides the connection's default sorting.

File Path Modifiers

The FROM/File path accepts modifiers appended to the wildcard. This is the complete syntax:

With subfolders enabled, the exclude and include lists accept folder-qualified paths (archive/*.*) as well as file names.

The File Path Modifiers dialog

Click the filter icon next to the File path to open File Path Modifiers.

A dialog that edits this same syntax visually will open. It parses the current value when it opens and writes the standard syntax back when you click Apply; Cancel leaves the value untouched. The field remains directly editable — the dialog is a convenience, not a replacement for manual entry.

The dialog has up to three sections:

• File selection — the wildcard, include/exclude lists, Skip previously processed files (with Cache name and Cache retention), and, for file loops, Process folders matching the wildcard.
• Sorting — the sorting override.
• Transformation file filters — shortcuts that are not File Path syntax. They read and write separate transformation parameters: Include files, Exclude files, Include files in subfolders, and — for Copy/Move/Rename and Delete/Move — Preserve source folder structure and Delete empty source subfolders. They appear only when the current flow type supports them, and are applied only when you click Apply.

Two mechanisms, one dialog:

Process folders matching the wildcard (<*test>) is a genuine File Path modifier that turns matching folders into file-loop iterations. It is meaningful only for Files-by-wildcard loops and is not shown for ordinary ETL or file-management transformations. See Looping in Workflows.

How Copy, Move, and Rename place files in the destination

When Include files in subfolders is enabled, matched files keep their relative source path for reading — but by default they are written directly into the configured destination folder, without recreating the source subfolders:

source:                     destination (default):
north/customer.csv    →     customer.csv
south/orders.csv      →     orders.csv

If two source folders contain files with the same name, the flattened destination would collide — Etlworks prevents the overwrite by appending a generated unique suffix to the later file's name.

To recreate the source hierarchy instead, enable Preserve source folder structure:

source:                     destination (preserved):
north/customer.csv    →     north/customer.csv
south/orders.csv      →     south/orders.csv

Notes:

• Disabled by default, and applies only to recursive source processing — it does not change non-recursive transformations.
• Missing destination folders can be created automatically for supported destination connections.
• It is separate from the Automatically Create Missing Folders option available on some connections.

Delete empty source subfolders

Move and Delete operations can optionally clean up source folders that became empty. Enable Delete empty source subfolders (disabled by default):

• Files are processed first; then the remaining source folders are evaluated and empty ones are deleted deepest-first.
• The configured source root is never deleted.
• A folder is kept if it still contains anything — an unmatched file, an excluded file, a file skipped by the processed-file cache, or a file whose operation failed.

This is source cleanup after a successful run — not an instruction to delete every folder the flow traversed.

Previewing the selection in FROM and TO

The FROM/TO object selector for file connections previews the files your current configuration selects — wildcard, folder prefix, inline include/exclude modifiers, transformation Include/Exclude files, subfolder recursion, and sorting are all taken into account. Skipped-by-cache state is not previewed, because it can change at runtime: the preview is an editing aid, and the running flow remains authoritative. Objects are grouped into categories (File, CDC events, File loop variable); see Setting Source and Destination in Transformation.

Creating folders

Missing destination folders referenced by a relative TO path can be created automatically by connections that support Automatically Create Missing Folders. Folders can also be created explicitly with a Create Folder file-management flow (see Managing Files and Folders with File Management Flows) or interactively in Etlworks Explorer.

Integration Agent compatibility

When a flow runs on an Integration Agent, the recursive options are available only on agents that support them. The host checks the selected agent's capabilities: older agents keep the previous non-recursive behavior, the unsupported options are not silently executed, and an error for an unsupported operation reports the currently required agent version. Upgrade the agent to the version shipped with the current Etlworks release to use all options.

Related articles

• File as a transformation source and File as a transformation destination
• Get started with files: Process files in an ETL flow using wildcard filenames and Folders and relative paths
• Managing Files and Folders with File Management Flows
• Looping in Workflows — file loops, folder loops, and loop variables
• The destination file name in file operations
• Parameterization of file operations