Articles in this section

Using the Etlworks CLI

  • Updated
Follow

Overview

The Etlworks CLI is an interactive, scriptable command-line environment built directly into the Etlworks UI. It is designed for ETL developers, data engineers, automation engineers, and operators who need precise, reliable, repeatable control over Etlworks objects, flows, and data.

The CLI provides a unified way to:

  • inspect, filter, and transform Etlworks metadata

  • run flows and orchestrate complex pipelines

  • explore and query data exposed by connections and formats

  • automate multi-step sequences using variables, loops, and conditions

  • apply SQL-like transformations directly to JSON results

  • build operational and administrative workflows without external tooling

The environment runs entirely inside Etlworks. No installation and no credential management. Commands always execute using the permissions of the authenticated user.

In addition to the in-UI interactive console, the entire CLI engine is also available through a single API endpoint, allowing external systems to execute the exact same commands and scripts programmatically. This makes it possible to:

  • trigger multi-step CLI scripts from CI/CD

  • run migrations, audits, or validations from external automation tools

  • embed Etlworks control flows into Airflow, Jenkins, GitHub Actions, or custom orchestration systems

See: Executing CLI via API

The result is a flexible execution layer that supports both ad-hoc interactive work and fully automated workflows, all powered by the same language, the same commands, and the same security model.

Built-in CLI vs External CLI

The built-in Etlworks CLI described in this article is not the same as the external Etlworks command-line tool used for system-level administration tasks such as upgrading Etlworks instances, managing backups, or controlling services.

The built-in CLI operates inside the Etlworks application and is focused on managing metadata, flows, data, and automation using the permissions of the authenticated user.

For system administration tasks performed outside the application, see: External Etlworks CLI (System Administration)

Accessing the CLI

Open CLI from the main navigation panel:

The interface contains:

Command Editor (top pane)

Write and edit commands.

Supports single-statement and multi-statement execution.

Command Palette

A searchable list of all available CLI commands.

Shortcut: Ctrl+Shift+O

History

Complete execution history with timestamps.

Shortcut: Ctrl+Shift+H

Output Panel

Displays JSON output, SQL-transformed tables, errors, logs, and captured values.

Keyboard Shortcuts

Action

Shortcut

Run command

Ctrl + Enter

Run command (alternative)

Ctrl + F2

Clear output

Ctrl + L

Open commands list

Ctrl + Shift + O

Open history

Ctrl + Shift + H

Close dropdown

Esc

Undo

Cmd + Z

Redo

Shift + Cmd + Z

Find & Replace

Cmd + Option + F

Go to Line

Alt + G

How Commands Execute

The Etlworks CLI is a statement-based execution engine.

A script may contain one or many statements.

If code contains multiple statements Each statement must end with a semicolon ;.

Single command

list-connections

Single Multi-line command

data-get-raw
  connectionId=1435
  object=customers.csv
  formatId=221;

Multiple commands

list-connections;
list flows;

Comments

// Single-line comment

/*
Multi-line comment
*/

Command Structure

Every CLI command follows a predictable, AWS-style pattern:

command-name positional1 'positional2 with spaces'
param1=value param2='value with spaces'
--modifier1 --modifier2
select ...
from ...
where ...
order by ...
limit ...
| capture var=stdout.field
& >> relative/path/name.json;

Components

Element

Meaning

command-name

Required. One of the registered Etlworks CLI commands.

positional parameters

Optional. Evaluated left to right.

name=value

Optional. Named parameters.

Quotes

'value' or "value" allowed.

Modifiers

--force

--silent

--silent-if-empty

--stop-on-error

SQL clauses

Inline SQL for JSON filtering/flattening

Capture

capture variables

Parallel

Append & to execute command in parallel thread

Redirect

Save the stdout of a command to a file within the tenant’s application data directory.

Syntax: >> filename

Redirect always appears at the end of the statement.

Parameters

CLI commands support positional and named parameters.

  • Positional parameters are passed in order, without names.

  • Named parameters use the format name=value.

If a parameter contains spaces, special characters, or structured data such as JSON, it must be enclosed in single quotes, for example:

print '{ "id": 10, "name": "Sample" }'

Named parameters follow the same rule:

connection='{ "status": "active" }'

Both single and double quotes can be escaped inside a quoted value. This allows you to preserve quotes inside JSON or text:

print '{ "message": "He said \"hello\"" }'
print "{ 'path': 'c:\\\\data\\\\file.txt' }"

If quoting is correct, the CLI will treat the entire payload as a single parameter, regardless of whitespace or internal punctuation.

Modifiers

Modifier

Description

--force

Allow destructive operations (delete-flow, delete-tenant, etc).

--silent

Suppress normal output. Errors still surface.

--silent-if-empty

Suppress output if stdout is null or empty array []. Errors still surface.

--stop-on-error

Stop the entire script on the first failure.

Modifiers must appear after the command name and before SQL or capture.

list-flows --silent select id where flowType like '%cdc%' | capture f=stdout.id;

Variables

Variables are fundamental to the CLI.

Any JSON value from any command can be captured and reused.

Variables can hold:

  • scalars

  • objects

  • arrays

  • results of SQL transformations

  • timestamps

  • values returned by functions inside SELECT

Capturing values

get-connection 120 | capture connId=stdout.id;

Capturing multiple fields

get-connection 120 | capture id=stdout.id name=stdout.connection.name;

Using variables

print '{"id":"{id}","name":"{name}"}';

Capturing arrays

list-tenants | capture tenants=stdout;

Now we can loop:

for-each (t in tenants) {
    print 't.name';
};

Control Structures

Control structures are built directly into the CLI language.

if control structure

if ({connId} == 100) {
    print 'Matched';
};

Supports:

  • comparison operators

  • boolean logic

  • nested if

  • expressions referencing JSON values

  • SQL-extracted fields inside {var}

for-each control structure

Iterate over arrays:

for-each (tenant in tenants) {
   print 'tenant.email';
};

Default behavior:

The loop continues execution even if a command inside the loop returns a non-zero exit code. This is consistent with the default CLI behavior (continue-on-error).

Stop on first error:

To stop the loop immediately when a command fails, use the --stop-on-error flag:

for-each --stop-on-error (agent in agents) {
    update-agent {agent.id} --force;
}

Supports:

  • nested loops

  • captures inside the loop

  • SQL inside the loop

  • running flows per element

Parallel Execution

Append & to run a command asynchronously.  The main script continues without waiting.

run-flow 100 &;

Add optional timeout:

run-flow 200 timeout=15000 &;

Redirects can be used together with parallel execution.

The redirect operator must still appear at the end:

get-flow 100 & >> f100.json

Inline SQL for JSON Output

Inline SQL is for ETL/data engineers:

  • filter

  • flatten

  • restructure

  • transform

  • extract values

  • prepare data for loops

list-flows select id,name where flowType like '%cdc%' order by name;

SQL features:

  • SELECT, DISTINCT

  • FROM (array flattening)

  • WHERE (expression engine)

  • ORDER BY

  • LIMIT

  • scalar & datetime functions

  • parent flattening via parent.*

Read about using SQL in CLI.

Redirecting Output

You can save the stdout of any command to a file by appending >> filename at the end of the statement.

Examples:

Save all flows:

list-flows >> flows.json

Save to a subfolder:

list-connections >> exports/conn-list.json

Export a specific flow:

get-flow 200 >> flows/200.json

Dynamic file name:

list-flows --silent where name like "%script%" | capture flows=stdout;
for-each (f in flows) {
   get-flow {f.id} >> backup/{f.name}.json;
};

Redirect works with SQL:

list-flows select id,name where flowType like '%cdc%' >> cdc.json

Notes about redirect

  • Redirect paths are always resolved relative to the tenant’s application data directory.

  • Subfolders are created automatically if missing.

  • For security reasons, absolute paths are not allowed.

  • Both stdout and stderr can be redirected.

  • Redirect file name can contain variables. Example: >> {f.name}.json. Read more here.

Example Workflows

List CDC flows

list-flows scope=all select id,name where flowType like '%cdc%';

Get last 10 failed executions

flow-executions-aggregated offset=5 
select flowId, auditId, started, ended, status
from executions
where status != 'success' and started > ts("now - 10d") order by ended desc limit 10

Capture flow id and run a flow

list-flows scope=all select id where name like 'postges cdc&' | capture flowId=stdoutp[0].id;
run-flow {flowId} sync=false;

More CLI automation examples.

Related articles