Overview
The Etlworks CLI gives data engineers, SREs, and platform operators the ability to manage flows, schedules, agents, tenants, messages, and data directly from the command line. It is designed for teams who prefer automation, reproducibility, and Infrastructure-as-Code workflows similar to Airflow, dbt, or Terraform, while keeping the simplicity of Etlworks’ orchestration engine.
This article provides real automation scenarios based entirely on real CLI commands and the built-in SQL layer.
Monitoring and Troubleshooting
Find long-running flows
What it does:
Retrieves all running flows and filters for those running longer than one hour.
Why it is useful:
Useful for detecting stuck flows, deadlocks, or slow external systems before they cause downstream failures.
Commands:
list-running-flows
select flowId,flowName,tenantId,date(started) as started
where started < ts("now - 1h")
Get recent flow failures
What it does:
Lists executions from the last 2 days (including today) and extracts only failed runs.
Why it is useful:
Allows operators to quickly diagnose issues without opening UI logs, especially in large multi-flow environments.
Commands:
(all tenants)
flow-executions-aggregated tenants=all offset=1 includeMetrics=true
select flowId, eventName, tenantId, status, user,
date(started) started, date(ended) ended, exception
from executions
where status != "success"
order by auditId desc
(single/current tenant)
flow-executions offset=1 includeMetrics=true
select flowId, eventName, tenantId, status, user,
date(started) started, date(ended) ended, exception
from executions
where status != "success"
order by auditId desc
Investigate flows with high data volume
What it does:
Shows the flows that read or processed the most data in the last 24 hours.
Why it is useful:
Detects performance hotspots and heavy workflows that may require partitioning or schedule rescheduling.
Commands:
(all tenants)
flow-executions-aggregated tenants=all offset=1 includeMetrics=true
select flowId, eventName, tenantId, status, user,
date(started) started, date(ended) ended, records
from executions
order by records desc limit 10
(single/current tenant)
Note: requires date
flow-executions date=2025-11-30 includeMetrics=true
select flowId, eventName, tenantId, status, user,
date(started) started, date(ended) ended, records
from executions
order by records desc limit 10
Inspect a problematic flow for structural issues
What it does:
Runs a static inspection on a flow definition.
Why it is useful:
Quickly highlights anti-patterns such as missing formats, incomplete mappings, or inefficient configurations.
Commands:
inspect-flow 123
Output (example)
{
"severity": "CRITICAL",
"flowIssues": [
{
"flowId": "345",
"flowName": "Copy into S3 using REST",
"severity": "CRITICAL",
"issues": [
{
"severity": "CRITICAL",
"issueType": "PERFORMANCE",
"description": "Streaming is disabled for the transformation between flat datasets",
"why": "When streaming is disabled, the entire source dataset is loaded into memory before being transformed and loaded into the destination, causing significant performance issues, especially with large datasets.",
"suggestion": "Enable \"Stream Data\" to optimize performance"
},
{
"severity": "INFO",
"issueType": "STRUCTURE",
"description": "In the ETL transformation where the file is a source, the option \"Ignore when there is no file\" is disabled",
"why": "When this option is disabled and the file does not exist, the flow will fail with an error",
"suggestion": "Enable \"Ignore when there is no file\""
}
]
}
]
}
Check system health
What it does:
Returns node-level status, memory, thread pools, and service availability.
Why it is useful:
Provides a health snapshot for cluster-wide automation and monitoring tools.
Commands:
health
Flow Execution Automation
Run a flow and wait for completion
What it does:
Triggers a flow by id synchronously and waits for the result.
Why it is useful:
Useful for CI/CD pipelines and chained workflows requiring deterministic execution.
Commands:
run-flow 123 sync=true
Run a flow by name (multi-tenant)
What it does:
Executes a flow using its name and an optional tenant override.
Why it is useful:
Ideal for shared environments where flow IDs differ across tenants.
Commands:
run-flow-by-name 'load data' tenant=test sync=false
Stop running flow
What it does:
Attempts a graceful stop of a currently running flow.
Why it is useful:
Allows automated safeguard scripts that prevent uncontrolled resource consumption.
Commands:
stop-flow 123 --force
Flow Management
Delete flows matching specific conditions
What it does:
Retrieves all flows whose names match a substring, then deletes these flows without user's confirmation. Use with caution.
Why it is useful:
Allows users to batch-delete deprecated flows.
Commands:
list-flows where name like '%test cdc%' | capture flows=stdout;
for-each (f in flows) {
delete-flow {f.id} deleteSchedules=true --force;
}
Retrieve a flow configuration
What it does:
Returns flow object by id.
Why it is useful:
Useful for auditing and configuration-as-code workflows.
Commands:
get-flow 123
Retrieve configuration of flows and store it in the data folder
What it does:
Retrieves all flows whose names match a substring, then exports each flow’s full configuration and saves it as a JSON file under the account’s data folder.
Why it is useful:
Ideal for auditing, backup, migration, and configuration-as-code workflows.
Teams can easily version flow definitions, review history, or synchronize configurations across environments.
Commands:
The script below selects all flows whose names include script, then writes each flow’s configuration into the backup/ subfolder of the account’s data directory.
list-flows --silent where name like "%script%" | capture flows=stdout;
for-each (f in flows) {
get-flow {f.id} >> backup/{f.name}.json;
};
Identify where a flow is referenced
What it does:
Shows schedules, agents, and other flows that depend on a given flow.
Why it is useful:
Critical before deleting or modifying a flow to avoid breaking downstream workloads.
Commands:
flow-usage 123
Schedule Management
List enabled schedules
What it does:
Shows all schedules and allows filtering for enabled ones.
Why it is useful:
Useful for auditing schedules after major deployments or tenant migration.
Commands:
(all tenants)
list-schedules scope=all where enabled=true
(single/current tenant)
list-schedules where enabled=true
Temporarily disable a schedule
What it does:
Disables a schedule for maintenance or incident handling.
Why it is useful:
Prevents unexpected flow execution during a maintenance window.
Commands:
disable-schedule 123 --force
Enable a schedule
What it does:
Re-enables a schedule that was previously turned off.
Why it is useful:
Simple but essential when automating multi-environment lifecycle management.
Commands:
enable-schedule 123 --force
Disable schedules whose names match a substring
What it does:
Finds all schedules whose names contain a specified substring and disables them.
Why it is useful:
Helps administrators manage schedules in large or multi-tenant environments, especially when bulk-disabling related schedules during maintenance or migration.
Commands:
The script below selects all enabled schedules whose names include a given substring, then disables each one.
list-schedules scope=all where name like "%your_substring_to_match%"
and enabled = true | capture schedules=stdout;
for-each (s in schedules) {
disable-schedule {s.id} --force;
};Compare schedule versions
What it does:
Shows diffs between current and previous versions of a schedule.
Why it is useful:
Useful when debugging unexpected schedule behavior or verifying configuration drift.
Commands:
schedule-history-diff 123 current last
Connections and Formats
Test a connection
What it does:
Executes a connection test using the Etlworks engine or an agent.
Why it is useful:
Ensures connectivity before running flows dependent on that connection.
Commands:
test-connection 123
Find where a connection is used
What it does:
Lists flows referencing a given connection.
Why it is useful:
Prevents accidental breaking changes when altering or replacing connections.
Commands:
find-connection-usage 123
Retrieve a connection configuration
What it does:
Returns connection object by id.
Why it is useful:
Useful for auditing and configuration-as-code workflows.
Commands:
get-connection 123
Build a default connection from a descriptor
What it does:
Creates a connection template based on the descriptor type.
Why it is useful:
Accelerates provisioning in infrastructure-as-code workflows.
Commands:
build-default-connection 'connection.cloud.s3.sdk'
Data and SQL Exploration
Retrieve raw data behind a connection and format
What it does:
Returns the raw untransformed data (file, object, API output).
Why it is useful:
Useful for validation, debugging mappings, and verifying permissions.
Commands:
(connectionId, formatId and object are required parameters)
data-get-raw connectionId=38 formatId=16 object=test.csv
Run SQL directly against a connection
What it does:
Executes SQL or extraction logic defined on the connection.
Why it is useful:
Allows data engineers to quickly validate schemas, inspect anomalies, or perform spot checks.
Commands:
(sql must be enclosed in "")
data-run-sql connectionId=123 sql="select * from orders"
Query messages for specific patterns
What it does:
Filters messages (requests to user-defined APIs) for size, format, status, or substring.
Why it is useful:
Useful in systems that rely heavily on message queues and need diagnostics or auditing.
Commands:
list-messages-filtered message=Notification
Get a specific message body
What it does:
Fetches and prints the message payload.
Why it is useful:
Useful for debugging parsing issues, API responses, or malformed events.
Commands:
get-message 24a6e77b-c05f-45ed-b1e8-908ce361bf31
Agents and Distributed Processing
Lists agents active as of 10 minutes ago
What it does:
Shows all agents which were active as of 10 minutes ago.
Why it is useful:
Important when diagnosing remote execution issues.
Commands:
(all tenants)
list-agents scope=all
select parent.id, parent.name, parent.clientId, ping_date
from ping
where parent.enabled=true and ping_date < ts("now - 10m")
(single/current tenant)
list-agents
select parent.id, parent.name, parent.clientId, ping_date
from ping
where parent.enabled=true and ping_date < ts("now - 10m")
Update All Enabled Agents to the Latest Version
What it does:
Automatically finds all enabled agents and issues an update-agent command for each one, upgrading them to the latest available version.
Why it is useful:
This is the fastest way to roll out a new agent version across your entire infrastructure.
Useful when performing cluster-wide upgrades, fixing agent-level bugs, or ensuring consistent runtime environments.
Commands:
(all tenants)
list-agents scope=all
select id, name where enabled=true | capture agents=stdout;
for-each (agent in agents) {
update-agent {agent.id} --force;
}
(single/current tenant)
list-agents
select id, name where enabled=true | capture agents=stdout;
for-each (agent in agents) {
update-agent {agent.id} --force;
}
Object History and Versioning
Diff between two flow versions
What it does:
Shows a detailed diff between current and previous versions of a flow.
Why it is useful:
Critical for debugging regressions and reviewing changes prior to deployment.
Commands:
flow-history-diff 123 from=current to=last
View change history for a connection
What it does:
Returns all past versions of a connection.
Why it is useful:
Useful for auditing migration or verifying configuration drift.
Commands:
connection-history 123
Diff connection versions
What it does:
Shows differences between two versions of a connection.
Why it is useful:
Quickly identifies changes that lead to connection issues.
Commands:
connection-history-diff f2472631-dedd-40ff-9238-abed8090fd16
75cc324e-bc6c-44e7-b60c-9a0eaeb95c18
Tenant and User Operations
List all tenants
What it does:
Displays the tenants configured in the system.
Why it is useful:
Useful for multi-tenant deployments or SaaS providers.
Commands:
list-tenants
Retrieve a tenant configuration
What it does:
Returns all settings for a tenant.
Why it is useful:
Useful for auditing and configuration-as-code workflows.
Commands:
get-tenant 1
List all users
What it does:
Displays users in the current tenant or environment.
Why it is useful:
Provides a quick overview of access control.
Commands:
list-users
Search and Discovery
Search for objects
What it does:
Searches flows, connections, formats, schedules, and other objects by keyword.
Why it is useful:
Saves time in large environments with thousands of objects.
Commands:
search 'etlConfig'
Tags and Metadata Management
List tags
What it does:
Shows all tags configured in the account.
Why it is useful:
Provides visibility into the existing tagging taxonomy, which is especially useful in large environments with many resources and contributors.
Commands:
list-tags
List tags across All Tenants
What it does:
Displays all tags configured across all tenants.
Why it is useful:
Gives super administrators a global view of tag usage and naming patterns across the entire platform, helping identify inconsistencies, duplicates, or unused tags.
Commands:
list-tags global=true
Update Tags in All Tenants
What it does:
Renames and deletes specific tags across all tenants.
Why it is useful:
Allows super administrators to centrally clean up, standardize, or retire tags system-wide, ensuring consistent tagging across all tenants and resources.
Commands:
update-all-tags '{"trim":true, "caseSensitive":true,
"rename":[{"oldTag":"bigquery","newTag":"google bq"}],
"delete":[{"tag":"bulk_merge"}, {"tag":"bulk_ merge"}]}'
Update Tags in the Current Account
What it does:
Renames and deletes specific tags within the current account only.
Why it is useful:
Enables administrators to manage and normalize tags at the account level.
Commands:
update-tags '{"trim":true, "caseSensitive":true,
"rename":[{"oldTag":"bigquery","newTag":"google bq"}],
"delete":[{"tag":"bulk_merge"}, {"tag":"bulk_ merge"}]}'
Update Tags in All Production Tenants
What it does:
Renames and deletes tags across multiple tenants selected by name pattern.
Why it is useful:
Allows super administrators to apply consistent tag changes to a subset of tenants (for example, all production environments) without impacting non-production accounts.
Commands:
list-tenants --silent where name like "%prod%" | capture tenants=stdout;
for-each (t in tenants) {
update-tenant-tags {t.id} '{"trim":true, "caseSensitive":true,
"rename":[{"oldTag":"bigquery","newTag":"google bq"}],
"delete":[{"tag":"bulk_merge"}, {"tag":"bulk_ merge"}]}'
};
Runtime Administrative Controls
Suspend all flow executions
What it does:
Pauses scheduler and manual executions.
Why it is useful:
Important during maintenance windows or outages.
Commands:
flows-suspend --force
Resume all flow executions
What it does:
Re-enables the scheduler and manual executions.
Why it is useful:
Restores normal operations after maintenance.
Commands:
flows-resume --foce
Force-suspend and stop active flows
What it does:
Suspends execution and attempts to stop currently running flows.
Why it is useful:
Used for emergency stops.
Commands:
flows-suspend-force --force
Documentation Automation
Generate flow documentation
What it does:
Outputs the full Markdown documentation for a flow.
Why it is useful:
Integrates with automated documentation pipelines.
Commands:
flow-documentation 123