Composer

Overview

Composer is the canvas-based flow builder in Etlworks. Instead of picking a flow type first — the gallery approach, where you commit to "Database to Warehouse" or "File to API" before you understand the shape of the integration — you drop connections on a canvas, wire them, and Composer infers the appropriate flow type from the pattern.

Underneath, Composer reaches across the platform’s full surface: 200+ flow types, 3,979 templates, and the entire connector library. The classic gallery is still available; Composer is an alternative entry point that suits exploratory work, mixed-pattern flows, and anyone who has not memorized the platform’s flow-type taxonomy.

Where Composer appears

Composer became the default flow-creation path in version 9.4.4. There are three entry points:

1. From the home screen

The home screen has three Create widgets that open Composer directly:

• Vibe-build a flow — describe what you want in natural language; Simba opens a new Composer flow with your prompt and starts building.
• Blank canvas — opens a new, empty Composer flow.
• Start from template — opens the template browser; importing a template seeds the Composer canvas with pre-wired pairs.

See Home Screen and Widgets for the full home-screen reference.

2. From the flow gallery

In the flow gallery (the Select Flow Type dialog), Composer is the first flow type. Composer is treated as just another flow type alongside the 200+ classic types — Database to Database, File to File, Web Service to Database, and so on. The classic types remain available for users who already know which type they want.

3. Convert an existing flow

Any existing flow can be converted into a Composer flow. Select one or more flows in the Flows list and click Convert to Composer in the toolbar (the swap-arrows icon; the tooltip names the selection count).

The Convert to composer flow(s) dialog has:

• Collision Policy — what to do when a connection, format, or flow with the same name already exists. Defaults to Replace.
• Add tags — optional tags applied to the converted flow.
• Import schedules — when on, schedules attached to the source flow are imported. Imported schedules are disabled by default to avoid unexpected executions.
• Do not import connection credentials — when on, connections used by the converted flow are imported without their secret fields; fill them in manually after conversion.
• Import Message — free-text audit message attached to the conversion event.

Click Convert All to convert every entry in the dialog, or Convert Selected to convert only the entries with their checkboxes ticked.

Canvas concepts

Pairs

A pair is the smallest unit on the Composer canvas. Most pairs represent a source-to-destination move (a connection on the left, a connection on the right, optional staging in between, with format and transformation attached). Some pairs represent standalone blocks — a file-management operation (copy, move, zip), a script (JavaScript, SQL, HTTP), or a control structure (loop, conditional, parallel).

Regions

A region is a named group of pairs. Use regions to organize larger flows into logical sections — "Extract", "Transform", "Load to Snowflake" — without leaving the canvas. Regions can be renamed, grouped, or removed; they do not affect execution order on their own.

Linked and nested flows

A pair can be a reference to another flow, either an existing one or a draft that Composer is building alongside the current flow. This is how nested workflows are expressed: parent Composer flow on the canvas, child flows as pairs that fan out into their own canvases.

Recipes (flow-type inference)

When you wire a source and destination, Composer matches the combination against a set of resolution rules to pick the right flow type. For example, wiring a Salesforce connection to a Snowflake connection with S3 staging resolves to the Snowflake-via-S3-staging warehouse recipe; wiring a Kafka topic to a database resolves to the streaming/CDC recipe. The flow type is shown on the pair so you always know what is running underneath.

What Composer can build

Composer covers the platform’s main flow families:

• Any-to-any ETL. Move data between any two supported connectors with format conversion, transformation, lookups, and filtering.
• Warehouse and bulk loads. Snowflake, Redshift, BigQuery, Synapse, Greenplum, Oracle, Vertica via staging (S3, Azure Blob, GCS, or local), or direct bulk load where the warehouse supports it.
• Streaming and CDC. Kafka, RabbitMQ, database CDC sources into databases, warehouses, files, or queues.
• File management. Copy, move, rename, delete, mkdir, split (CSV, XML, JSON), merge, zip, unzip.
• Scripts and ad-hoc work. JavaScript, Python, SQL, SSH script, and HTTP-request flows can be added as pairs alongside data movement.
• Email. Send and read emails, including outbound-with-attachments toggles per pair.
• Nested workflows. Loops, conditionals, parallel branches, and error-handling regions, all expressed in the canvas.

Anything you can build in the classic flow editor, you can build in Composer. The full template library (3,979 templates) is searchable from within Composer.

Building a flow

There are three primary ways to put pairs on a canvas. They all draw from the same set of dockable panels along the left edge of the editor.

Tools panel

Icons along the left edge open dockable panels for the building blocks you place onto the flow. Each panel has Existing and New tabs, plus a tag filter and a search box:

• Connections — drag a connection onto the canvas to start a new pair. Existing shows the connections already defined in your tenant; New lets you create one from a connector descriptor.
• Flows — drag an existing or draft flow onto the canvas to create a nested-flow pair. This is the entry point for building nested workflows: one Composer flow that calls another.
• Controls — drag a control block (Condition, Loop, Parallel, On Error) onto a pair or region to wrap it with that control.

The same sidebar also includes the Schedules panel (covered under Schedule) and help / refresh shortcuts.

Drag and drop

Open the Connections panel, drag connections onto the canvas, and wire source to destination. Add staging, attach a format, open the transformation modal to configure mappings, and attach control structures from the Control Tools panel (Condition, Loop, Parallel, On Error). Composer resolves the flow type as you wire.

Ask Simba

Open the Simba chat panel and describe what you want — for example, "load Stripe charges into Snowflake every hour, full refresh for the first run and incremental after that." Simba creates connections (including HTTP connections built from public vendor docs), drops pairs on the canvas, sets formats and transformations, configures the schedule, and shows the result live on the canvas. You approve each state-changing action before it applies.

More on Simba →

From a template

Use the Flows panel to browse the template library or ask Simba to find one ("find a template for Salesforce to BigQuery with hourly schedule"). Importing a template seeds the canvas with pre-wired pairs that you then adapt.

Working with the canvas

Canvas toolbar

The toolbar above the canvas surfaces flow-wide controls:

• Validity indicator — at-a-glance status (Flow is valid or an error/warning label). Click for the full validation report.
• Breadcrumbs — show your position when you drill into a nested flow. Click any crumb to jump back up.
• Zoom controls — zoom out, reset to 100%, zoom in.
• Compact view — toggle between full-size and compact pair rendering. Compact view is useful for canvases with many pairs.
• Group into nested — select two or more consecutive ungrouped pairs and use this button to extract them into a nested flow.
• Layout actions — auto-arrange the canvas and other layout helpers (see Layout and regions).
• Search canvas — find pairs, regions, or fields on the canvas by name. The filter icon scopes the search.

Transformations and inline testing

Each pair has a transformation surface where you configure source-to-destination field mappings, filters, lookups, and computed columns. The transformation modal includes Test Transformation, so you can validate mappings against sample source data without leaving the canvas.

Conditions, loops, parallel, and error handling

Control tools are draggable onto the canvas alongside pairs:

• Condition — branch execution based on flow parameters, run results, or expressions. Conditional branches are marked on the canvas with decision dots so you can see at a glance which pairs are unconditional and which are gated.
• Loop — iterate over a list, query result, or file set; the looped body runs once per item.
• Parallel — run independent branches concurrently.
• On Error — catch failures from a pair or region and route to recovery, notification, or skip behavior.

Schedule

Open the Schedules panel to attach one or more schedules to the flow. Schedules support cron expressions, listener-driven triggers, and webhook endpoints; multiple schedules on the same flow can run in parallel.

Layout and regions

Pairs auto-arrange into columns, and Composer provides an Arrange canvas action for one-click layout reset. Group related pairs into regions, rename them, and resize them to organize large flows.

Save and run

Saving a Composer flow makes it a first-class Etlworks flow — it appears in the gallery, supports the same audit trail, can be exported and imported, can run from the CLI and from the REST API, and can be referenced as a nested flow from other Composer flows.

From the canvas you can save, run, or save-and-run in a single action. Runs can be synchronous (the canvas waits for completion and shows results) or asynchronous (the run starts and the canvas remains editable).

Editing after save

Composer supports incremental editing of saved flows. Edits made after save are tracked against a baseline so you can continue working without a full reload — the canvas stays in place, dirty state is scoped to the actual changes, and a second save applies only the delta.

Draft persistence

A Composer flow that has not yet been saved is held as a draft so you can step away, switch flows, or hand off to Simba and come back without losing the in-progress canvas. Drafts are short-lived; save the flow when you want to persist it indefinitely.

Composer flows in CLI, API, and exports

Composer flows are not a separate flow class — they are regular Etlworks flows with a Composer-rendered canvas. That means:

• CLI: Composer flows run via the standard run-flow commands.
• REST API: Composer flows are accessible through every flow-related API endpoint (list, get, run, schedule, audit).
• Export / import: Composer flows export to the same flow archive format used elsewhere in the platform and can be imported into other tenants.
• Version control: exported archives can be checked into Git or any other VCS.

CLI Commands Reference →

Etlworks REST API →

Composer and Simba together

Simba and Composer are designed to interleave. Typical patterns:

• Describe the goal to Simba, watch it build, take over manually for the last 10% — for example, tweak a transformation or attach a specific schedule.
• Build the structure manually, then ask Simba to fill in mappings, find a template for a specific connector, or write the validation script.
• Hand off mid-build either direction. Edits made manually do not interrupt Simba; edits Simba makes do not lock the canvas.

Using Simba in Composer is optional. Composer works the same whether or not AI features are enabled for the tenant.

Composer vs. the classic flow editor

Both are supported. The classic flow editor remains available for every flow type and is often faster when you already know exactly which flow type you want and prefer the dedicated, type-specific editor. Composer is the better entry point when:

• You are exploring how to model an integration before committing.
• The flow mixes patterns (for example, file management plus warehouse load plus an HTTP call).
• You want Simba to do the heavy lifting.
• You are onboarding a teammate who has not memorized the flow-type taxonomy.

Permissions

Composer honors the standard role hierarchy:

• Editor and above can create, modify, save, and run Composer flows.
• Operator can run saved Composer flows and view canvases read-only; cannot save changes.
• Viewer can open and inspect saved Composer flows.

FAQs

How is Composer different from picking a flow type from the gallery?

The gallery commits you to a specific flow type before you have fully understood the integration. Composer skips that step: you drop connections, wire them, and Composer infers the type from the wiring. The gallery is still useful when you already know what you want; Composer is better for exploration and mixed-pattern work.

Do I need Simba to build in Composer?

No. Composer works manually (drag and drop on the canvas) without any AI involvement. Simba is optional and can be disabled at customer, tenant, or user scope.

Can I still start from a template?

Yes. The full template library is searchable from within Composer, either through the Flows panel or by asking Simba to find one. Importing a template seeds the canvas with pre-wired pairs.

Can Composer flows be exported, version-controlled, and run from the CLI?

Yes — Composer flows are regular Etlworks flows. Export, version-control, run from CLI, and orchestrate via the REST API exactly as you would any other flow.

What scripting languages does Composer support inline?

JavaScript, Python, and SQL (across the supported dialects). Script blocks live on the canvas alongside data-movement pairs.

Is Composer available in on-premise deployments?

Yes. Composer is part of the standard Etlworks application and ships in every deployment model — shared cloud, dedicated cloud, hybrid, and fully on-premise.