Data Integration as Code (Flows as Code)

Overview

Since release 10.1.0, Etlworks supports data integration as code (also called flows as code): flows, connections, formats, macros and schedules become readable files that you keep in Git. You review changes in pull requests, and deploy them to any environment — dev, test, prod, another tenant or another instance.

There are two ways to work, and both are fully supported:

• In Etlworks (server-side Git). You create a Git Repository connection, put flows under Git with Manage in Git, and sync with the git-sync-in and git-sync-out commands, deploy on push, and Explorer. Etlworks holds the Git credentials.
• From a laptop or CI (client-side). The etlworks command-line client exports, plans and applies a project folder checked out by your pipeline. Etlworks never needs Git credentials.

The UI keeps working as before. Nothing changes for users who don't use Git; the new badges, panels and warnings appear only when objects are managed in Git.

In this article:

• Why use it
• Before you start
• Concepts
• Set up a Git Repository connection
• Put your flows under Git (Manage in Git)
• The project folder explained
• Credentials and secrets
• Export and import readable files in the UI
• Collision policies and modifiers
• Environments
• Plan and deploy
• Promote from dev to test to prod
• Day-to-day work with managed objects in the UI
• Code from Git: SQL and scripts in flows
• Automation: Run CLI Script flows, deploy on push, webhooks and CI
• Git in Explorer
• Command reference
• Security and shared environments
• Limits and defaults
• Troubleshooting
• Frequently asked questions
• Glossary
• Related articles and pages

Why use it

• Change history and review. Every change to a flow is a readable diff in a commit; you review it in a pull request before it reaches production.
• Repeatable deployments. Deploy the same project from a branch or tag to any tenant or instance. A tag never moves, so production runs exactly what was tested.
• One set of objects, per-environment values. Environment files and stored values supply hosts, warehouses and credentials per environment.
• Seeing and controlling drift. Etlworks tracks changes made directly in an environment after a deploy and shows them before they are overwritten.
• CI/CD. The same plan-and-apply steps run from GitHub Actions, GitLab CI or Azure DevOps with the etlworks client.
• SQL and scripts in Git. Flows can read SQL and script files straight from a Git repository at run time (Code from Git).
• Backups. A project export is a complete, readable, restorable copy of your flows and everything they use.

When you don't need it: if one environment and the built-in version history (object history and revisions) are enough, nothing changes for you — the UI works exactly as before.

Before you start

• Etlworks release 10.1.0 or newer, Cloud or on-premises.
• A Git repository on GitHub, GitLab, Bitbucket, Azure DevOps or any Git server reachable over HTTPS or SSH, and an access token (recommended for HTTPS) or an SSH private key.
• Git installed on the Etlworks server. Etlworks' Docker images include it. On other installs, Test Connection tells you when it is missing: "Git isn't installed on this server. Install git, or set Settings > Git > Git Executable."
• For CI or working from a laptop: Java 17 or later and the etlworks client (install instructions).
• Permissions. What each role can do:

Git commands also require access to the Git Repository connection: read access for git-refs, and Edit on the connection for commands that change the repository or the workspace. The CLI asks for a confirmation before commands that push, deploy, tag or discard work.

Concepts

These terms are used throughout this article (the glossary has the full list):

• Project — a named set of objects deployed together. On disk it is a project folder with a manifest and one file per object.
• Object — a flow, connection, format, macro or schedule that belongs to a project.
• Manifest — the etlworks.json file at the root of a project folder: the project name and the authoritative list of its objects.
• Readable bundle — a single .etlworks.json file: the readable twin of the encrypted .etld export. Every Import accepts both.
• Matching by name and type — objects are identified by their name and type (trimmed, case-insensitive), never by internal IDs, so the same project deploys to any tenant or instance. Two objects with the same name and type on the target are an error: a deploy never guesses.
• Environment — a named set of per-environment values (hosts, warehouses, variables, schedule settings) applied on top of the project when it is deployed, from an environments/<env>.json file and from values stored in Etlworks.
• Collision policy — what happens when an object in the project already exists on the target: replace it, keep it, copy it, or fail.
• Plan and apply — project-plan shows what a deploy would do, object by object, and changes nothing; project-apply deploys in one transaction.
• Managed object — an object recorded as belonging to a project after Manage in Git or a deploy. Managed objects get a Git badge, a source panel and drift tracking.
• Drift — a change made directly in Etlworks to a managed object after its last deploy. The next deploy overwrites it, and Etlworks shows it first.
• Prune — deleting objects that this project deployed earlier but that are no longer in it.
• Ownership and takeover — every deployed object is recorded as managed by its project; moving an object to another project requires an explicit takeover.

Set up a Git Repository connection

A Git Repository connection tells Etlworks where a repository is and how to authenticate. Server-side Git features — Manage in Git, the Git commands, Code from Git, Explorer — use it. (The etlworks client does not need one: your pipeline checks the repository out itself.)

1. Open Connections and click Add connection.
2. Search for Git Repository (group Developer Tools) and select it.
3. Fill in the fields (table below), starting with Repository URL and Authentication.
4. Click Test Connection.

Result: Test Connection connects to the Git host and lists what it found: "Connected. Default branch: main; 4 branches, 2 tags." If Git is not installed on the server, or the URL or credentials are wrong, the message says exactly what to fix.

The connection works with GitHub, GitLab, Bitbucket, Azure DevOps and any other Git server reachable over HTTPS or SSH.

Two copies of the repository: remote and workspace

Etlworks reads and writes a repository in two distinct ways. Knowing which one a feature uses explains most of its behavior:

• The remote is the repository on your Git host. Deploys (git-sync-in), Code from Git and the Explorer's Remote view read any branch, tag or commit directly from it, through a per-commit snapshot cache on each server.
• The workspace is Etlworks' own working copy for one connection (one per tenant and connection). The Explorer's Local view and the git-status, git-commit, git-push and related commands work in it. An unused workspace is removed after the retention period (30 days by default).

Server settings (Settings → Git)

A Super Admin can tune the server-wide Git behavior under Settings → Git. Tenant configuration can override most of these per tenant.

Put your flows under Git (Manage in Git)

Manage in Git is the one-step start: it exports selected flows — and everything they use — to a folder on a branch, pushes, and records the objects as managed in Git at that commit. Nothing changes in Etlworks itself: your flows keep working exactly as before. From then on, the Git badges, the source panel and drift tracking are active.

1. Open Flows and select the flows you want to manage.
2. Click Bulk actions → Manage in Git….
3. In the Manage in Git dialog, choose:
   • Git connection — the Git Repository connection to push to.
   • Branch — an existing branch (the default branch is marked), or New branch… with a name. A new branch starts from the default branch; open a pull request from it on your Git host.
   • Project name — for example analytics.
   • Folder — auto-filled as projects/<project> until you edit it.
   • Also manage connections, formats and schedules — see below.
   • Take over objects managed by another project — only when moving objects between projects.
4. Click Manage in Git.

Result: the dialog shows Done. with what was exported and pushed, and the selected flows show a Git badge in the flows list. In the repository, the folder now contains the project manifest and one file per object.

What gets managed:

• By default, only flows and macros are managed. Connections, formats and schedules are exported to the project (so it is complete and deployable), but deploys leave them as they are on the target.
• With Also manage connections, formats and schedules ticked (or all-types=true in the CLI), those types are managed and updated by deploys too.

Rights: Manage in Git needs Edit on the Git connection (to push) and Edit on each selected object. If any selected flow is already managed by another project, nothing is pushed: "Nothing was pushed: flow 'Load orders' is managed by project 'analytics'. Use takeover=true to move them to this project."

The CLI equivalent:

git-manage "Analytics repo" flows=<id|name,...>|all [project=<name>] [path=<folder in the repository>] [branch=<name>] [all-types=true] [takeover=true] [connections=nosecrets|reference|encrypted] [schedules=true|false] [message="..."]

Without Git: the same project workflow works from a plain folder. Export with project-export (or Export → Readable JSON in the UI), keep the folder wherever you like, and deploy it with project-apply — applying records the objects as managed by the project.

The project folder explained

A project folder is the Git layout of a project — readable, diffable, and deployable from any source:

etlworks.json                      manifest: project name, objects, connections mode, collision policy
flows/load-orders.json             one file per object
flows/load-orders/<field>.sql      long code stored as files (.sql, .js, .py, .sh, .xsl, .txt)
connections/snowflake.json
formats/csv-default.json
macros/common-filters.json
schedules/nightly.json
environments/prod.json             per-environment values (you write these; export never touches them)

The manifest, etlworks.json, holds:

• formatVersion — the file-format version;
• project — the project name (default: the folder name);
• connections — the export mode for connections (three modes);
• secrets — the key fingerprint, when credentials are encrypted;
• collision — the project's default collision policy;
• notes — free text;
• objects — the authoritative list of the project's files. Files not listed are ignored, and project-validate reports them.

Inside the files:

• Names are identity. File names are slugs of object names, but the name inside the file is what matters. Renaming an object in Git means changing the name inside its file — the next deploy creates the object under the new name, and the old object stays on the target unless you prune.
• References between objects are by name:

{"$ref": {"type": "connection", "name": "Snowflake"}}

• Macros in code are written {{macro:<name>}} and turned back into local IDs on import.
• Code becomes files. Code values of 20 lines or more go into separate files next to the object (flows/load-orders/source-query.sql), so diffs read like code.
• Canvas layout is kept in one layout object, so moving boxes on the canvas is a small diff.
• No noise. No IDs, timestamps, owners, run state, or export metadata (who, when, which version). An export is byte-identical when nothing changed, so an unchanged export produces an empty diff. Line endings are LF.

What a project includes: flows (nested flows and Composer children are followed automatically), the connections, formats and macros they use (including a connection's Explorer format and HTTP listener connections used by flows), tenant macros, and schedules.

What a project does not include:

• admin-level (global) macros — reported, but not exported;
• flows that a dynamic flow runs by name at run time — reported as notes;
• webhooks, Integration Agents, templates, users and permissions.

Note: when an import creates a new HTTP listener connection whose URL already exists on the server, the new listener gets a dated URL (<url>_<MM-dd-yyyy_HH-mm-ss>) so the two never clash; an MLLP listener's port is changed instead.

Newer files: a file from a newer format version is refused: "This export was made by a newer Etlworks version (format N). Upgrade this instance, or export again from the older one."

Credentials and secrets

How connection credentials are handled is the most important choice you make. There are three modes, set in the Export dialogs, in the manifest, or with connections= on the CLI:

Recommended practice: keep secrets out of Git. Export without secrets, and set credentials once per environment — either directly in the target's connection, or as stored environment values with project-env-set … secret=true (Environments).

Keys for encrypted credentials

Keys live in Settings → Security, section Export/Import Encryption:

• Private Key (RSA; the same key also encrypts .etld exports) with its Private Key Passphrase, or
• Readable JSON Key — a passphrase of at least 32 characters, used when there is no Private Key.

Import uses whichever configured key matches the file's fingerprint. Exporting with encrypted credentials without a key fails: "Can't include credentials: set the Private Key or the Readable JSON Key in Settings → Security → Export/Import Encryption."

Each encrypted envelope is bound to its object and field — a value copied into another field won't decrypt — and is deterministic, so an unchanged export stays byte-identical.

Shared servers (Etlworks Cloud, any server with tenants): the export key is shared by every tenant on the server, so tenant users can't export or import encrypted credentials. Only Super Admins can, and users of single-tenant installs. The message explains the alternatives: "Encrypted credentials aren't available to tenant users: the export key is shared by every tenant on this server. Export connections without secrets (nosecrets) and set credentials in each tenant, or keep them as environment values stored in Etlworks (project-env-set)."

Two related fixes shipped in 10.1.0: importing a connection with "Replace" but without credentials now keeps the target's passwords (it used to erase them), and exports without credentials also blank secure schedule parameters.

Export and import readable files in the UI

The existing Export and Import dialogs gained the readable format. Nothing changed for encrypted exports.

Export (flows; connections and formats):

• File format: Encrypted (.etld) — the default, unchanged — or Readable JSON (.etlworks.json).
• Connections: Without secrets (default) / Reference only (flows only) / With encrypted credentials. The encrypted choice appears only when a key is configured; otherwise a note says which setting to fill in.

Import:

• The file picker accepts .etld and .json files.
• Collision Policy starts empty; you must choose one, as before.
• Also replace / Also keep: after choosing a policy, these multi-selects fine-tune it per type (Flows, Macros, Formats, Schedules, Connections), with a one-line summary such as "Replace: flows, macros, formats, schedules. Keep: connections." (The connections/formats Import dialog offers Macros, Formats and Connections.)
• The results list warnings — for example, credentials that couldn't be decrypted because the key isn't configured here.

Collision policies and modifiers

A collision is an object in the project whose name and type already exist on the target. The collision policy decides what happens, per type:

* Connections always follow their export mode too: without secrets, a replaced connection keeps the target's passwords.

The actions:

• replace — the object is updated in place; its ID, history and permissions stay.
• keep — the object is left alone.
• copy — a dated copy is created: "Load orders [11-14-2026_09-30-00]".
• fail (modifier only) — the collision is an error and nothing is written.

Missing objects are always created, under every policy.

Modifiers adjust the base policy per type: replace=<types>, keep=<types>, copy=<types>, fail=<types>, where types are flows, macros, formats, schedules, connections. A type may appear in only one modifier ("'formats' is in both replace= and keep=. Name each type once."). Example:

git-sync-in "Analytics repo" ref=main path=projects/analytics collision=Keep replace=flows,macros

Defaults, per entry point:

• Project deploys (project-apply, git-sync-in, the etlworks client) and readable-bundle imports from the CLI: Keep All, Replace Flows and Macros.
• UI import: no default; you choose.
• The existing import-* CLI commands for .etld files: Add (unchanged).

Precedence: command options win over the environment file's collision block, which wins over the manifest's collision, which wins over the default. The effective policy is printed at the top of every plan: "Collision: Replace: flows, macros. Keep: formats, schedules, connections."

Add and copy are refused for project deploys: "Add (copy) isn't allowed for project deploys: applying again would create new copies each time. Use Replace, Keep or KeepAllReplaceFlowsAndMacros."

Environment values on kept objects are not applied, and the plan warns about it. Use replace=connections (for example) when you want the environment's connection values to land.

Environments

An environment supplies the values that differ between dev, test and prod — hosts, warehouses, flow variables, schedule settings — so one project deploys everywhere. Values come from two places:

Environment files (in Git)

One file per environment, environments/<env>.json, chosen at deploy time with env=<env> (CLI) or --env <env> (client). Export never touches these files — you write them. A full example:

{
  "collision": { "policy": "KeepAllReplaceFlowsAndMacros", "replace": ["connections"] },
  "connections": { "Snowflake": { "host": "acme.snowflakecomputing.com", "warehouse": "PROD_WH" } },
  "formats": { "CSV default": { "properties": { "delimiter": ";" } } },
  "flows": { "Load orders": { "variables": { "batch_size": "50000" }, "properties": { } } },
  "schedules": { "Nightly": { "enabled": true, "cron": "0 2 * * *", "parameters": { "region": "us" } } }
}

• Connection and format keys are descriptor property keys; the field.connection. / field.format. prefix is optional.
• Flows take variables and properties.
• Schedules take enabled, cron and parameters.
• Unknown keys fail validation, and the message lists the valid ones.

Values apply after the project is read and before planning, so the plan always shows the final values.

project-env-init writes a template file listing the project's objects; entries you leave empty change nothing.

Stored environment values (in Etlworks, not in Git)

Each environment can also hold values stored in Etlworks itself — this is where secrets belong:

project-env-set analytics connection Snowflake password value=<password> secret=true
project-env-get analytics
project-env-unset analytics connection Snowflake password

• Stored values win over the environment file.
• Values set with secret=true are stored encrypted and always shown as ******.
• Setting and removing stored values needs the Tenant Admin role; listing them (project-env-get, secrets hidden) needs Editor.

Encrypted values in environment files

project-encrypt-value prints an encrypted value (enc:v1:…) that you can paste into an environment file instead of plain text. Like every encrypted-credentials feature, it is not available to tenant users on shared servers (Credentials and secrets).

Worked example — one connection, two environments. The project has a connection "Snowflake". environments/dev.json points it at the dev account; environments/prod.json at production:

// environments/dev.json
{ "connections": { "Snowflake": { "host": "acme-dev.snowflakecomputing.com", "warehouse": "DEV_WH" } } }

// environments/prod.json
{ "connections": { "Snowflake": { "host": "acme.snowflakecomputing.com", "warehouse": "PROD_WH" } } }

The password is never in Git. Set it once per environment: in dev, type it into the connection; in prod, run project-env-set analytics connection Snowflake password value=<password> secret=true once. Every later deploy reuses it.

Plan and deploy

Deploying a project is always the same three commands, whatever the source. A source can be:

• a folder under your app data: projects/analytics;
• an uploaded zip or readable bundle: upload:<id> (what the etlworks client sends);
• Git, read directly from the remote: git:<connection>[@<branch|tag|commit>] with path=<folder in the repository>.

project-validate: check without touching anything

project-validate checks the project itself — files, references, environment overrides (unknown keys list the valid ones), credentials, and conflicts with this environment — without comparing or changing anything.

project-plan: see what a deploy would do

project-plan compares the project with the target and prints one line per object. It changes nothing.

project-plan git:"Analytics repo"@main path=projects/analytics env=test

The actions a plan can show:

Diffs are shown as JSON paths with old and new values, and as line diffs for code files. Credentials show only as "credentials changed" — never values. The plan ends with a plan hash that identifies exactly this reviewed plan. Options: verbose=true lists unchanged objects too; json=true prints machine-readable output.

project-apply: deploy

project-apply git:"Analytics repo"@main path=projects/analytics env=test plan-hash=<hash>

1. It re-plans inside a lock — one deploy per tenant and project at a time; a second deploy waits up to 60 seconds, then fails with "Another deploy of this project is running. Try again when it finishes."
2. With plan-hash=<hash>, it refuses if anything changed since the plan was reviewed: "The plan changed since it was reviewed (expected <hash>, now <hash>). Nothing was deployed; review the new plan."
3. It imports everything in one transaction. Unchanged objects aren't written — no new revision, no webhook event. When nothing at all changed, apply reports "Up to date: nothing deployed."

Behavior worth knowing:

• Schedules: a schedule created by a deploy starts disabled, unless the environment file sets enabled: true. An existing schedule keeps its enabled state unless the environment sets it.
• Revision message: each written object gets the revision "Deployed from <project> @ <branch or tag> (<commit>) "<commit subject>" by <user>: <message>" — or "Deployed from <project> by <user>" when the source isn't Git.

Prune: deleting what the project no longer has

prune=true (Administrator only) deletes objects this project deployed that are no longer in it. Pruned objects go to the Recycle Bin, so they can be restored. Prune never touches objects the project didn't deploy, and it is blocked when something outside the prune set still uses the object.

Ownership and takeover

Every deployed object is recorded as managed by its project: the project, source, branch or tag, commit, folder, environment, who deployed it, and a hash of the deployed content. Deploying an object that belongs to another project is an error — "managed by project 'x' (takeover=true moves it to this project)" — and takeover=true (Administrator) moves it.

Drift: what changed here since the last deploy

project-drift [project=analytics] [changed=true]

It lists every managed object and whether each one changed here since its deploy. The canvas layout and credentials don't count as changes. The same data is on the Monitor → Git drift page (below).

Promote from dev to test to prod

The recommended workflow end to end. Secrets stay per environment as stored values; use prune only when you mean to delete.

1. Develop in a dev tenant as usual.

2. Export to a feature branch: select the flows → Bulk actions → Manage in Git… with New branch… feature/orders — or, once managed:

   git-sync-out "Analytics repo" branch=feature/orders path=projects/analytics project=analytics message="Faster orders load"

3. Review and merge the pull request on your Git host, from feature/orders into main.

4. Deploy to test from main. Plan first, then apply the reviewed plan:

   git-sync-in "Analytics repo" ref=main path=projects/analytics env=test dry-run=true
   git-sync-in "Analytics repo" ref=main path=projects/analytics env=test plan-hash=<hash from the dry run>

5. Tag the release once test passes:

   git-tag "Analytics repo" name=release-2026.11 ref=main

6. Deploy the tag to production:

   git-sync-in "Analytics repo" ref=release-2026.11 path=projects/analytics env=prod dry-run=true
   git-sync-in "Analytics repo" ref=release-2026.11 path=projects/analytics env=prod plan-hash=<hash>

   Result: production runs exactly what was tested — a tag never moves. The deployed objects' revision history shows the tag, the commit and who deployed.

CI can run the same steps with the etlworks client, and a push webhook can run them automatically (deploy on push).

Day-to-day work with managed objects in the UI

Badges and the Git filter

• Managed flows, connections, listeners, formats, schedules and macros show a Git badge in their lists (and in the macro pane). A dot on the badge means "changed here since deploy". Hovering shows "Managed in Git: <project> · <ref> (<commit>)".
• The Git filter on the flows, connections, formats, schedules and listeners lists narrows to All, Managed in Git, Changed since deploy or Not managed in Git.
• Badge state is cached for a short time (up to about half a minute), so a badge can briefly lag behind a change.

The source panel

Each managed object's editor shows a Managed in Git panel at the top with the facts: the project, the state (As deployed or Changed here since deploy), the repository, the branch or tag and environment, the commit (linked to the commit page on GitHub, GitLab — including self-hosted GitLab hosts — Bitbucket Cloud and Azure DevOps) with its subject, the folder, and when and by whom it was deployed. For objects deployed from a folder or upload, a Source line replaces the Git facts.

For users with edit rights on the object, the panel's buttons:

• Compare with Git version — what differs here from the deployed version. Credentials and canvas layout aren't compared. When nothing differs: "Same as the Git version."
• Revert to Git version (shown when the object changed here) — re-applies the deployed version of this one object; objects it references stay as they are. Revert warns that unsaved editor changes are lost, then reloads the editor. The object's revision history records ": Reverted to the Git version".
• Export changes to a branch (flows deployed from Git that changed here) — pushes the changed flow to a branch to review and merge. The dialog suggests a branch name (etlworks/<flow>-<date>), asks for a Commit message, and its Push to branch button shows a link to the branch when done.
• Stop managing in Git — the object stays exactly as it is; the next deploy treats it as not managed yet.

Revert isn't available for objects deployed from a folder or upload, or for objects renamed here — there is no Git version to go back to.

The CLI equivalents are project-compare <type> <name|id> and project-revert <type> <name|id>.

Save and delete warnings

• Saving a managed object asks once per editor session: "This <type> is managed in Git (<project>@<branch>). The next deploy will overwrite your changes."
• Deleting warns: "This <type> is managed in Git (<project>). The next deploy recreates it unless it is also removed from Git."

Monitor → Git drift

Monitor → Git drift (Super Admins and Tenant Admins; it appears once anything is managed) shows every managed object by project: "N managed objects, N changed here since deploy", with a Project filter and a Changed only tick box, columns Type, Name, State, Deployed from and Deployed, and links to the object editors and the commits. When everything matches: "Everything in this project is as deployed." It is the same data as project-drift.

How-to: someone changed a managed flow in production

1. Find the change: Monitor → Git drift → tick Changed only — or run project-drift changed=true. The flow shows Changed here since deploy.
2. Open the flow and click Compare with Git version to see exactly what differs.
3. Decide:
   • The change was a mistake → Revert to Git version. The flow goes back to the deployed state.
   • The change should be kept → Export changes to a branch, then review and merge the pull request on the Git host. The next deploy from the merged branch makes Git and production match again.
   • Do nothing for now → the next deploy will show the flow as update (drift) in the plan and overwrite the change.

Result: drift is gone — the Git drift page shows "Everything in this project is as deployed."

Code from Git: SQL and scripts in flows

Any code field of a flow — SQL, JavaScript, Python, shell scripts — can read its code from a Git repository at run time, so the code is reviewed and versioned like the rest of your project.

The Code from Git button

1. In a flow or Composer editor, click the Code from Git icon next to Open in Editor on a code field.
2. In the Code from Git dialog, choose the Connection, the Branch or tag (the list marks the default branch), and the File — browse the repository and preview the file; the preview header shows the commit.
3. Click Use this file.

Result: the field shows a chip — "From Git: <connection> · <path> @ <branch>" (or "@ default branch") — and every run reads the file from Git. The chip's buttons: Preview, Change, and Edit inline, which copies the file's current content into the field and stops reading from Git. The dialog's Copy reference button copies the reference instead, to paste inside other code.

The reference syntax

{{git:<connection>:<path>[@<branch|tag|commit>]}}

Example: {{git:Analytics repo:sql/orders/merge.sql@release-2026.11}}

• Without @, the connection's default branch is used.
• The connection can be a name or an ID.
• A path that itself contains @ needs an explicit ref: {{git:Analytics repo:a@b.sql@main}}.
• The Explorer's Copy reference gives the exact reference for any file (Git in Explorer).

References are resolved in the code fields of the flow and its transformations, source and destination queries, mapping formulas, nested conditions and loops, and in Test transformation.

At run time

• One commit per run. Each run asks the Git host once for the newest commit of each branch or tag it uses, and every file in that run comes from that commit — including nested flows and Composer steps. Different steps can never mix two versions of the repository.
• Expansion order: macros are expanded first, then Git references. References and macros inside a Git file are not expanded; {variables} work as usual.
• Limits: 5 MB per file, 50 files per run. Paths must be relative, with no "..", no ".git" and no links.
• Run details: the run shows a Code from Git section with each file, its branch or tag and the commit — plus "last fetched commit: the repository was unavailable" when the fallback was used.
• Integration Agents don't need Git: the resolved code is placed into the agent's configuration, and the agent picks up a new commit on its next check-in. When the Git host is unreachable during that check, the last fetched commit is used and the agent keeps running.
• When Git can't be reached, the run fails: "Couldn't read <path> from <connection>@<branch>: <reason>" — unless the connection's If the Repository Is Unavailable, Use the Last Fetched Commit option is on. A missing file, a missing branch or tag, or refused credentials always fail the run.
• Saving a flow checks that you may use each referenced Git connection: "Code from Git {{git:…}}: Git Repository connection '<name>' was not found."

Automation: Run CLI Script flows, deploy on push, webhooks and CI

Run CLI Script flows

The flow type Run CLI Script (group Automation and Orchestration) runs Etlworks CLI commands on the app, as a flow — so deploys, exports and Git commands can be scheduled, chained in workflows, or triggered by webhooks. Only Administrators see the type in the gallery and can create or edit these flows; for everyone else the editor is read-only.

Variables available in the script:

• the flow's own variables (run parameters, workflow variables);
• for a Git push: {git_branch}, {git_tag}, {git_ref}, {git_commit}, {git_repository}. They are set only when the pushed name contains just letters, digits and . _ - / + @; otherwise the run says "The pushed branch or tag name has characters that aren't allowed in scripts; {git_ref}, {git_branch} and {git_tag} aren't set."
• for an Etlworks webhook: {webhook_event_id}, {webhook_event_type}.
• Request bodies, listener parameters and headers never become variables, so outside input can't add commands to the script. For the same reason, don't pass untrusted data into the script through workflow variables — values are substituted into the script as text.

Results: the log shows each command with its output (capped at 1 MB) and exit code. The flow returns CLI_EXIT_CODE, CLI_LAST_OUTPUT, and any variables the script set (x = …, | capture) to a parent workflow. As a workflow condition ("Use the step result as condition"): the variable condition decides, otherwise "the script succeeded".

The rules:

• Only Administrators can create, edit or restore a Run CLI Script flow, and only Administrators can save a workflow or a schedule that runs one.
• It runs on the app, never on Integration Agents — including inside a parent workflow.
• It runs as the user who started it, the schedule owner, the listener's authenticated caller, or the Administrator who saved the webhook. Grants never raise the user's own rights.
• A scheduled run refuses to start if the schedule was last changed by a non-Administrator: "This schedule runs a Run Simba Prompt or Run CLI Script flow as its owner, but it was last changed by a user who isn't an Administrator. An Administrator must save the schedule again."
• It works as a nested or Composer step.

Deploy on push

Deploy a project automatically when its branch is pushed:

1. Create an HTTP listener connection — for example URL path /git/push, method POST, Include request headers on, and authentication JWT (Query Parameter) or Basic. Make the listener asynchronous: runs started by a listener run in the background.

2. Create a Run CLI Script flow with:

   • HTTP listener set to the listener from step 1;
   • Allow changes and Allow destructive and operational actions on;
   • a Git webhook secret;
   • the script:

   git-sync-in "Analytics repo" ref={git_branch} path=projects/analytics env=prod --force

3. Schedule the flow on the listener (schedule type listener).
4. Add the webhook on the Git host:
   • URL: https://<your instance>/rest/v1/httplistener/git/push?Authorization=Bearer%20<API key>
   • content type JSON; event: push; the same secret as in the flow.

Result: a push to the repository starts the flow; the run's log shows the plan and the deploy, and "Webhook signature verified." confirms the signature check.

Per Git host:

• GitHub and Bitbucket sign the body (X-Hub-Signature-256) with the secret.
• GitLab sends the secret as the X-Gitlab-Token header.
• Azure DevOps sends no signature: use Basic authentication on the listener instead.

A wrong or missing signature fails the run before any command runs. A listener without authentication requires the webhook secret: "A Run CLI Script flow started by an HTTP listener without authentication needs a Git webhook secret. Set one in the flow, or turn on authentication in the listener." (Run Simba Prompt flows can't be started by an unauthenticated listener at all.)

Etlworks webhooks that run a flow

Etlworks' own webhooks (event notifications) have a Run Simba or CLI flow option: when the webhook's events fire, the selected flow runs on the app, as the Administrator who saved the webhook. The flow gets {webhook_event_id} and {webhook_event_type}; the Git signature check doesn't apply here. Events about Simba or CLI flows themselves are ignored, so a flow can't trigger itself, and a manual run can't pretend to be a webhook event ("This Etlworks webhook event didn't start this flow").

CI with the etlworks client

The etlworks client is a command-line tool for laptops and CI runners. Your pipeline checks out the repository itself — Etlworks never needs Git credentials — and the client sends the project folder to the instance, plans and applies.

Install: Java 17 or later, and an Etlworks instance on 10.1.0 or later. Download https://etlworks.com/installers/download/client (one zip for macOS, Linux and Windows; checksum with ?asset=sha256, a specific release with ?version=10.1.0). Unzip anywhere and add the folder to PATH; on Windows the command is etlworks.cmd. Run etlworks version to check the install — it also shows whether Git is available on the server.

Sign in: create an API key in Etlworks under Preferences → Security (your user menu), then:

etlworks login --url https://<your instance> --api-key <key> [--profile prod]

The key is stored owner-only in ~/.etlworks/config.json (or under ETLWORKS_HOME). In CI, skip login and set ETLWORKS_URL, ETLWORKS_API_KEY and optionally ETLWORKS_PROFILE — environment variables win over the file. --insecure accepts self-signed TLS (test servers only).

Commands: login, logout, profiles, version; run "<cli command>" or run -f script.cli (with --var k=v and --tenant); project export|validate|plan|apply|drift|env-get|env-set|env-unset; flow run "<name>" [--wait --timeout 30m]. Global options: --json, --quiet. The project commands also accept a Git source directly — etlworks project plan git:"Analytics repo"@main --path projects/analytics — which uses the server-side connection instead of a local checkout.

project apply shows the plan and asks; in CI it needs --yes. --plan plan.json applies a previously reviewed plan.

A minimal GitHub Actions deploy step (ETLWORKS_URL and ETLWORKS_API_KEY set as repository secrets):

- name: Deploy to test
  run: |
    etlworks project plan ./projects/analytics --env test
    etlworks project apply ./projects/analytics --env test --yes

Complete, copy-ready pipelines for GitHub Actions, GitLab CI and Azure DevOps are on the developer page: Etlworks client — CI examples.

Exit codes:

Uploads: the client sends the project as a temporary upload — up to 200 MB, kept for one hour, per user. Only the files listed in the manifest and the environments/ folder are sent.

Git in Explorer

Git Repository connections appear in Explorer. Expanding one gives a Remote / Local switch: the read-only Remote view browses any branch or tag — open files, download, Copy path and Copy reference (ready to paste into Code from Git) — while the Local view is a full working copy where you edit and save files, upload, see status badges, diff against the last commit or the remote, and commit, push, pull and branch without leaving Etlworks.

The complete guide is Working with Git Repositories in Etlworks Explorer.

Command reference

One line per command; the full syntax, options and roles are on the developer page: Command reference — Git and Projects (flows as code). All git-* commands take the Git Repository connection's name or ID first; commands that change the repository or the workspace need Edit on the connection.

Caution with git-sync-out: it makes the branch match Etlworks. A change made only in Git (not deployed here) is reverted by the next sync-out from a tenant that doesn't have it — the commit keeps it in the Git history. Run git-sync-in first, or sync out to a feature branch and review the pull request.

For general CLI usage — the CLI page, scripts, variables and capture — see Using the Etlworks CLI.

Security and shared environments

• Etlworks Cloud and other servers with tenants keep tenants fully separate: Git snapshot caches and workspaces are per tenant and connection, and SSH keys and known-hosts files come only from the tenant's own uploaded keys.
• No secrets in Git, URLs, scripts or environment files. Use stored environment values (project-env-set … secret=true), or set credentials directly in each environment's connections.
• Administrator-only: prune and takeover; Run CLI Script flows, and the workflows and schedules that run them. Stored environment values need Tenant Admin.
• The CLI asks for confirmation before commands that push, deploy, tag or discard work; Run CLI Script flows need explicit grants (Allow changes; Allow destructive and operational actions) before a script may change anything.
• Webhook payloads and pushed branch names can't inject commands into scripts: request bodies, parameters and headers never become variables, and a pushed name with unexpected characters leaves {git_branch}, {git_tag} and {git_ref} unset. Don't pass untrusted data into scripts yourself through workflow variables.
• Simba, the Etlworks AI assistant, knows these features and confirms before deploying or reverting anything.

Limits and defaults

Troubleshooting

Frequently asked questions

Why didn't my connection's password arrive on the target? By design. The default export mode is Without secrets: password fields are never written to the files, and a deploy keeps the target's own secrets. Set the password once in the target connection, or store it per environment with project-env-set … secret=true.

Why is my schedule disabled after the deploy? A schedule created by a deploy always starts disabled, so a deploy can never start running things in an environment by surprise. Enable it in the environment file ("schedules": { "Nightly": { "enabled": true } }) or by hand.

Why does the plan say update (drift)? Someone changed that object directly in this environment after its last deploy. The deploy will overwrite the change. Use Compare with Git version (or project-compare) to see it, and Export changes to a branch to keep it.

Can I use this on Integration Agents? Flows deployed from Git run on agents like any other flows, and Code from Git works on agents without the agent needing Git access. Run CLI Script flows, though, always run on the app, never on an agent.

Do I need the etlworks client? Only for CI pipelines or working from a laptop. Everything server-side — Manage in Git, git-sync-in and git-sync-out, deploy on push, Explorer — works without it.

Does this work without Git? Yes. project-export writes a project folder, project-plan and project-apply deploy it from a folder or upload, and ownership, drift and environments all work the same. Only the Git-specific parts (badges with commits, Revert to Git version, deploy on push, Code from Git) need a repository.

Can two people deploy the same project at once? No — deploys are serialized per tenant and project; the second waits up to 60 seconds, then fails with a clear message.

Glossary

• Apply — deploying a project in one transaction (project-apply, git-sync-in).
• Code from Git — a flow's code field reading a file from a Git repository at run time.
• Collision — an object in the project whose name and type already exist on the target.
• Collision policy — what a deploy or import does on a collision: replace, keep, copy or fail, per type.
• Drift — a change made directly in Etlworks to a managed object after its last deploy.
• Environment — a named set of per-environment values applied at deploy time (file plus stored values).
• Environment file — environments/<env>.json in the project folder.
• Etlworks client — the etlworks command-line tool for laptops and CI.
• Git badge — the list marker on a managed object; a dot means changed here since deploy.
• Git Repository connection — the connection type holding a repository's URL and credentials.
• Manage in Git — the one-step action that exports flows to Git and starts managing them.
• Managed object — an object recorded as belonging to a project.
• Manifest — etlworks.json: the project's name and authoritative object list.
• Ownership — the record of which project manages an object, and from where it was deployed.
• Plan — the object-by-object preview of a deploy; its hash identifies the reviewed plan.
• Project — a named set of objects exported, versioned and deployed together.
• Project folder — the on-disk and in-Git layout of a project.
• Prune — deleting objects this project deployed that are no longer in it (to the Recycle Bin).
• Readable bundle — a single .etlworks.json file; the readable twin of the encrypted .etld export.
• Reference — a by-name link between objects in project files ($ref), or a {{git:…}} code reference.
• Remote — the repository on the Git host, read through a per-commit snapshot cache.
• Source panel — the Managed in Git panel at the top of a managed object's editor.
• Stored environment values — per-environment values kept in Etlworks, not in Git; where secrets belong.
• Takeover — explicitly moving an object from one project to another.
• Workspace — Etlworks' own working copy of a repository, per tenant and connection; what the Explorer's Local view and the commit and push commands use.

Related articles and pages

• Working with Git Repositories in Etlworks Explorer — the Explorer guide.
• Using the Etlworks CLI and CLI Commands Reference.
• Git in Etlworks — the developer reference for Git features.
• The etlworks client — install, commands and complete CI pipelines.
• Command reference — every CLI command.
• Etlworks Explorer Overview and User Interface.
• Webhooks API.