Articles in this section

See more

Execute dbt Project flow

  • Updated
Follow

Overview

The Execute dbt Project flow runs dbt Core as a local process on the selected execution node — the Etlworks host or an Integration Agent. The project can live in Etlworks-managed storage or in a Git repository; warehouse credentials come from an existing Etlworks connection, projected into an ephemeral dbt profile for each execution.

The flow is a standard Etlworks flow: it can run standalone or inside nested flows, Composer, and loops, on schedules, through the API and CLI, with standard permissions, notifications, audit, retries, and cancellation.

Available in Etlworks 9.9.9 and later. If you are new to the integration, start with Get started with dbt in Etlworks.

Create the flow

1. In the Flows window click + and type dbt. Select Execute dbt Project.

2. Select the destination warehouse connection. Supported types: PostgreSQL, Aurora PostgreSQL, Snowflake, Amazon Redshift, Databricks, Google BigQuery, SQL Server, Azure Synapse Analytics, Microsoft Fabric Warehouse, ClickHouse, or a Dynamic connection resolving to one of these. See Connect dbt to your warehouse.

3. If the project comes from Git, select a Git Repository connection in the Git repository slot.

4. Configure the parameters below.

Project sources

Local project

With Project Source = local, enter a relative directory in dbt Project Directory. The directory is resolved below the configured dbt Project Root (dbt.project.root, default {app.data}/dbt; tenant settings override global settings). Absolute paths, path traversal, unsafe symlinks, and directories without a dbt_project.yml are rejected.

Git-backed project

With Project Source = git, select a Git Repository connection and set:

  • Git Ref — branch, tag, or commit (default HEAD). Etlworks resolves the ref to an exact commit and records it with the execution.
  • dbt Project Subdirectory — for monorepos; the selected directory must contain dbt_project.yml.

The repository is materialized as a private, immutable snapshot for the execution. Recorded provenance includes the project source, repository connection, requested ref, exact commit, subdirectory, and dependency information — all visible in the dbt Results tab.

Parameters

  • Command: the dbt command to execute — build (default), run, test, seed, snapshot, compile, debug, list, source freshness, docs generate, run-operation, or retry. Etlworks passes structured options directly to dbt without using a shell.
  • Macro and Macro Arguments: used only by run-operation. Macro accepts package-qualified names such as my_package.grant_select; Macro Arguments is an optional JSON object passed as one --args value. Names and arguments are validated separately and never concatenated into a shell command.
  • Dependencies: Auto or Never — see Manage dbt packages and dependencies.
  • Target Schema: optional dbt target schema override. Database credentials always come from the selected connection.
  • Select / Exclude: passed as one --select / --exclude argument.
  • Variables: JSON or inline YAML passed as one --vars argument. Do not put secrets here.
  • Environment Values: non-secret environment pairs exposed to dbt deps and the selected command. Etlworks-managed process and dbt path variables cannot be overridden.
  • Secret Environment References: maps DBT_ENV_SECRET_* environment names to secure Etlworks execution or schedule parameters. Store the secret value in Secure parameters when running or scheduling the flow — never in the flow itself. Secrets are redacted from logs, audit data, notifications, and errors, and artifacts containing resolved secret values are not retained.
  • State, Defer, and Retry: state-based selection, defer, and native dbt retry — see Use dbt state, defer, and retry.
  • Threads: dbt worker threads, 1–128 (default 4).
  • Full Refresh: pass --full-refresh to commands that support it.
  • Fail Fast: pass --fail-fast.
  • Timeout (ms): maximum process run time (default 3600000 = 1 hour).
  • Keep Log: include dbt standard output and error in the Etlworks flow log (default on).
  • Execute if Error, On Exception (Raise/Ignore), Exception Mask, Disable manual flow execution: standard Etlworks flow behavior.

Variables and tokenization

Text fields support standard Etlworks runtime substitution: {variableName} for flow and global variables and ${VARIABLE_NAME} for environment references. This covers the command-specific fields, project, ref, subdirectory, selectors, exclusions, variables, schema, macro, and macro arguments — the command itself can be supplied through a token. After tokens are resolved, Etlworks performs the same command, type, path, compatibility, and security validation used for literal values.

How the execution works

  1. The project is resolved (local directory or Git snapshot pinned to an exact commit).
  2. Dependencies are restored or installed per the package policy.
  3. An ephemeral profiles.yml is generated from the warehouse connection in a private per-execution workspace.
  4. dbt runs as an argument vector without a shell; output streams into the Etlworks execution log.
  5. manifest.json, run_results.json, catalog.json, and sources.json are captured, parsed, and shown in the dbt Results tab.
  6. Transient credentials, profiles, and work files are removed.

Cancelling the flow (manually or by timeout) terminates the dbt process tree and runs the same cleanup.

Orchestration and automation

  • Use the flow inside nested flows, Composer, and loops — for example, extract and load with regular ETL flows, then transform with dbt in the same pipeline. When a parent execution contains several dbt invocations, the dashboard shows all of them.
  • Schedule it like any other flow.
  • Execute it through the Run Flow API with an optional dbt override object, or through the CLI dbt commands.
  • Assign it to a compatible Integration Agent to run dbt next to a private warehouse.

Related articles