Overview
Starting with version 9.9.9, Etlworks includes a deep integration with dbt: you can run dbt Core projects and trigger dbt Platform jobs as regular Etlworks flows, in the same orchestration graph that extracts, moves, validates, and delivers your data.
The integration reuses existing Etlworks concepts — there is no separate "dbt transformation" and no second credential store. A dbt flow is a standard flow: it participates in nested flows, Composer, loops, schedules, the REST API, the CLI, notifications, permissions, audit, and Integration Agents exactly like any other flow type.
What the integration includes
| Component | What it does |
|---|---|
| Execute dbt Project flow | Runs dbt Core locally on the selected execution node against a local or Git-backed project. |
| Execute dbt Platform Job flow | Triggers and monitors an existing dbt Platform deployment job through the dbt Platform API. |
| Materialize Git Repository flow | Resolves a Git ref to an exact commit and materializes a read-only repository snapshot — useful with or without dbt. |
| Git Repository connection | Reusable Git credentials and repository details for Git-backed dbt projects and Git flows. |
| dbt Platform connection | Stores the dbt Platform API host, account ID, and encrypted API token. |
In addition, the flow-execution dashboard gains a dbt Results tab with parsed models, seeds, tests, sources, timing, artifacts, and visual lineage, and the Etlworks CLI and flow-execution API gain dbt-specific capabilities.
dbt Core or dbt Platform?
- Use Execute dbt Project when you want Etlworks to run dbt Core itself: your project lives in Etlworks-managed storage or a Git repository, and your existing Etlworks warehouse connection supplies the credentials. There is no profiles.yml to maintain — Etlworks generates an ephemeral profile for each execution from the selected connection.
- Use Execute dbt Platform Job when your team already manages projects, environments, and jobs in dbt Platform. Etlworks triggers the existing job, polls it, captures logs and artifacts, and reports the result — the remote job keeps its own warehouse credentials, so no Etlworks warehouse connection is needed.
Both flow types share the same execution dashboard, orchestration, and automation surface, so you can start with one and mix or migrate later.
Prerequisites for local dbt Core
Etlworks runs dbt Core as a local operating-system process on the execution node (the Etlworks host or an Integration Agent). The runtime is an administrator-installed prerequisite — Etlworks never installs dbt, adapters, Git, ODBC drivers, or native libraries automatically:
- dbt Core — the validated baseline is dbt Core 1.12.3.
- The dbt adapter for your warehouse — see the validated adapter versions in Connect dbt to your warehouse.
- Git — required for Git-backed projects and Git flows.
- unixODBC and Microsoft ODBC Driver 18 — required on the execution node for SQL Server, Azure Synapse, and Microsoft Fabric Warehouse.
Two settings control where dbt runs (tenant settings override global settings, which override the defaults; on an Integration Agent the equivalent values come from the Agent parameters):
- dbt Executable (dbt.executable) — executable name or absolute path; default dbt.
- dbt Project Root (dbt.project.root) — root directory for local projects; default {app.data}/dbt.
Note: The dbt work directory is not configurable; it is created below the Etlworks temporary folder and removed after the execution.
How a local execution works
- Etlworks resolves the project — a directory below the project root, or a private snapshot of the selected Git repository pinned to an exact commit.
- Dependencies are restored or installed according to the package policy — see Manage dbt packages and dependencies.
- An ephemeral profiles.yml is generated from the selected Etlworks warehouse connection in a private per-execution workspace.
- The dbt command is built as an argument vector and executed without a shell — shell metacharacters in selectors, variables, or macro names cannot become shell commands.
- dbt output streams into the Etlworks execution log; results and artifacts are captured and parsed.
- Transient credentials, profiles, and work files are removed.
Security model
- Warehouse and Git credentials stay in encrypted Etlworks connections — never in flow fields, Git URLs, or a second dbt credential store.
- Profiles and credential files are ephemeral and private to the execution.
- No shell command construction; strict path containment, traversal, and symlink checks.
- Secrets are redacted from logs, audit data, notifications, webhooks, errors, and cache keys. Artifacts containing resolved secret values are not retained.
- Project, package-cache, and artifact storage are bounded per account, with tenant-aware isolation and quotas.
What the integration does not do
- No dbt transformation type — dbt runs as a flow.
- No separate dbt scheduler, notification system, credential store, or permission model — the standard Etlworks ones apply.
- No creation or editing of dbt Platform projects, environments, jobs, or schedules.
- No Git commit, push, merge, or history editing — Git access is read-only.
- No automatic installation of dbt, adapters, Git, ODBC drivers, or native libraries.
- No silent fallback to the Etlworks host when an Integration Agent is unavailable or incompatible.
Where to go next
- Execute dbt Project flow — the complete flow reference.
- Connect dbt to your warehouse — supported warehouses and authentication.
- Manage dbt packages and dependencies
- Use dbt state, defer, and retry
- View dbt results, artifacts, and lineage
- Run dbt and Git flows on the Integration Agent