Articles in this section

See more

Get started with databases

  • Updated
Follow

Etlworks works with most relational databases. This article is the entry point for everything you can do with a database connection — reading, writing, schema management, transactions, change replication, partitioning, and more.

What can you do with databases in Etlworks?

Topic What it covers
Connecting

Database connectors — the full list of supported databases and how to configure each. 

Test a connection. 

Use a temporary in-memory database for staging.
Browsing data Explore connections and metadata, run SQL queries in the Etlworks Explorer.
ETL between databases

ETL with relational databases. 

Move tables matching a wildcard name from one schema or database to another.
ETL when the database is the source or destination

Source is a relational database. 

Destination is a relational database.
Bulk load Bulk load data into a database — the fast path for large volumes.
SQL actions

INSERT, UPDATE, DELETE, MERGE, IFEXIST, CUSTOM — how each record is applied to the destination. 

Advanced SQL with the CUSTOM action.
ELT ELT with databases — load first, transform in the target database using Before / After SQL or a dedicated SQL flow.
Executing SQL

Execute any SQL outside of a regular source-to-destination transformation. 

Execute SQL from JavaScript.
Schema management

Automatically create or alter a destination table. 

Automatically create staging tables from any data source. 

Override data types when creating or altering tables. 

Override the auto-generated CREATE and ALTER TABLE SQL.
Partitioning Automatic partitioning — extract large tables in parallel chunks.
Transactions and connections

Transaction management. 

Open the database connection only when needed (required when the connection URL is parametrized with global variables).
Change replication

Change Data Capture (CDC). 

CDC into a relational database. 

Change replication using a high watermark. 

Change Tracking (CT) for SQL Server.
Multi-table loads Load one source dataset into multiple destination tables.
Database loop Run a flow once for each record returned by a query.
Expose as API Expose a database object or SQL query as an API endpoint.
On-premise data Use the data integration agent to reach databases behind a firewall.
Performance Performance tips when working with databases.
Troubleshooting

Cannot connect to the database. 

Foreign key constraint violation when loading or deleting data.
Connected Apps

Connected App as source. 

Connected App as destination. 

Connectors for Connected Apps.

Related articles