Change data capture (CDC) is an approach to data integration that is based on the identification, capture, and delivery of the changes made to the source database and stored in the database redo log (also called transaction log).
Etlworks supports native log-based CDC for PostgreSQL, SQL Server, MySQL, and Oracle. Read about other change replication techniques available in Etlworks Integrator.
Setup CDC for the source database
Enabling CDC is different for each database. Please use the following tutorials:
Create CDC flow
Step 1. Create a source CDC connection for the source database.
Step 2. Create a connection to the destination database. Disable the auto-commit.
Step 3. Create a new flow by selecting any of the CDC flows from the gallery.
Step 4. Add a new source-to-destination transformation where:
- the FROM connection is the CDC connection created in step 1.
- the FROM is a fully qualified source table name.
- the TO is a destination table name.
- the TO connection is the destination database connection created in step 2.
Step 5. Click the
MAPPING button and select the Parameters tab. Set Action to
Record and Lookup fields to the comma-separated list of fields which uniqule identify the record in the source database.