Articles in this section

Change Data Capture (CDC) from transaction log

  • Updated
Follow

Overview

Change data capture (CDC) is an approach to data integration that is based on the identification, capture, and delivery of the changes made to the source database and stored in the database redo log (also called transaction log).

The changes are captured without making application-level changes and without having to scan transactional tables. CDC is a perfect solution for non-intrusive, low-impact real-time data integration.

Etlworks supports native log-based change data capture for PostgreSQL, SQL Server, MySQL, Oracle, DB2, and MongoDB. 

We are using a heavily modified embedded Debezium engine for CDC.

Read about other change replication techniques available in Etlworks Integrator.

Enable change data capture for the source database

Enabling CDC is different for each database. Please use the following tutorials:

CDC connectors

Read about available CDC connectors. 

CDC pipeline

The end-to-end CDC pipeline extracts data from a source CDC-enabled database and loads it into any supported destination:

  • another relational database,
  • cloud data warehouse such as Snowflake, Redshift or Azure Synapse,
  • a web service,
  • a data lake,
  • a file storage system.

In Etlworks the end-to-end pipeline typically includes 2 flows:

  • Extract flow - this flow extracts data from a CDC-enabled database
  • Load flow - this flow loads data into any supported destination

The extract and load flows are running in parallel which guarantees a very high processing speed and low latency.  In fact, the actual pipeline can include multiple independent extract and load flows which allows it to scale horizontally between multiple processing nodes.      

The Extract

This flow extracts data from a CDC-enabled database.

There are 2 types of extract CDC flows in Etlworks: 

Stream CDC events, create files

In this scenario, the CDC events (INSERTS, UPDATES, and DELETES) are extracted from the transaction log of the source database and stored as CSV files in the local or cloud storage.

This flow type does not require any additional elements of the infrastructure, such as the message queue. It is also the fastest type of CDC supported by Etlworks.

The separate load flow reads the files, then transforms and loads data into the destination. The load flow must delete the processed files.

Creating a flow that streams CDC events and creates files in the local or cloud storage

There is no need to create a separate flow for the initial load.  The first time it connects to a CDC-enabled source database, it reads a consistent snapshot of all of the whitelisted databases. When that snapshot is complete, the flow continuously reads the changes that were committed to the transaction log and generates the corresponding insert, update, and delete events.

Step 1. Create a CDC connection for the source database. Configure the following parameters:

  • Enable Serialize CDC events as CSV files 
  • Include Databases(s) - the comma-separated list of databases(s) to poll CDC events from.
  • Include Table(s) - the command-separated list of fully qualified table names or regular expressions to match the table names. You can override this in the FROM attribute of the source-to-destination transformation.  If you are planning to poll data from tens or hundreds of tables consider using regular expressions or {tokenizing} the Whitelisted Table(s) or FROM.

Step 2.  Create new flow using flow type Stream CDC events, create files 

mceclip1.png

Step 3. Add a single source-to-destination transformation and set the following parameters:

  • Source connection - CDC connection created in step 1
  • FROM - the following options are available:
    • * - the system will get the list of the whitelisted tables from the CDC connection, or
    • a regular expression to match the table names, or 
    • a comma-separated list of tables to poll data from, or
    • {token}

Example with a comma-separated list of whitelisted tables:

  • CDC key - [db]_[table]_cdc_stream
  • Include Table(s)- test.inventory,test.payment,test.customer
  • FROM - *
  • The following files will be created in the local storage:
    • {app.data}/debezium_data/events/test_inventory_cdc_stream_uuid.csv
    • {app.data}/debezium_data/events/test_payment_cdc_stream_uuid.csv
    • {app.data}/debezium_data/events/test_customer_cdc_stream_uuid.csv

Example with a regular expression:

  • CDC key - [db]_[table]_cdc_stream
  • Include Table(s)- ^[t][e][s][t][.][aA][a-zA-Z\d_\s]+
  • FROM - *
  • The following files will be created in the local storage:
    • {app.data}/debezium_data/events/test_inventory_cdc_stream_uuid.csv
    • {app.data}/debezium_data/events/test_payment_cdc_stream_uuid.csv
    • {app.data}/debezium_data/events/test_customer_cdc_stream_uuid.csv

The {app.data}/debezium_data is a default root location for the CDC events serialized as CSV files. You can override it by changing the value of the property Location of the CDC events under the CDC connection->Storage.

mceclip0.png

Step 4. Schedule the extract flow. We recommend using a continuous run schedule type. The idea is that the extract flow runs until there are no CDC events to poll, stops for the configured number of seconds, then starts again.

mceclip2.png

Stream CDC events to a message queue

In this scenario, the CDC events (INSERTS, UPDATES, and DELETES) are extracted from the transaction log of the source database and sent to the message queue, such as Kafka and Azure Event Hubs.

Message queues such as Kafka and Azure Event Hubs support transactional polling which makes the entire process very reliable and highly scalable. The only downside is a requirement to have a message queue, which is a separate element of the infrastructure that is not managed by Etlworks. Here is a list of fully managed message queue services, compatible with Kafka consumer and producer APIs:

The separate load flow polls the events from the topics in the message queue, then transforms and loads data into the destination.

Creating a flow that streams CDC events to a message queue

There is no need to create a separate flow for the initial load. The first time it connects to a CDC-enabled source database, it reads a consistent snapshot of all of the whitelisted databases. When that snapshot is complete, the flow continuously reads the changes that were committed to the transaction log and generates the corresponding insert, update, and delete events. 

Step 1. Create a CDC connection for the source database. Configure the following parameters:

  • Include Databases(s) - the comma-separated list of databases(s) to poll CDC events from.
  • Include Table(s) - the command-separated list of fully qualified table names or regular expressions to match the table names. You can override this in the FROM attribute of the source-to-destination transformation.  If you are planning to poll data from tens or hundreds of tables, consider using regular expressions or {tokenizing} the Whitelisted Table(s) or FROM.

Step 2. Create a Kafka or Azure Event Hub connection.

Step 3. Create JSON (recommended) or Avro format.

Step 4. Start creating a flow by selecting the Stream CDC event to message queue from the gallery. 

mceclip0.png

 Step 5. Add a single source-to-destination transformation where:

  • Source connection - CDC connection created in step 1
  • FROM - the following options are available:
    • * - the system will get the list of the whitelisted tables from the CDC connection, or
    • a regular expression to match the table names, or 
    • a comma-separated list of tables to poll data from, or
    • {token}
  • Destination connection - Kafka or Azure Event Hub connection created in step 2
  • Format- JSON or Avro format created in step 3
  • TO - any name, for example, topic

Example:

  • CDC key - [db]_[table]_cdc_stream
  • Include Table(s)- test.inventory,test.payment,test.customer
  • FROM - *

The following topics will be created and the flow will send CDC events to these topics: 

  • test_invemtory_cdc_stream
  • test_payment_cdc_stream
  • test_customer_cdc_stream

Step 6. Configure mapping and other transformations if needed.

Step 7. Schedule the extract flow. We recommend using a continuous run schedule type. The idea is that the extract flow runs until there are no CDC events to poll, stops for the configured number of seconds, then starts again.

mceclip2.png

The Load

This flow loads data files created by the extract flow into any supported destination.

Just like with extract flow there are two load flows types:

Load data from files

Load data from CSV files when the destination is Snowflake

This flow reads CDC events from the CSV files in local or cloud storage, then transforms and loads data into the Snowflake. 

The staged CSV files created by the extract flow are processed from there, then deleted after the processing.

Step 1. Create a Server storage connection for the staged CSV files. The default location is {app.data}/debezium_data/events.

mceclip4.png

Step 2. Create CSV format with all default settings. Enable Save metadata parameter.

mceclip1.png

Step 3. Create all required for Snowflake flows connections and formats.

Step 4. Create a new flow using flow type file to snowflake 

mceclip2.png

Follow the same steps as suggested in this article.

Step 5.  Add as many source-to-destination transformations as there are distinct filename patterns to poll data from.

For example, if the following files are created by the extract flow:

  • {app.data}/debezium_data/events/test_inventory_cdc_stream_uuid.csv
  • {app.data}/debezium_data/events/test_payment_cdc_stream_uuid.csv
  • {app.data}/debezium_data/events/test_customer_cdc_stream_uuid.csv

you will need 3 pairs of source-to-destination transformations, where the sources (FROM) will need to be set to the following wildcard file names:

  • test_inventory_cdc_stream_*.csv
  • test_payment_cdc_stream_*.csv
  • test_customer_cdc_stream_*.csv

Alternatively, if you want to process all files without creating separate pairs of source-to-destination transformations: 

  • Set the FROM to the wildcard file name that matches all files to load, for example *_*_cdc_stream_*.csv
  • Set the TO to schema.*, for example public.*
  • Set Calculate Destination Object Name under MAPPING->Parameters to
var start = name.indexOf('-');
var end = name.indexOf('_cdc_stream');

value = name.substring(start + 1, end);

You can also use regular expression _(.*?)_cdc_stream to match the table names.

Read more about configuring TO when processing files by a wildcard name.

For each transformation set the following parameters:

  • Source connection - the connection created in step 1.
  • CSV format created in step 2.
  • FROM - a wildcard filename as described above.
  • Destination connection - a connection for the Snowflake stage (internal or external)
  • Destination format - CSV format created in step 2 
  • TO -  a destination table name or a wildcard table name

Step 6. If you are not planning to transform the data before loading into the Snowflake enable Ignore Transformations under Parameters. This will have a positive impact on performance.

mceclip0.png

Step 7. Optionally configure the number of files to process in one batch. It is useful if you expect that a large (many thousands) number of files could be created by the extract flow in a short period of time and don't want the load flow to be overwhelmed. Processing a smaller number of files (hundreds) in micro-batches will be more efficient. 

mceclip2.png

Step 8. Enable Delete loaded source files under Parameters.

Step 9. All other parameters are the same as in the regular Snowflake flow, except the Action, which can be set to one of the following: 

  • CDC MERGE - the system will use the event type stored in the record: "c" for INSERT, "u" for UPDATE and "d" for DELETE. This option requires setting the Lookup fields to the comma-separated list of fields that uniquely identify the record in the target database. Optionally, you can enable the Predict Lookup Fields option. If neither works you can specify the list of table=fields pairs in the Lookup Field. Use the fully-qualified table names and ';' as a separator between table=field pairs.
  • COPY INTO - in some cases, it makes sense to always insert unmodified CDC events into the staging table. Then load events from the staging table(s) into the final table using SQL. When this option is enabled, the following fields will be automatically added to the staging table:
    • debezium_cdc_op - event type, "c" for create, "u" for update and "d" for delete.
    • debezium_cdc_timestamp - the event timestamp. 

Step 10. Schedule load flow to run in parallel with the extract flow. Just like with extract flow we recommend using a continuous run schedule type. 

Load data from CSV files when the destination is Amazon Redshift

This flow reads CDC events from the CSV files in local or cloud storage, then transforms and loads data into the Amazon Redshift. 

The staged CSV files created by the extract flow are processed from there, then deleted after the processing.

Step 1. Create a Server storage connection for the staged CSV files. The default location is {app.data}/debezium_data/events.

mceclip4.png

Step 2. Create a CSV format. Enable Save metadata parameter.

Step 3. Create all required for Redshift flows connections and formats.

Step 4. Create a new flow using flow type file to redshift

mceclip3.png

Follow the same steps as suggested in this article.

Step 5.  Add as many source-to-destination transformations as there are distinct filename patterns to poll data from.

For example, if the following files are created by the extract flow:

  • {app.data}/debezium_data/events/test_inventory_cdc_stream_uuid.csv
  • {app.data}/debezium_data/events/test_payment_cdc_stream_uuid.csv
  • {app.data}/debezium_data/events/test_customer_cdc_stream_uuid.csv

you will need 3 pairs of source-to-destination transformations, where the sources (FROM) will need to be set to the following wildcard file names:

  • test_inventory_cdc_stream_*.csv
  • test_payment_cdc_stream_*.csv
  • test_customer_cdc_stream_*.csv

Alternatively, if you want to process all files without creating separate pairs of source-to-destination transformations: 

  • Set the FROM to the wildcard file name that matches all files to load, for example *_*_cdc_stream_*.csv
  • Set the TO to schema.*, for example public.*
  • Set Calculate Destination Object Name under MAPPING->Parameters to
var start = name.indexOf('-');
var end = name.indexOf('_cdc_stream');

value = name.substring(start + 1, end);

You can also use regular expression _(.*?)_cdc_stream to match the table names.

Read more about configuring TO when processing files by a wildcard name.

For each transformation set the following parameters:

  • Source connection - connection created in step 1.
  • CSV format created in step 2.
  • FROM - a wildcard filename as described above.
  • Destination connection - a connection for the Redshift stage.
  • Destination format - CSV format created in step 2 
  • TO -  a destination table name or a wildcard table name

Step 6. If you are not planning to transform the data before loading into the Redshift enable Ignore Transformations under Parameters. This will have a positive impact on performance.

mceclip0.png

Step 7. Optionally configure the number of files to process in one batch. It is useful if you expect that a large (many thousands) number of files could be created by the extract flow in a short period of time and don't want the load flow to be overwhelmed. Processing a smaller number of files (hundreds) in micro-batches will be more efficient. 

mceclip2.png

Step 8. Enable Delete loaded source files under Parameters.

Step 9. All other parameters are the same as in the regular Redshift flow, except the Action, which can be set to the one of the following: 

  • CDC MERGE - the system will use the event type stored in the record: "c" for INSERT, "u" for UPDATE and "d" for DELETE. This option requires setting the Lookup fields to the comma-separated list of fields that uniquely identify the record in the target database. Optionally, you can enable the Predict Lookup Fields option. If neither works you can specify the list of table=fields pairs in the Lookup Field. Use the fully-qualified table names and ';' as a separator between table=field pairs.
  • COPY - in some cases, it makes sense to always insert unmodified CDC events into the staging table. Then load events from the staging table(s) into the final table using SQL. When this option is enabled the following fields will be automatically added to the staging table:
    • debezium_cdc_op - event type, "c" for create, "u" for update and "d" for delete.
    • debezium_cdc_timestamp - the event timestamp. 

Step 10. Schedule load flow to run in parallel with the extract flow. Just like with extract flow we recommend using a continuous run schedule type. 

Load data from CSV files when the destination is Azure Synapse Analytics

This flow reads CDC events from the CSV files in local or cloud storage, then transforms and loads data into the Azure Synapse Analytics. 

The staged CSV files created by the extract flow are processed from there, then deleted after the processing.

Step 1. Create a Server storage connection for the staged CSV files. The default location is {app.data}/debezium_data/events.

mceclip4.png

Step 2. Create CSV format with all default settings. Enable Save metadata parameter.

mceclip1.png

Step 3. Create all required for Synapse Analytics flows connections.

Step 4. Create a new flow using flow type file to azure synapse 

mceclip0.png

Follow the same steps as suggested in this article.

Step 5.  Add as many source-to-destination transformations as there are distinct filename patterns to poll data from.

For example, if the following files are created by the extract flow:

  • {app.data}/debezium_data/events/test_inventory_cdc_stream_uuid.csv
  • {app.data}/debezium_data/events/test_payment_cdc_stream_uuid.csv
  • {app.data}/debezium_data/events/test_customer_cdc_stream_uuid.csv

you will need 3 pairs of source-to-destination transformations, where the sources (FROM) will need to be set to the following wildcard file names:

  • test_inventory_cdc_stream_*.csv
  • test_payment_cdc_stream_*.csv
  • test_customer_cdc_stream_*.csv

Alternatively, if you want to process all files without creating separate pairs of source-to-destination transformations: 

  • Set the FROM to the wildcard file name that matches all files to load, for example *_*_cdc_stream_*.csv
  • Set the TO to schema.*, for example public.*
  • Set Calculate Destination Object Name under MAPPING->Parameters to
var start = name.indexOf('-');
var end = name.indexOf('_cdc_stream');

value = name.substring(start + 1, end);

You can also use regular expression _(.*?)_cdc_stream to match the table names.

Read more about configuring TO when processing files by a wildcard name.

For each transformation set the following parameters:

  • Source connection - the connection created in step 1.
  • CSV format created in step 2.
  • FROM - a wildcard filename as described above.
  • Destination connection - a connection for the Snowflake stage (internal or external)
  • Destination format - CSV format created in step 2 
  • TO -  a destination table name or a wildcard table name

Step 6. If you are not planning to transform the data before loading into the Synapse Analytics enable Ignore Transformations under Parameters. This will have a positive impact on performance.

mceclip1.png

Step 7. Optionally configure the number of files to process in one batch. It is useful if you expect that a large (many thousands) number of files could be created by the extract flow in a short period of time and don't want the load flow to be overwhelmed. Processing a smaller number of files (hundreds) in micro-batches will be more efficient. 

mceclip2.png

Step 8. Enable Delete loaded source files under Parameters.

Step 9. All other parameters are the same as in the regular Synapse Analytics flow, except the Action, which can be set to one of the following: 

  • CDC MERGE - the system will use the event type stored in the record: "c" for INSERT, "u" for UPDATE and "d" for DELETE. This option requires setting the Lookup fields to the comma-separated list of fields that uniquely identify the record in the target database. Optionally, you can enable the Predict Lookup Fields option. If neither works you can specify the list of table=fields pairs in the Lookup Field. Use the fully-qualified table names and ';' as a separator between table=field pairs.
  • COPY INTO - in some cases, it makes sense to always insert unmodified CDC events into the staging table. Then load events from the staging table(s) into the final table using SQL. When this option is enabled, the following fields will be automatically added to the staging table:
    • debezium_cdc_op - event type, "c" for create, "u" for update and "d" for delete.
    • debezium_cdc_timestamp - the event timestamp. 

Step 10. Schedule load flow to run in parallel with the extract flow. Just like with extract flow we recommend using a continuous run schedule type. 

Load data from CSV files when the destination is a relational database

This flow reads CDC events from the CSV files in local or cloud storage then transforms and loads data into the destination database. 

The staged CSV files created by the extract flow are processed then deleted after the processing.

Step 1. Create a Server storage connection for the staged CSV files. The default location is {app.data}/debezium_data/events.

mceclip4.png

Step 2. Create a CSV format. Enable Save metadata parameter.

mceclip1.png

Step 3. Create a connection to the destination database.

Step 4. Create a new flow using a flow type file to database (assuming that the destination is a database).

mceclip7.png

Step 5.  Add as many source-to-destination transformations as there are distinct filename patterns to poll data from.

For example, if the following files are created by the extract flow:

  • {app.data}/debezium_data/events/test_inventory_cdc_stream_uuid.csv
  • {app.data}/debezium_data/events/test_payment_cdc_stream_uuid.csv
  • {app.data}/debezium_data/events/test_customer_cdc_stream_uuid.csv

you will need 3 pairs of source-to-destination transformations, where the sources (FROM) will need to be set to the following wildcard file names:

  • test_inventory_cdc_stream_*.csv
  • test_payment_cdc_stream_*.csv
  • test_customer_cdc_stream_*.csv

Alternatively, if you want to process all files without creating separate pairs of source-to-destination transformations: 

  • Set the FROM to the wildcard file name that matches all files to load, for example *_*_cdc_stream_*.csv
  • Set the TO to schema.*, for example public.*
  • Set Calculate Destination Object Name under MAPPING->Parameters to
var start = name.indexOf('-');
var end = name.indexOf('_cdc_stream');

value = name.substring(start + 1, end);

You can also use regular expression _(.*?)_cdc_stream to match the table names.

Read more about configuring TO when processing files by a wildcard name.

Step 6. For each transformation set the following parameters:

  • Source connection - the connection created in step 1.
  • Source format - the CSV format created in step 2.
  • FROM- a wildcard filename as described above.
  • Destination connection - a destination database connection created in step 3.
  • TO -  a destination table name or a wildcard table name as described above
  • Enable Delete loaded sources files
  • Set the Action under MAPPING/Parameters- action defines a SQL command to generate. Set Action to one of the following: 
    • Record - the system will automatically generate INSERT, UPDATE or DELETE SQL statement based on the value of the debezium_cdc_op field (c, u, d) populate during extract from the CDC-enabled database
    • Record with MERGE (recommended)- same as Record, except the system, will generate MERGE SQL statement for inserts and updates. Note that not all databases support native MERGE, also some databases, for example, MySQL and PostgreSQL, require a unique index for the fields that are used to uniquely identify the record.
    • Record with IfExist - same as Record with MERGE, except it uses a SELECT statement to check if record exists and then conditionally generates either INSERT or UPDATE. 
  • Set the Lookup fields under MAPPING/Parameters - a comma-separated list of fields that uniquely identify the record. Alternatively, you can enable the Predict Lookup Fields which if enabled will force the flow to use various algorithms to automatically predict the fields that uniquely identify the record. Note that it is not always possible to correctly detect the unique fields. If neither works you can specify the list of table=fields pairs in the Lookup Field. Use the fully-qualified table names and ';' as a separator between table=field pairs.

If the SQL action is set to CDC merge it is possible to configure the load flow to skip DELETE events by enabling the property Do not execute DELETE when Action is Record with MERGE.

It greatly improves the performance of the MERGE because it automatically enables batch processing, otherwise disabled for the CDC merge action.

mceclip9.png

Step 7. Optionally configure the number of files to process in one batch. It is useful if you expect that a large (many thousands) number of files could be created by the extract flow in a short period of time and don't want the load flow to be overwhelmed. Processing a smaller number of files (hundreds) in micro-batches will be more efficient. 

mceclip2.png

Step 8. Schedule load flow to run in parallel with the extract flow. Just like with extract flow we recommend using a continuous run schedule type. 

Load data from a message queue

Load data from a message queue when the destination is Snowflake

This flow reads CDC events from the message queue's topics, then transforms and loads data into the Snowflake.

Step 1. Create a Kafka or Azure Event Hub connection. You can reuse the same connection that you created for the extract flow.

You must enter a unique name in the Group ID field.

Also, consider changing the default values for the following parameters:

  • Number of retries before stop polling - The number of retries before stop polling if poll returns no records. By increasing this number, you can give the flow more time to read the events from the queue.
  • Max number of records to poll - The maximum number of records to poll in one call. By changing this number, you can tune the flow for maximum performance or minimum RAM consumption.
  • Max number of records to read - The total maximum number of records to read from the queue. Set it to a reasonable number to allow the system to process records in micro-batches. If nothing is entered, the system will read records from the queue until there are no more records. By changing this number, you are either increasing or decreasing the size of the micro-batches.

Step 2. Create all required connections and formats.

Step 3. Start creating a flow by typing in queue to snowflake.

mceclip0.png

Step 4. Select Queue to the Snowflake.

Follow the same steps as suggested in this article.

Step 5.  Add as many source-to-destination transformations as there are queue's topics to poll data from. Most likely you are going to have one topic per source table.

For each transformation set the following parameters:

  • Source connection - queue connection created in step 1.
  • Source format - a format used in the extract flow, most likely JSON.
  • FROM- a topic name to poll data from.
  • Destination connection - a connection for the Snowflake stage.
  • Destination format - CSV format 
  • TO -  a destination table name

Step 6. All parameters are the same as in the regular Snowflake flow, except the Action, which can be set to one of the following: 

  • CDC MERGE - the system will use the event type stored in the record: "c" for INSERT, "u" for UPDATE and "d" for DELETE. This option requires setting the Lookup fields to the comma-separated list of fields that uniquely identify the record in the target database. Optionally you can enable the Predict Lookup Fields option. If neither works you can specify the list of table=fields pairs in the Lookup Field. Use the fully-qualified table names and ';' as a separator between table=field pairs.
  • COPY INTO - in some cases, it makes sense to always insert unmodified CDC events into the staging table. Then load events from the staging table(s) into the final table using SQL. When this option is enabled, the following fields will be automatically added to the staging table:
    • debezium_cdc_op - event type, "c" for create, "u" for update and "d" for delete.
    • debezium_cdc_timestamp - the event timestamp. 

Step 7. Schedule load flow to run in parallel with the extract flow. Just like with extract flow we recommend using a continuous run schedule type. 

Load data from a message queue when the destination is Amazon Redshift

This flow reads CDC events from the message queue's topics, then transforms and loads data into the Amazon Redshift.

Step 1. Create a Kafka or Azure Event Hub connection. You can reuse the same connection that you created for the extract flow.

You must enter a unique name in the Group ID field.

Also, consider changing the default values for the following parameters:

  • Number of retries before stop polling - The number of retries before stop polling if poll returns no records. By increasing this number, you can give flow more time to read the events from the queue.
  • Max number of records to poll - The maximum number of records to poll in one call. By changing this number, you can tune the flow for maximum performance or minimum RAM consumption.
  • Max number of records to read - The total maximum number of records to read from the queue. Set it to a reasonable number to allow the system to process records in micro-batches. If nothing is entered, the system will read records from the queue until there are no more records. By changing this number you are either increasing or decreasing the size of the micro-batches.

Step 2. Create all required connections and formats.

Step 3. Start creating a flow by typing in queue to redshift.

mceclip1.png

Step 4. Select Queue to the Redshift.

Follow the same steps as suggested in this article.

Step 5.  Add as many source-to-destination transformations as there are queue's topics to poll data from. Most likely you are going to have one topic per source table.

For each transformation set the following parameters:

  • Source connection - queue connection created in step 1.
  • Source format - a format used in the extract flow, most likely JSON.
  • FROM- a topic name to poll data from.
  • Destination connection - a connection for the Redshift stage.
  • Destination format - CSV format 
  • TO -  a destination table name

Step 6. All parameters are the same as in the regular Redshift flow, except the Action, which can be set to the one of the following: 

  • CDC MERGE - the system will use the event type stored in the record: "c" for INSERT, "u" for UPDATE and "d" for DELETE. This option requires setting the Lookup fields to the comma-separated list of fields that uniquely identify the record in the target database. Optionally, you can enable the Predict Lookup Fields option. If neither works you can specify the list of table=fields pairs in the Lookup Field. Use the fully-qualified table names and ';' as a separator between table=field pairs.
  • COPY INTO - in some cases, it makes sense to always insert unmodified CDC events into the staging table. Then load events from the staging table(s) into the final table using SQL. When this option is enabled, the following fields will be automatically added to the staging table:
    • debezium_cdc_op - event type, "c" for create, "u" for update and "d" for delete.
    • debezium_cdc_timestamp - the event timestamp. 

Step 7. Schedule load flow to run in parallel with the extract flow. Just like with extract flow, we recommend using a continuous run schedule type. 

Load data from a message queue when the destination is Synapse Analytics

This flow reads CDC events from the message queue's topics, then transforms and loads data into the Synapse Analytics.

Step 1. Create a Kafka or Azure Event Hub connection. You can reuse the same connection that you created for the extract flow.

You must enter a unique name in the Group ID field.

Also, consider changing the default values for the following parameters:

  • Number of retries before stop polling - The number of retries before stop polling if poll returns no records. By increasing this number, you can give flow more time to read the events from the queue.
  • Max number of records to poll - The maximum number of records to poll in one call. By changing this number, you can tune the flow for maximum performance or minimum RAM consumption.
  • Max number of records to read - The total maximum number of records to read from the queue. Set it to a reasonable number to allow the system to process records in micro-batches. If nothing is entered, the system will read records from the queue until there are no more records. By changing this number you are either increasing or decreasing the size of the micro-batches.

Step 2. Create all required connections and formats.

Step 3. Start creating a flow by typing in queue to azure synapse.

mceclip2.png

Step 4. Select Queue to Azure Synapse Analytics.

Follow the same steps as suggested in this article.

Step 5.  Add as many source-to-destination transformations as there are queue's topics to poll data from. Most likely you are going to have one topic per source table.

For each transformation set the following parameters:

  • Source connection - queue connection created in step 1.
  • Source format - a format used in the extract flow, most likely JSON.
  • FROM- a topic name to poll data from.
  • Destination connection - a connection for the Synapse Analytics stage.
  • Destination format - CSV format 
  • TO -  a destination table name

Step 6. All parameters are the same as in the regular Synapse Analytics flow, except the Action, which can be set to the one of the following: 

  • CDC MERGE - the system will use the event type stored in the record: "c" for INSERT, "u" for UPDATE and "d" for DELETE. This option requires setting the Lookup fields to the comma-separated list of fields that uniquely identify the record in the target database. Optionally, you can enable the Predict Lookup Fields option. If neither works you can specify the list of table=fields pairs in the Lookup Field. Use the fully-qualified table names and ';' as a separator between table=field pairs.
  • COPY INTO - in some cases, it makes sense to always insert unmodified CDC events into the staging table. Then load events from the staging table(s) into the final table using SQL. When this option is enabled, the following fields will be automatically added to the staging table:
    • debezium_cdc_op - event type, "c" for create, "u" for update and "d" for delete.
    • debezium_cdc_timestamp - the event timestamp. 

Step 7. Schedule load flow to run in parallel with the extract flow. Just like with extract flow, we recommend using a continuous run schedule type. 

Load data from a message queue when the destination is a relational database

This flow reads CDC events from the message queue's topics, then transforms and loads data into the destination database.

Step 1. Create a Kafka or Azure Event Hub connection. You can reuse the same connection that you created for the extract flow.

You must enter a unique name in the Group ID field.

Also, consider changing the default values for the following parameters:

  • Number of retries before stop polling - The number of retries before stop polling if poll returns no records. By increasing this number, you can give the flow more time to read the events from the queue.
  • Max number of records to poll - The maximum number of records to poll in one call. By changing this number, you can tune the flow for maximum performance or minimum RAM consumption.
  • Max number of records to read - The total maximum number of records to read from the queue. Set it to a reasonable number to allow the system to process records in micro-batches. If nothing is entered, the system will read records from the queue until there are no more records. By changing this number you are either increasing or decreasing the size of the micro-batches.

Step 2. Create a connection to the destination database.

Step 3. Start creating a flow by typing in queue to.

mceclip0.png

Step 4. Select Queue to the database (assuming that your destination is a relational database).

Step 5.  Add as many source-to-destination transformations as there are queue's topics to poll data from. Most likely you are going to have one topic per source table.

For each transformation set the following parameters:

  • Source connection - queue connection created in step 1.
  • Source format - a format used in the extract flow, most likely JSON.
  • FROM- a topic name to poll data from.
  • Destination connection - a destination database connection created in step 2.
  • TO -  a destination table name
  • Set the Action under MAPPING/Parameters- action defines a SQL command to execute. Set Action to one of the following:
    • Record - the system will automatically generate INSERT, UPDATE or DELETE SQL statement based on the value of the debezium_cdc_op field (c, u, d) populate during extract from the CDC-enabled database
    • Record with MERGE - same as Record, except the system, will generate MERGE SQL statement for inserts and updates. Note that not all databases support native MERGE, also some databases, for example, MySQL and PostgreSQL, require a unique index for the fields that are used to uniquely identify the record.
    • Record with IfExist - same as Record with MERGE, except it uses a SELECT statement to check if record exists and then conditionally generates either INSERT or UPDATE. 
  • Set the Lookup fields under MAPPING/Parameters - a comma-separated list of fields that uniquely identify the record.
    • Alternatively, you can enable the Predict Lookup Fields which, if enabled, will force the flow to use various algorithms to automatically predict the fields that uniquely identify the record. Note that it is not always possible to correctly detect the unique fields.
    • When enabling the Predict Lookup Fields (which is not always accurate) is not an option you can specify the list of table=fields pairs in the Lookup Field. Use the fully-qualified table names and ';' as a separator between table=field pairs.

      Example:

      test1.inventory=inventory_id,database_name;
      test1.payment=payment_id,database_name;
      test1.rental=rental_id,database_name;

      mceclip0.png

Step 6. Schedule load flow to run in parallel with the extract flow. Just like with extract flow we recommend using a continuous run schedule type. 

Handling SQL NULL and empty ('') values when loading files into databases

If SQL NULL is not the same as empty ('')

Step 1. When creating a CDC connection keep Convert null to empty disabled. Note that it is disabled by default.

mceclip0.png

Step 2. When creating a CVS format for FROM and TO:

  • Enable Convert 'null' to null
  • Disable Convert empty string to null
  • Set Value for null to 'null'

mceclip1.png

Step 3. When creating a flow to load files into Snowflake set String used to convert to and from SQL NULL to ('NULL','null').

mceclip2.png

If SQL NULL is the same as empty ('')

Step 1. When creating a CDC connection enable Convert null to empty. Note that it is disabled by default.

Step 2. When creating a CVS format for FROM and TO:

  • Disable Convert 'null' to null
  • Enable Convert empty string to null
  • Set Value for null empty

Step 3. When creating a flow to load files into Snowflake set String used to convert to and from SQL NULL to ('NULL','null','').

Handling source table DDL changes

If you are using a flow that stores CDC events as CSV files and are expecting that the structure of the source table can be modified when the flow is still streaming data from this table - when creating the CDC connection enable Close CSV file if header has changed.

mceclip3.png      

Preserving the exact data types of the columns in the source table when creating a destination table 

Etlworks can automatically create a destination table when loading CSV files. By default, the flow will sample the file and set the data types of the columns in the destination table. This mechanism is not 100% accurate so the destination table might end up with columns that have different data types compare to the source table.

To prevent this from happening:

Step 1. When creating the CDC connection enable Save Metadata.

mceclip4.png   

Step 2. When creating a CVS format for FROM and TO

  • Enable Save Metadata
  • Keep All fields are strings disabled

mceclip5.png

In addition, when enabled, this feature allows the CDC load flow to automatically set the Lookup Fields required for the CDC MERGE by extracting the information about the primary keys from the source database. It significantly enhances the accuracy of the Predict Lookup fields feature. Basically makes it bulletproof and very low cost.    

Create all columns as strings (VARCHAR) 

Etlworks can automatically create a destination table when loading CSV files. If you wish to create all columns as strings (VARCHAR) then:

Step 1. When creating the CDC connection disable Save Metadata.

Step 2. When creating a CVS format for FROM and TO

  • Disable Save Metadata
  • Enable All fields are strings

Implementing soft deletes

By default, CDC flow creates, updates, and deletes records in the destination table based on the type of the CDC event - 'c' for create, 'u' for update, and 'd' for delete.

If you need to implement the soft delete - add the Extra column with the specific function to the CDC connection. The following functions are available:

  • cdc_boolean_soft_delete -  the boolean column with a user-defined name will be added to the CDC stream. Values returned by the function: true for delete, false for all other events   
  • cdc_timestamp_soft_delete(yyyy-MM-dd HH:mm:ss.SSS) - the timestamp column with a user-defined name will be added to the CDC stream. Values returned by the function: timestamp for delete, null for all other events

mceclip7.png

In this example, the column deleted_at will be added to the CDC stream and its value will be set to the current timestamp with a milliseconds precision.  

If any of the functions above are used the CDC event type is automatically switched to 'u' (update).

Adding columns to the CDC stream

By default, the CDC stream contains all columns in the source table. You can add columns with user-defined names to the stream using the Extra columns property of the CDC connection.

mceclip8.png    

Format of the property: column_name[=function[(parameter)],column_name[=function[(parameter)]

Available functions:

  • cdc_op - the CDC event type: 'c' for create, 'u' for update, and 'd' for delete
  • cdc_key - the CDC key in the user-define format
  • cdc_timestamp(yyyy-MM-dd HH:mm:ss.SSS) - the timestamp of the CDC event. It is not guaranteed that this value will be unique for each event in the sequence of events.
  • cdc_event_seq - the unique 'number' of the event. This number grows sequentially.  
  • cdc_boolean_soft_delete -  the boolean column: true for delete, false for all other events   
  • cdc_timestamp_soft_delete(yyyy-MM-dd HH:mm:ss.SSS) - the timestamp column: timestamp for delete, null for all other events

Changing CDC key and CDC event

The CDC event represents a row (a record) in the CDC stream. The CDC key represents the unique identifier of the group of events. All events with the same CDC key are stored in the same file.

Use Preprocessor to change the CDC event and/or CDC key. 

mceclip0.pngAvailable scripting languages  

  • JavaScript.

Available variables

Getting and setting the values of the column in the event

To get the value of the column, serialized as a string:

var val = event.get('columns_name').asText();  

To set the value:

event.put('columns_name', string);

Possible return values

  • true/false - if false the event will be skipped
  • TypedKeyValue<Boolean, String> -  the key is an equivalent of true/false above, the value is CDC key to replace the automatically calculated CDC key

Example:

var newCdcKey= 'abc';

value = new TypedKeyValue(true, newCdcKey);

Creating Files in a cloud storage

By default, the extract flow creates files with CDC events in the local (server) storage. It is possible to configure the CDC connection to creates files directly in any of the following cloud storage services:

  • Amazon S3
  • Azure Storage
  • Google Cloud Storage

When files are created in a cloud it opens the possibility of using serverless data pipelines for loading data into the databases and data warehouses. For example, read about loading data continuously using  Snowflake Snopilies.  

The other advantage of storing files directly in a cloud is eliminating the step that requires copying files to the cloud in the first place.  For example, Amazon Redshift (or any RDS database) can bulk-load files from the S3. When files are already in S3 there is no need to create an additional flow to move files from the local storage to S3. The same is true for Azure databases and data warehouses, for example, Azure Synapse Analytics can bulk load files from the Azure storage so creating the files in the Azure storage makes the pipeline much faster by eliminating a step that moves the files from the local storage to the Azure Blob.

Read how to configure the storage, other than local (server), when creating a CDC connection.

Creating CSV and JSON files using the specified character encoding

If the source database has tables with data encoded using any character set other than ASCII (for example UTF-8) you can configure the connector to create files using that specific encoding.

mceclip4.png

To preserve the encoding when loading files into the databases you will need to set the same character encoding for the CSV format.

mceclip5.png

If the encoding is not specified or set to "no encoding" the JSON files are created with UTF-8 encoding.

Transaction markers

Consider for instance an e-commerce application that manages purchase orders. In a relational database, the information representing such order will typically be spread across multiple tables, e.g. PURCHASE_ORDER, ORDER_LINE, and SHIPMENT_ADDRESS. To examine the complete state of a purchase order in the database itself, you’d run a JOIN query against the three tables.

Things get more challenging when looking at the change events produced by CDC connectors.

The situation can be addressed by leveraging the new transaction metadata supported by most CDC connectors. When enabled, separate transaction markers will be created at the begin and end of each transaction.

To enable transaction markers, modify the CDC connection by enabling the Provide Transaction Metadata parameter. Optionally set the maximum wait time (in minutes) before forcing the creation of the "commit transaction" marker.

mceclip0.png

The transaction markers are not available for the MongoDB CDC connector. MySQL CDC connector requires enabling the GTID.

How to stop CDC Flow

You can configure the CDC connection to automatically stop streaming the CDC events. The CDC flow will stop if any of the above reaches the configured threshold.

  • Number of retries before giving up - the number of retries before giving up if poll returns no records. The default is 100.
  • Retry N minutes before giving up - the number of minutes to retry before giving up if the poll returns no records.
  • Always stop after N minutes - the number of minutes to run the connector before stopping. The connector will stop even if there are new records in the queue. Note that the connector will NOT stop during the snapshot

If none of the parameters are set the CDC stream will run forever until manually stopped.

To stop CDC flow manually click the Stop/Cancel button.

mceclip1.png

Monitoring CDC Flows

Our ETL engine records real-time metrics for each active CDC stream. The following metrics are available:

  • Last Update (timezone) - the time stamp in the server timezone of the last checkpoint. The metrics are updated every 60 seconds.
  • Last Record Processed (time ago) - the human-readable timestamp of the last processed CDC event. 
  • Last Recorded Latency - the last recorded latency. The latency is the difference between a timestamp of the change event in the source database and a timestamp of when the change event was processed by the CDC connector.
  • Max Latency - the maximum recorded latency since the last restart of the CDC connector.  
  • Min Latency - the minimum recorded latency since the last restart of the CDC connector.  
  • Avg Latency - the average latency since the last restart of the CDC connector.  
  • Total Records Processed - the total number of the change events processed since the last restart of the CDC connector. 
  • Records Processed since Last Update - the number of the change events processed since the last update.  Note that the metrics are updated every 60 seconds.

Accessing CDC Metrics from UI 

To access CDC metrics from UI go to Flows or Schedules and click the Running status link next to the flow or schedule name.

mceclip0.png

Then click the View Running tasks link.

mceclip1.png

The metrics are displayed at the bottom of the Running Tasks window.

mceclip2.png

You can refresh the metrics manually

mceclip3.png

or enable the auto-refresh  

mceclip5.png

Accessing CDC Metrics using API

GET CDC metrics for the specific flow

This API returns metrics for the specific CDC flow.

Authentication

Before making a call to API, the user must receive a JWT token from the authentication API.

Step 1. Use any user with the Administratorrole to call an Integrator authentication endpoint and receive an access token.

Step 2. Use the access token, received in Step 1, to call Integrator API endpoints. The access token must be submitted as a header parameter, as in: Authorization:Bearer access-token, or query parameter, as in: ?Authorization=Bearer%20access-token.

Access tokens are short-lived and self-expiring. An access token gives you access to the APIs for approximately 10 minutes. It is recommended that you refresh the access token before each call to the API.

The API endpoint parameters

  • PATH: /etl/rest/v1/tasks/{audit_id}/?type=stream

EXAMPLE: https://app.etlworks.com/etl/rest/v1/tasks/80610/?type=stream

  • METHOD: GET
  • HEADER: Authorization:Bearer access-token
  • REQUEST BODY: none
  • REQUEST CONTENT TYPE: none

Audit ID

audit_id is a required URL parameter. To get the audit_id for the specific flow use the flow status API. 

Response

The response is a JSON document in the following format. The highlighted are CDC metrics. 

[{
   "requestId": "string",
   "owner": "string",
   "name": "string",
   "started": timestamp_in_milliseconds_since_epoch,
   "duration": number,
   "code": "the formatted CDC metrics",
   "type": "CDC",
   "tenant": "cdc_prod",
   "language": null,
   "latency": number,
   "maxLatency": number,
   "minLatency": number,
   "avgLatency": number,
   "recordsProcessed": number,
   "recordsProcessedSinceLastCheck": number,
   "lastCheck": timestamp_in_milliseconds_since_epoch,
   "lastTimeRecordsReceivedDate": timestamp_in_milliseconds_since_epoch}
]

Response codes

200 for success, 401 for not authorized, 403 for forbidden, and 500 for an internal error.

GET CDC metrics for all running flows

This API returns metrics for all currently running CDC flows.

Authentication

Before making a call to API, the user must receive a JWT token from the authentication API.

Step 1. Use any user with the Administratorrole to call an Integrator authentication endpoint and receive an access token.

Step 2. Use the access token, received in Step 1, to call Integrator API endpoints. The access token must be submitted as a header parameter, as in: Authorization:Bearer access-token, or query parameter, as in: ?Authorization=Bearer%20access-token.

Access tokens are short-lived and self-expiring. An access token gives you access to the APIs for approximately 10 minutes. It is recommended that you refresh the access token before each call to the API.

The API endpoint parameters

  • PATH: /etl/rest/v1/tasks/?type=stream

EXAMPLE: https://app.etlworks.com/etl/rest/v1/tasks/?type=stream

  • METHOD: GET
  • HEADER: Authorization:Bearer access-token
  • REQUEST BODY: none
  • REQUEST CONTENT TYPE: none

Response

The response is a JSON document in the following format. The highlighted are CDC metrics. 

[{
   "requestId": "string",
   "owner": "string",
   "name": "string",
   "started": timestamp_in_milliseconds_since_epoch,
   "duration": number,
   "code": "the formatted CDC metrics",
   "type": "CDC",
   "tenant": "cdc_prod",
   "language": null,
   "latency": number,
   "maxLatency": number,
   "minLatency": number,
   "avgLatency": number,
   "recordsProcessed": number,
   "recordsProcessedSinceLastCheck": number,
   "lastCheck": timestamp_in_milliseconds_since_epoch,
   "lastTimeRecordsReceivedDate": timestamp_in_milliseconds_since_epoch}
]

Response codes

200 for success, 401 for not authorized, 403 for forbidden, and 500 for an internal error.

Supported CDC topologies

Cloud, with optional SSH tunnel

Read more about using SSH tunnel to connect to the on-premise database.

mceclip3.png

Hybrid-cloud with data integration agents

Read more about data integration agents.

mceclip1.png

On-premise

Read more about deployment options.

mceclip4.png

 

Related articles

Comments

0 comments

Please sign in to leave a comment.