Articles in this section

See more

Stream CDC Events into NoSQL Databases

  • Updated
Follow

You can use Etlworks CDC Flows to stream real-time database changes—INSERTs, UPDATEs, and DELETEs—directly into NoSQL databases such as MongoDB. This is a flexible and scalable pattern ideal for operational analytics, audit trails, or real-time data sync into document-based stores.

Captured CDC events are serialized as JSON and inserted directly into MongoDB collections, where they can be queried, transformed, or accessed by downstream systems.

How It Works

  • Etlworks captures CDC events directly from the transaction log of the source database.

  • The CDC Flow writes each event directly to MongoDB in real time.

  • Each event becomes a JSON document inside a MongoDB collection.

Note: The CDC Flow automatically performs an initial snapshot of all included tables before switching to real-time streaming.

When to Use This Flow Type

Use this Flow type when:

  • You want to stream CDC events directly into MongoDB collections.

  • You need real-time replication of relational data into NoSQL.

  • You are building a microservice architecture where MongoDB acts as the operational datastore.

  • You want to decouple extraction from transformation/loading with NoSQL flexibility.

Prerequisites

Enable CDC in the source database:

Step-by-Step Guide

Step 1. Create a CDC Connection

Create a CDC connection for the source database. This connection will extract changes from the transaction log.

Step 2. Create a MongoDB Connection

Create a connection for MongoDB.

Provide authentication credentials, database name, and other required connection parameters.

Refer to the MongoDB connector documentation for details.

Step 3. Create a JSON Format

CDC events are serialized as JSON documents.

We recommend using a JSON format unless you have specific structure overrides.

Step 4. Create the CDC Flow

Go to Flows > Add Flow.

Choose Flow type:

➝ Stream CDC events into NoSQL database

Step 5. Configure the Source (FROM)

In the newly created Flow:

  • Source Connection: use the CDC connection from Step 1.

  • FROM: choose which tables to include. You can use:

    • A comma-separated list of fully qualified table names (recommended).

    • An asterisk (*) includes all tables configured in the CDC connection.

    • A Regex expression to match table names.

    • A {token} → use tokenized values for dynamic configurations.

Example 1: Comma-separated tables

Select or enter tables or collections to stream data from.

FROM: inventory.city, inventory.country

Example 2: Asterisk (*)

Include all tables configured in the CDC connection:

FROM: *

Example 3: Regex match

Include only tables in the inventory schema whose names start with “c”:

FROM: ^inventory\.c[a-zA-Z0-9_]+

Example 4: {token}

Use a token for dynamic inclusion of tables (e.g., provided at runtime):

FROM: {included.tables}

When the Flow runs, {included.tables} is replaced with the actual list or pattern defined in your parameters/configuration.

Step 6. Configure the Destination (TO)

On the right-hand side (destination) of the transformation:

  • Destination Connection: use the MongoDB connection from Step 2.

  • Format: use the JSON format from Step 3.

  • TO: defines the MongoDB collection name for each table.

Optionally, you can specify the field which will be used to generate the unique value of the MongoDB object id (_id column).

Format: collection_name.unique_field_name

Example: [table].mongo_id

You can use a tokenized pattern for dynamic collection naming (see CDC Key below).

Step 7. Schedule the CDC Flow

We recommend using a continuous run schedule type.

  • The Flow runs indefinitely or until:

    • There are no more CDC events,

    • An error occurs,

  • After stopping, the Flow automatically restarts after a configured delay.

Propagating Deletes to MongoDB

By default, the CDC→MongoDB Flow writes every CDC event — inserts, updates, and deletes — as a new document. The matching MongoDB document is not deleted when the source row is deleted; it stays in the collection alongside the historical events.

If you want a source-side delete to remove the matching MongoDB document, three settings need to line up.

Enable “Perform delete on matching MongoDB document for CDC 'delete' operation”

On the MongoDB streaming destination connection, enable Perform delete on matching MongoDB document for CDC 'delete' operation. This tells the destination to delete the matching document when it receives a CDC event with type d.

Keep “Add Event Type and Unique Sequence” enabled on the CDC source

The destination decides whether to delete by reading the debezium_cdc_op column on each event. That column is only present when the source CDC connection has Add Event Type and Unique Sequence enabled, which is the default. If the option is disabled on the source, the event stream does not carry the op marker, the destination cannot distinguish deletes from inserts and updates, and all delete events are silently dropped. Leave the source setting enabled.

Do not configure soft deletes on the CDC source

Soft deletes — configured by adding any of the cdc_boolean_soft_delete, cdc_long_soft_delete, or cdc_timestamp_soft_delete entries to the source CDC connection's Extra columns field — rewrite delete events as updates before the destination ever sees them. The destination's Perform delete setting then has nothing to act on, and no documents are removed. Pick one model: either soft-delete on the source by writing a marker column, or hard-delete on the destination by leaving soft deletes off and enabling Perform delete. The two are mutually exclusive.

Etlworks' Flow Findings inspector flags both incompatibilities — Add Event Type disabled with destination Perform delete on, and soft deletes configured with destination Perform delete on — as Critical structural issues. See CDC settings reference → Perform delete on matching MongoDB document for the full dependency map.

What is the CDC Key?

The CDC Key defines the collection name in MongoDB where each CDC event is stored.

By default, Etlworks uses:

  • [db]-[table]_cdc_stream → when schema is not available (e.g., MySQL)

  • [db]_[schema]-[table]_cdc_stream → when schema is available (e.g., SQL Server)

You can customize the collection name using tokens.

Available Tokens

  • [db] → Source database name

  • [schema] → Schema name (if available)

  • [table] → Table name

  • [collection] → Collection name (from MongoDB; same as table if not overridden)

Examples

Example 1: Collection with default key

  • Database: test

  • Table: inventory

  • CDC Key: [table]

  • Collection: inventory

Example 2: Collection with user defined key

  • Database: test

  • Table: inventory

  • CDC Key: [table].key

  • Collection: inventory

  • Key: key

Understanding the Document Structure

The document stored in the destination collection directly depends on the Structure of the CDC Event setting defined in the CDC connection.

CDC events can include both data and metadata, such as:

  • before: row before the operation

  • after: row after the operation

  • source: metadata about the event

  • op: operation type (c = create, u = update, d = delete)

  • ts_ms: timestamp

  • transaction, schema, fields, etc. 

IMPORTANT: If the Structure of the CDC Event option is left empty, Etlworks automatically flattens the event and includes only the after block in the document stored in the collection. This configuration is recommended when the destination collection is expected to contain only the current (post-operation) state of each record rather than the full CDC event envelope.

What’s Next

Once CDC events are streamed into MongoDB:

  • Use aggregation pipelines to transform and enrich.

  • Build dashboards, microservices, or alerting systems.

  • Integrate MongoDB with downstream systems using Etlworks Flows.

Related Articles

Related articles