Articles in this section

See more

Stream CDC Events into Web Service

  • Updated
Follow

You can use Etlworks CDC Flows to stream real-time database changes—INSERTs, UPDATEs, and DELETEs—directly into Web Service (API). 

How It Works

  • Etlworks captures CDC events directly from the transaction log of the source database.

  • The CDC Flow writes each event directly to API endpoint in real time.

Note: The CDC Flow automatically performs an initial snapshot of all included tables before switching to real-time streaming.

When to Use This Flow Type

Use this Flow type when:

  • You want to stream CDC events directly into a Web Service endpoint.

  • You need real-time replication of relational data into an API.

Prerequisites

Enable CDC in the source database:

Step-by-Step Guide

Step 1. Create a CDC Connection

Create a CDC connection for the source database. This connection will extract changes from the transaction log.

Step 2. Create HTTP connection

Provide URL, method, authentication credentials, and other required connection parameters.

Refer to the HTTP connector documentation for details.

Step 3. Create destination Format

You can use one of the following formats for API payloads:

  • JSON
  • XML,
  • CSV (with any delimiter)
  • Fix-length
  • Avro
  • Parquet

Refer to your API documentation when choosing a format.

Most APIs expect JSON.

Step 4. Create the CDC Flow

Go to Flows > Add Flow.

Choose Flow type:

➝ Stream CDC events into Web Service

Step 5. Configure the Source (FROM)

In the newly created Flow:

  • Source Connection: use the CDC connection from Step 1.

  • FROM: choose which tables to include. You can use:

    • A comma-separated list of fully qualified table names (recommended).

    • An asterisk (*) includes all tables configured in the CDC connection.

    • A Regex expression to match table names.

    • A {token} → use tokenized values for dynamic configurations.

Example 1: Comma-separated tables

Select or enter tables or collections to stream data from.

FROM: inventory.city, inventory.country

Example 2: Asterisk (*)

Include all tables configured in the CDC connection: 

FROM: *

Example 3: Regex match

Include only tables in the inventory schema whose names start with “c”:

FROM: ^inventory\.c[a-zA-Z0-9_]+

Example 4: {token}

Use a token for dynamic inclusion of tables (e.g., provided at runtime):

FROM: {included.tables}

When the Flow runs, {included.tables} is replaced with the actual list or pattern defined in your parameters/configuration.

Step 6. Configure the Destination (TO)

On the right-hand side (destination) of the transformation:

  • Destination Connection: use the HTTP connection from Step 2.

  • Format: use format from Step 3.

  • TO: enter any placeholder value. Data will be written to the API endpoint configured in the HTTP connection.

Step 7. Schedule the CDC Flow

We recommend using a continuous run schedule type.

  • The Flow runs indefinitely or until:

    • There are no more CDC events,

    • An error occurs,

  • After stopping, the Flow automatically restarts after a configured delay.

Understanding the Payload Structure

The payload sent to the Web Service endpoint directly depends on the Structure of the CDC Event setting defined in the CDC connection.

CDC events can include both data and metadata, such as:

  • before: row values before the operation

  • after: row values after the operation

  • source: metadata about the source table and database

  • op: operation type (c = create, u = update, d = delete)

  • ts_ms: timestamp of the event

  • Other optional metadata such as transaction, schema, and fields

IMPORTANT: If the Structure of the CDC Event option is left empty, Etlworks automatically flattens the event and includes only the after block in the payload sent to the API. This behavior is recommended when the destination API expects only the current (post-operation) row state rather than the full CDC event envelope.

Related Articles

Related articles