Articles in this section

See more

Stream CDC Events into Files (CSV, JSON)

  • Updated
Follow

You can use Etlworks CDC Flows to capture database changes — INSERT, UPDATE, and DELETE operations — and store them as CSV or JSON files in local or cloud storage. This flexible, infrastructure-light approach eliminates the need for intermediate systems like message queues.

Captured CDC events can then be processed using separate load Flows for transformation and delivery to final destinations (e.g., data warehouse, analytics platform, or API).

How It Works

  • Etlworks captures CDC events directly from the transaction log of the source database.

  • The CDC Flow writes these events to disk as CSV or JSON files.

  • A separate Flow can:

    • Load the generated files,

    • Transform the data,

    • Deliver it to the final destination,

    • Optionally delete the files after processing.

Note: The CDC Flow automatically performs a one-time snapshot of all included tables before switching to real-time streaming.

When to Use This Flow Type

Use this Flow type when:

  • You need a file-based handoff of CDC events (for audit, archiving, or downstream processing).

  • You want to decouple extraction from transformation/loading.

  • You are working with data lakes or object storage platforms (e.g., S3, GCS, Azure).

  • You want maximum control over how and when CDC events are loaded into downstream systems.

Prerequisites

Enable CDC in the source database:

Step-by-Step Guide

Step 1. Create a CDC Connection

Create a CDC connection for the source database. This connection will extract changes from the transaction log.

Step 2. Choose the Destination Connection

Create a file storage connection for where you want the CDC files to go:

Step 3. Create the CDC Flow

  1. Go to Flows > Add flow.

  2. Choose Flow type:

    ➝ Stream CDC events, Create Files

Step 4. Configure the Source (FROM)

In the newly created Flow:

  • Source Connection: use the CDC connection from Step 1.

  • FROM: choose which tables to include. You can use:

    • A comma-separated list of fully qualified table names (recommended).

    • An asterisk (*) includes all tables configured in the CDC connection.

    • A Regex expression to match table names.

    • A {token} → use tokenized values for dynamic configurations.

Example 1: Comma-separated tables

Select or enter tables or collections to stream data from.

FROM: inventory.city, inventory.country

Example 2: Asterisk (*)

Include all tables configured in the CDC connection:

FROM: *

Example 3: Regex match

Include only tables in the inventory schema whose names start with “c”:

FROM: ^inventory\.c[a-zA-Z0-9_]+

Example 4: {token}

Use a token for dynamic inclusion of tables (e.g., provided at runtime):

FROM: {included.tables}

When the Flow runs, {included.tables} is replaced with the actual list or pattern defined in your parameters/configuration.

Step 5. Configure the Destination (TO)

In the right-hand side (destination) of the transformation:

  • Set the Destination Connection (from Step 2).

  • Optionally:

    • Set Format (CSV or JSON).

    • Override CDC Key (file naming template) in TO

Step 6. Schedule the CDC Flow

We recommend using a continuous run schedule type.

  • The Flow runs indefinitely or until:

    • There are no more CDC events,

    • An error occurs,

  • After stopping, the Flow automatically restarts after a configured delay.

What is the CDC Key?

The CDC Key defines the base filename for CDC event files created by the Flow.

Each file represents a stream of changes (INSERT, UPDATE, DELETE) for a specific table, and the CDC Key controls how those files are named.

By default, Etlworks uses one of the following CDC Key patterns:

  • [db]-[table]_cdc_stream — used when the schema is not relevant or available (e.g., in MySQL)

  • [db]_[schema]-[table]_cdc_stream — used when the schema is part of the fully qualified table name (e.g., in SQL Server)

Available Tokens

You can customize the CDC Key pattern using the following tokens:

  • [db] → Database name

  • [schema] → Schema name (if available)

  • [table] → Table name

  • [collection]→ Collection name (if available, e.q. in MongoDB)

Examples

Example 1: MySQL (no schema)

  • Database: test

  • Table: inventory

  • CDC Key: [db]-[table]_cdc_stream

  • Output files: /basefolder/test-inventory_cdc_stream_uuid.csv

Example 2: SQL Server (with schema)

  • Database: test

  • Schema:  dbo

  • Table: inventory

  • CDC Key:[db]_[schema]-[table]_cdc_stream

  • Output files: /basefolder/test_dbo-inventory_cdc_stream_uuid.csv

Example 3: Customized CDC Key

  • Database: test

  • Table: inventory

  • CDC Key: mycompany_[table]_events
  • Output files: /basefolder/mycompany_orders_events_uuid.csv

Understanding File Output and Rotation

Controlling Files Size

When streaming CDC events into CSV or JSON files, Etlworks continuously writes incoming change events to one or more files based on configurable limits and timing rules.

The size and structure of output files are controlled by these settings:

  • Max Rows in File — defines the maximum number of records per file.

  • Max File Size in Bytes — defines the maximum file size in bytes.

Etlworks automatically creates a new file when either limit is reached.

We recommend keeping the default values unless your destination system or storage limits require smaller file segments.

Controlling Frequency 

The frequency of file rotation and how long a file remains open before being finalized are controlled by the following options:

  • Keep File Open (ms) — specifies how long Etlworks keeps a file open to accumulate events before writing and closing it.

  • Close File After (ms) — sets the maximum duration a file can remain open, even if thresholds are not reached.

Lowering these values will result in more frequent file creation, which can be useful when downstream consumers require near real-time access to small batches of data.

Controlling File Format

In addition to file size and timing, several advanced options affect the content and compatibility of generated CDC files:

What’s Next

Once the CDC events are written as files, you can:

  • Use ETL or Bulk Load flows to ingest and transform them.

  • Build pipelines that perform validation, enrichment, or merging.

  • Route files to different destinations based on table or operation type.

Related Articles

Related articles