Articles in this section

See more

Monitor CDC buffer pressure and latency

  • Updated
Follow

Overview

Starting with Etlworks 10.0.8, CDC flows expose two kinds of diagnostics:

  • Live buffer pressure — current queue occupancy, estimated bytes, producer backpressure, and destination flush timing, shown in Running Tasks → View and through the cdc-buffers CLI command.
  • Historical latency — periodic CDC latency summary lines written to the execution log, giving a retained series for external monitoring for as long as execution logs are kept.

Both require an engine or Integration Agent with 10.0.8 CDC diagnostics support; older Agents ignore the latency setting and return an error for cdc-buffers.

Live buffer pressure (Running Tasks → View)

Each running CDC stream's task details include a CDC Buffer Pressure section, sampled with the one-minute running-task checkpoint (Refresh shows the last sample):

  • Source Event Queue: current records / capacity and % full. While the connector is starting, restarting, stopping, or busy, it shows unavailable.
  • Source Queue Peak: the highest observed occupancy.
  • Source Queue Bytes (estimated): current / budget and peak — shown only when the connection's maximum queue size in bytes is enabled (it is off by default). When the byte limit is disabled, the value is unavailable — never zero.
  • Source Backpressure: producers currently waiting, cumulative wait count, and completed wait time (a wait still in progress is not included).
  • Destination Flush Calls / Flush Time: call and failure counts, with last/max/total/active durations.

How to read it:

  • The scope is this CDC task's source event queue and explicit destination flush calls. It excludes transaction, JDBC, and internal buffers — queue bytes are estimates, not total connector memory or live heap.
  • Flush calls can be no-ops; counts are not durable commits.
  • With Fast CDC Snapshot, snapshot records bypass the source event queue.
  • Source counters belong to the current queue instance and destination counters to the current CDC run; samples are best effort, not atomic.

The cdc-buffers CLI command

cdc-buffers
  • Super Admin only; read-only; no confirmation.
  • Returns the same diagnostics as JSON for the responding ETL engine node — it is not a cluster or Agent aggregate.
  • The output includes a sharedQueueGroup; do not sum entries in the same group.
  • Unavailable byte measurements are null, not zero. Older engines return an error.

Historical latency in execution logs

Every CDC connection type has a Diagnostics section with CDC Latency Log Interval (minutes) — default 5, 0 disables logging. At each interval the execution log receives a line prefixed CDC latency summary: identifying the connection, connector, stream, flow, and execution, with:

  • Interval counts: streaming events, excluded snapshot events, missing-timestamp events, and clock-skew events.
  • Three latency series, each as last/min/avg/max for the interval (or n/a when the interval was idle):
    • source-to-capture — source timestamp to the capture envelope timestamp;
    • capture-to-consumer — envelope timestamp to the moment the consumer observes the record, including queue wait;
    • source-to-consumer — end to end of the two above.
  • Run-wide peaks with UTC timestamps. Peaks reset at the next CDC run, not at each interval.

How to read it:

  • The measurements are taken when the consumer observes the event — before mapping and before destination commit. Destination commit or delivery latency is not measured.
  • Snapshot events are excluded, and Fast CDC Snapshot records are never counted, so snapshots do not appear as false zero-latency samples. Missing or zero source timestamps are counted separately, and negative clock differences are clamped to zero and flagged.
  • A final partial interval is logged on normal or error shutdown, but is not guaranteed after a forced kill.
  • This is a log series, not a dashboard or time-series store. Search or export the CDC latency summary: lines; history lasts as long as execution logs are retained.

Using the two together

  • A consistently full Source Event Queue with waiting producers means the destination side (mapping plus load) is the bottleneck; correlate with Destination Flush Time and the capture-to-consumer latency series.
  • Rising source-to-capture latency with a quiet queue points at the database or capture side, not Etlworks processing.
  • For memory questions, remember the estimated queue bytes cover one queue only — use the Resource Usage dashboard for process-level evidence.

Related: CDC connection Diagnostics parameters, Snapshot Management.

Related articles