Articles in this section

Other change replication techniques

  • Updated
Follow

Overview

This article covers two change-replication techniques that fall outside of the primary patterns. Use these only when the primary techniques are unavailable or unsuitable for the source.

For the primary techniques, see:

Full refresh

Reload the entire source dataset on every run. There is no notion of "changed" rows — every row goes through the pipeline, every time.

When to use it.

  • The source is small enough that reloading is acceptable.
  • The source has no last-modified timestamp, no CDC log access, and no Change Tracking.
  • You want the destination to be a guaranteed exact copy after every run, with no risk of missed changes.

Trade-offs. Simplest to set up. Slowest pattern for large sources — volume scales linearly with source size on every run. Truncate-then-load is the typical implementation; some destinations also support full-replace via swap-table or atomic rename.

Replication using database triggers

A database trigger on each source table writes change events to an audit / log table. Etlworks then reads the audit table for changes and applies them to the destination, similar in shape to CDC but driven by triggers instead of the transaction log.

When to use it.

  • You don’t have access to the database's transaction log (so log-based CDC isn’t an option).
  • The source database supports triggers and you have DBA privileges to install them.
  • You can tolerate trigger overhead on writes to the source.

Trade-offs. Triggers add latency to every write on the source table. Requires schema changes (the audit/log table) and DBA cooperation. Maintenance overhead grows with the number of tables replicated. Generally a fallback when log-based CDC isn’t available.

Related articles