|
High Watermark
(uses a designated field, typically a TIMESTAMP, to track changes in the source)
|
- fast
- no extra moving parts
- works for all data sources, including all databases, files, and APIs
|
|
|
Change Data Capture (CDC)
(uses database redo [transaction] log to track changes in the source)
|
- fast
- no polling from database tables and uses database redo log instead
- supports deletes
- supports [almost] real-time replication
|
- currently supports Postgres, MySQL, SQL Server, Oracle, DB2, AS400, and MongoDB
- some older versions of the databases above do not support CDC
- requires extra set up in the source database
|
|
Change Data Tracking (CT)
(synchronous tracking mechanism, in which the changes on the information will be available directly once the DML change is committed)
|
- fast
- supports deletes
- supports [almost] real-time replication
|
- SQL Server only
- requires extra set up in the source database
|
|
Database Triggers
(uses table(s) updated by the database triggers to track changes in the source)
|
- works for any source database which has triggers
- no extra requirements for the specific version of the database or extra field in each table
|
- requires adding triggers to all database tables
- triggers can negatively impact performance of the source database
|
|
Real-time CDC with Kafka and Debezium
(polls CDC events from the Kafka topic(s) to track changes in the source)
|
- fast
- no polling from database tables
- supports deletes
- supports real-time replication
|
- complicated setup (requires Kafka, Zookeeper, Kafka Connect, and Debezium)
- currently supports Postgres, MySQL, SQL Server, Oracle, DB2, and MongoDB
- some older versions of the databases above do not support CDC
- requires extra set up in the source database
|
|
Full refresh
(always polls the entire data set from the source)
|
- the simplest to setup
- can be quite fast for the relatively small data sets (<100K records)
- works for all data sources
|
- not recommended for large data sets
|