CDC: The Missing Manual

A practitioner's guide to change data capture: what the events mean, what breaks in production, and how to build sinks that stay correct.

Read the manual · Start with the introduction · Try the playground

CDC: The Missing Manual (also called Let's Talk CDC) is a free, no-sign-up guide for engineers who run, or are about to run, change data capture pipelines. If you can read SQL and have seen a message queue, you are ready.

What you'll learn

Change data capture (CDC) reads a database's transaction log and turns each committed row change into an event that other systems can consume: a warehouse, a cache, a search index, another service. It often replaces nightly batch jobs with a stream. Capturing the changes is the easy part. Applying them correctly downstream is where pipelines go wrong.

The manual argues one thesis, and every module comes back to it:

1. Delivery is at-least-once. Expect duplicates, replays and connector restarts.
2. Correctness lives in the sink. Write idempotently, keyed on the primary key and ordered by log position (LSN, SCN, GTID), not by ts_ms.
3. End-to-end exactly-once across independent systems is not achievable. What you can build is exactly-once processing: at-least-once transport plus an idempotent sink.

Here is a change event, in the shape the introduction's event demo produces (the event envelope module explains each field):

{
  "op": "u",
  "ts_ms": 1724130000137,
  "source": { "db": "shop", "table": "customer", "lsn": "0/1A2B498" },
  "before": { "id": 1, "name": "Ada", "tier": "free" },
  "after": { "id": 1, "name": "Ada", "tier": "pro" }
}

ts_ms is a wall-clock time. Clocks drift, and two commits can share a millisecond. The lsn is the change's position in the source's log, and that is what you order by.

Start here

Take it in this order if you are new to CDC:

1. Interactive Introduction: the core ideas, with a live event demo.
2. Event Envelope: what is inside an event and what delivery guarantees mean.
3. Exactly-Once Semantics: why the sink has to be idempotent.
4. Snapshotting and Materialization 101: the first sync, then applying changes.
5. A Quickstart for Postgres, MySQL, Oracle or SQL Server.
6. Failure Drills: break it on purpose and recover.

All 37 modules, by stage

Module	Level	What it covers
Core concepts
Interactive Introduction	Beginner	The core ideas, methods, architectures and tooling
Event Envelope	Beginner	Keys, before/after images, tombstones, delivery guarantees
Materialization 101	Intermediate	MERGE patterns for upserts and deletes
Snapshotting	Intermediate	The initial consistent snapshot before live changes
CDC Beyond Relational Databases	Intermediate	MongoDB, DynamoDB Streams and Cassandra
Advanced patterns
Exactly-Once Semantics	Advanced	At-least-once versus exactly-once, the outbox
Is CDC Exactly-Once? Per Hop	Advanced	Check any exactly-once claim hop by hop
CDC Without Kafka	Advanced	Debezium Server, the embedded engine, managed services
Transactional Outbox and Relay	Advanced	Why dual writes fail, and how consumers dedupe and order
Partitioning	Advanced	Partition keys, skew, late arrivals
Which Row Wins in Your Target	Advanced	Ordering and delete markers in BigQuery, Iceberg, Delta, Snowflake
Schema Evolution	Advanced	Compatibility rules and schema registries
Data Contracts for Database Events	Advanced	What a consumer may rely on, and a gate that catches breaks
Deletes That Stay Deleted	Advanced	Tombstones, delete markers and late-update resurrection
Multi-Tenancy	Advanced	Isolation patterns, topic math, egress estimates
Reconciliation & Offset Surgery	Advanced	Repairing sinks, resetting offsets safely
Running it
Offsets & Replays	Intermediate	Offset stores, safe rewind, resync drills
Observability	Intermediate	Lag, throughput, error rate, minimal dashboards
Postgres Replication Slots & WAL Growth	Intermediate	A runbook for a disk filling behind a CDC slot
SQL Server & MySQL CDC Specifics	Intermediate	Positions, retention, binlog purge, GTIDs and LSNs
Backfill and Re-Snapshot Safely	Advanced	Reload history without overwriting newer changes
Security, PII & Access Control	Intermediate	Masking, least privilege, logs that outlive rows
Context
Use Cases	Beginner	Real-time analytics to cache invalidation
The Strategic Value of CDC	Beginner	The business case
The CDC Ecosystem	Beginner	Open-source and commercial tools
Real-World Case Study	Intermediate	From batch ETL to CDC, with the trade-offs
Hands-on
Quickstarts	Beginner	Pick a source database, 10 to 20 minutes
Kafka + Debezium Lab	Intermediate	Kafka, Connect, a Postgres source and sink
Cloud CDC Labs	Intermediate	AWS DMS, Snowflake and Matillion end to end
Acceptance Tests	Intermediate	Scripts that check your lab survives restarts
Testing a CDC Pipeline	Intermediate	Contract, duplicate, out-of-order and replay tests
Failure Scenario Drills	Advanced	Backpressure, DLQs, schema drift, offset replays
Tools and errata
Connector Config Builder	Intermediate	Debezium configs for Postgres, MySQL or Oracle
Debezium Event Decoder	Intermediate	Before/after diffs and MERGE-ready SQL
DLQ Triage Assistant	Advanced	Commands and playbooks for re-driving DLQ events
Nuances & Errata	Advanced	Corrections and sharp edges
Guided Paths	Beginner	Three routes through the lessons, with local progress

The merge cookbook is a reference page with sink MERGE templates. It sits alongside the modules and is not counted among them.

Try it

Many modules end with a short quiz, and your progress is kept in your browser on the progress page. Press / (or tap Search) on any page to search.

• Event demo. Insert, update and delete rows in a small customer table and watch the change event each one emits.
• Change Feed Playground. A browser simulator. Model a table, insert, update and delete by primary key, emit a snapshot, then inspect the Debezium-style events or copy them as NDJSON.
• Debezium tools. Build a connector config, decode a pasted event, or work through a dead-letter queue with the Connector Config Builder, Debezium Event Decoder and DLQ Triage Assistant (see the full module list above).
• Run a real stack. docker compose up -d starts Postgres, MySQL, Kafka, Debezium Connect and Kafka UI on your machine. See the sandbox guide.

It works on a phone, and has a light and a dark theme:

Reading paths

• I have ten minutes. Play with the event demo, read Exactly-Once Semantics, and keep the glossary open.
• I run Debezium in production. Start with the first 15 minutes of an incident, then snapshotting, schema evolution, offsets and replays, security and the errata. The merge cookbook has sink templates.
• I'm choosing a tool. Read the platform comparison (read the delivery-semantics row first), then the ecosystem overview, the strategy module and the case study.

Why trust it

• Vendor-neutral first. Concepts come in general terms, then map to stacks such as Debezium, Kafka, Snowflake and Matillion.
• Corrections are public. See the errata and the methodology page on how claims are checked.
• Checked by machines. Every change runs unit, pa11y, axe and Lighthouse checks, and a link check crawls the built site.
• Minimal, disclosed data collection. The site's own code sets no cookies. When the maintainer enables it, it counts page views with GoatCounter (no cookies, no cross-site tracking, Do Not Track respected; not loaded on the playground). An optional newsletter signup sends the address you type to Buttondown only when you press Subscribe. The assistant's 👍/👎 and the playground's saved scenarios and change events go to a Supabase database, and the playground loads the Supabase library from a CDN (Mermaid and Chart.js are self-hosted). Everything is listed on the privacy page, which is the source of truth.
• The assistant is optional. A thumbs-up or thumbs-down can store your last typed question with the vote, so don't paste secrets into it (privacy note, SECURITY.md).

Found a mistake? Want to help?

If something is wrong, unclear or out of date, please open an issue or start a discussion. Corrections with a source are the most useful kind.

Developers and contributors: start with docs/DEVELOPMENT.md for running the site locally, then docs/CONTRIBUTING.md. Reviewing the whole project? Start with the autopsy briefing pack.

Author and license

Written and maintained by Christopher Ennis (GitHub).

Code is MIT (LICENSE). Written lessons, diagrams and images are CC BY 4.0 (LICENSE-CC-BY-4.0.txt). Which files fall under which licence is in LICENSE-CONTENT.md. To reuse the content, attribute it like this:

CDC: The Missing Manual by Christopher Ennis (Let's Talk CDC), https://sandgraal.github.io/letstalkcdc/, licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)