The Engine: Debezium & Kafka Connect

The Standard Stack

The industry-standard open-source tool for CDC is Debezium, typically deployed as a plugin within Kafka Connect. Understanding each layer's role prevents confusion when things go wrong.

Debezium is a collection of database-specific source connectors. Each connector understands the binary format of one database's transaction log: there is a Postgres connector, a MySQL connector, a MongoDB connector, a SQL Server connector, and others. Debezium handles the low-level work of connecting to the replication slot, decoding binary log entries, and converting them into structured change events.

Kafka Connect is the runtime framework that hosts Debezium connectors. It manages connector lifecycle (start, stop, restart on failure), distributes work across multiple worker nodes for scalability, and tracks offsets — the position in the transaction log that has been successfully processed. If a connector crashes and restarts, Kafka Connect resumes from the last committed offset, ensuring no events are skipped.

Apache Kafka is the message broker that receives the change events. Debezium writes each change event as a message to a Kafka topic. The topic naming convention is typically <server-name>.<schema>.<table> — for example, postgres.public.orders.

Deploying a Debezium Connector

A Debezium Postgres connector is configured via a JSON document posted to the Kafka Connect REST API, naming the connector class, the source database's connection details, which tables to watch (table.include.list), and the logical decoding plugin and replication slot/publication to use.

This tells Debezium to connect to the ecommerce database, watch the orders and customers tables, and use the pgoutput logical decoding plugin.

The Message Format

Every change event Debezium emits follows a consistent envelope structure. Here is an UPDATE event on the orders table:

{
  "op": "u",
  "before": {
    "id": 1001,
    "status": "pending",
    "total_cents": 4999
  },
  "after": {
    "id": 1001,
    "status": "shipped",
    "total_cents": 4999
  },
  "source": {
    "db": "ecommerce",
    "table": "orders",
    "lsn": 24023128,
    "ts_ms": 1678888888888
  },
  "ts_ms": 1678888891000
}

The op field uses single-character codes: c for create (INSERT), u for update, d for delete, and r for read (snapshot). The before and after fields show the complete row state on both sides of the change. The source.lsn field is the Log Sequence Number — the exact position in the WAL where this change was recorded, which is critical for deduplication and exactly-once processing.

Note that for DELETE events, after is null. For INSERT events, before is null. Downstream consumers must handle all four operation types.