When the Upstream Changes
Schema evolution is one of the most operationally painful aspects of running a CDC pipeline. A developer runs ALTER TABLE users ADD COLUMN age INT on the production database. If the pipeline is not designed for this, it breaks: the Debezium message now contains a field the downstream schema does not recognize, and the Sink Connector throws a deserialization error.
Worse, consider a breaking change: ALTER TABLE users DROP COLUMN email. Now Debezium emits messages without a field that downstream consumers expect. Or a column type change: ALTER COLUMN price TYPE BIGINT — the binary encoding in the WAL changes, and a consumer expecting an INT will misparse the value.
The Schema Registry
The standard solution is a Schema Registry — a centralized service that stores and versions the schema of every Kafka topic's messages. The most widely used implementation is Confluent Schema Registry, which supports Apache Avro, Protobuf, and JSON Schema formats.
The workflow for schema evolution:
- Developer runs
ALTER TABLEon the production database. - Debezium detects the schema change in the WAL (the log records DDL changes alongside DML changes).
- Debezium registers the new schema version with the Schema Registry. The registry enforces a compatibility policy — for example,
BACKWARDcompatibility means new schema versions must be readable by consumers using the previous schema version. - If the change is compatible (e.g., adding a nullable column with a default), the registry accepts it. Debezium begins emitting messages with the new schema. Each message carries a schema ID, not the full schema, keeping message size small.
- The Sink Connector reads the schema ID, fetches the new schema from the registry, and automatically runs
ALTER TABLEon the warehouse to add the new column. - If the change is incompatible (e.g., dropping a required column), the registry rejects it, and the pipeline raises an alert before any data is corrupted downstream.
Compatibility Modes
Schema registries support several compatibility modes. Understanding them is essential for operating a CDC pipeline:
| Mode | Meaning | Use Case |
|---|---|---|
BACKWARD |
New schema can read data written with old schema | Safe for adding optional fields |
FORWARD |
Old schema can read data written with new schema | Safe for removing optional fields |
FULL |
Both backward and forward compatible | Most restrictive, most stable |
NONE |
No compatibility checks | Development only — never in production |
The senior insight: schema evolution failures are the most common cause of CDC pipeline outages in production. Enforce FULL compatibility in production, and require schema changes to go through a review process just like code changes. A dropped column in the source database can cascade into hours of pipeline downtime if the registry is not configured to catch it.