Treating Data Like an API
When two software teams share an API, they write a contract: "This endpoint accepts a JSON body with these fields, these types, and returns this response." If one team changes the API without updating the contract, the build breaks immediately.
A Data Contract applies the same discipline to data. It is a formal, versioned agreement between a Data Producer (e.g., the Billing Team writing events to Kafka) and a Data Consumer (e.g., the ML Team reading those events to train a fraud model).
A complete data contract specifies:
| Dimension | Example |
|---|---|
| Schema | Column names, data types, nullability |
| Semantics | "price is in USD cents, not dollars" |
| SLAs | "Data arrives by 08:00 UTC daily" |
| Volume | "Between 10,000 and 500,000 rows per day" |
| Ownership | "Billing Team owns this; contact @billing-oncall" |
The Enforcer: Schema Registry
A Schema Registry is a centralized server that stores and enforces schema contracts. The most widely used implementation is Confluent Schema Registry, which integrates with Apache Kafka. Here is how it works in practice:
-
Strongly Typed Formats: Producers serialize data using Apache Avro or Protocol Buffers (Protobuf) — binary formats with explicit schemas — rather than loose JSON where any field can be any type.
-
Schema Registration: When a Producer first publishes a message type, it registers the schema with the Registry. The Registry assigns it a unique ID and version number.
-
Compatibility Check on Write: Every time a Producer publishes a message, the Kafka client library automatically checks the message schema against the registered schema. If the Producer tries to make a breaking change — deleting a required field, changing a field's type — the Registry rejects the write before the message enters the topic.
-
Consumer Safety: Consumers can always deserialize messages because the schema is guaranteed to be compatible with what they registered against.
The key insight: the Producer breaks, not the Consumer. The failure is loud, immediate, and happens at the source — exactly like a broken API.
Compatibility Modes
Schema Registries support configurable compatibility rules. Understanding these is critical for production systems:
| Mode | Rule | Use Case |
|---|---|---|
BACKWARD |
New schema can read old data | Default; safe for consumers |
FORWARD |
Old schema can read new data | Safe for producers |
FULL |
Both backward and forward | Maximum safety |
NONE |
No compatibility check | Development only — dangerous in production |
BACKWARD compatibility is the default and most common. It means you can add optional fields (consumers that don't know about them will ignore them) but you cannot delete required fields or change types.
Interviewers frequently ask: "How do you prevent upstream schema changes from breaking your data pipeline?" The weak answer is "we have monitoring." The strong answer describes a Schema Registry with backward-compatible enforcement: "We use Confluent Schema Registry with Avro. Producers cannot publish a breaking schema change — the client library rejects the write before the message enters Kafka. This means the failure is at the source, not discovered hours later when a downstream job fails."