From "Exhaust" to "Asset"
In the centralized model, data is often treated as exhaust — a byproduct of running the application, like heat from an engine. It gets dumped somewhere and forgotten. Data Mesh treats data as a first-class product, with the same rigor applied to software products: ownership, quality standards, versioning, and user empathy.
A Data Product is not just a table in a database. It is a self-contained package that includes everything a consumer needs to use the data reliably:
- Code: The transformation pipelines, SQL views, or streaming jobs that generate the data from raw sources.
- Data: The actual information — tables, files, streams, or APIs — in a queryable form.
- Metadata: Documentation, schema definitions, data lineage, SLOs (Service Level Objectives), and ownership information.
- Access control: Policies defining who can read the data and under what conditions.
- Quality checks: Automated tests that validate the data meets its stated guarantees before it is published.
Think of a Data Product like a well-maintained npm package or a Docker image: it has a version, a README, a defined interface, and a maintainer. You can depend on it.
The Eight Qualities of a Valid Data Product
For a dataset to qualify as a Data Product, it must meet a set of usability standards. These are sometimes called the "eight qualities" in Data Mesh literature:
| Quality | What It Means | Example |
|---|---|---|
| Discoverable | Registered in a central catalog so consumers can find it | Listed in DataHub or Apache Atlas |
| Addressable | Has a unique, permanent, stable URI or location | s3://data-mesh/checkout/orders-daily/v2/ |
| Trustworthy | Has documented SLAs and automated quality checks | "Updated by 03:00 UTC daily, 99.5% uptime" |
| Self-describing | Includes schema, field definitions, and business context | Avro schema + README with field glossary |
| Interoperable | Uses global standards for formats and identifiers | Parquet format, global customer_uuid field |
| Secure | Access is controlled and auditable | IAM policies, column-level masking for PII |
| Natively accessible | Consumable without special tooling or asking the owner | Direct SQL query or S3 read |
| Valuable on its own | Provides standalone analytical value | Can answer business questions without joining 10 other tables |
Versioning and Breaking Changes
Because other teams depend on Data Products, schema changes must be managed like API versioning. A breaking change — removing a field, changing a data type, renaming a column — requires a new version of the product, not an in-place modification. Consumers subscribe to a specific version and migrate on their own schedule. This is the same contract-based thinking used in REST API design, applied to data.
A common interview question is: "How do you handle schema evolution in a Data Mesh?" The answer mirrors API versioning: use a schema registry (like Confluent Schema Registry for Kafka, or Avro/Protobuf schemas stored alongside the data), publish breaking changes as new versions with a deprecation timeline for old versions, and use backward-compatible changes (adding optional fields) wherever possible. The platform should enforce schema registration as a prerequisite for publishing a new Data Product.