Removing the Friction
If you tell every product team, "You are now responsible for your own data pipelines, storage, access control, monitoring, and schema registry," most of them will refuse — and rightfully so. They are busy building features. They don't want to become experts in Apache Spark cluster tuning, S3 bucket policy syntax, or data catalog APIs.
This is the critical enabler that makes Data Mesh operationally feasible: a Self-Serve Data Infrastructure Platform built by a dedicated Platform Engineering team. Without it, domain ownership becomes an unfunded mandate that collapses under its own weight.
What the Platform Provides
The Self-Serve Platform is an internal developer platform (IDP) specifically for data. It abstracts away infrastructure complexity so that a domain engineer can create, publish, and maintain a Data Product without deep infrastructure expertise.
Concretely, the platform provides:
| Capability | What It Abstracts | Example Implementation |
|---|---|---|
| Storage provisioning | Creating and configuring storage buckets or tables | Terraform module: create_data_product_storage(name, retention) |
| Pipeline scaffolding | Boilerplate for ingestion and transformation jobs | dbt project template, Airflow DAG generator |
| Schema registry | Registering and versioning schemas | Confluent Schema Registry, AWS Glue Data Catalog |
| Access control | Defining and enforcing who can read what | Attribute-based access control (ABAC) via OPA or Lake Formation |
| Data catalog integration | Automatically registering new products | Auto-registration hook on publish |
| Monitoring and SLO tracking | Data freshness, row count, null rate checks | Great Expectations, Monte Carlo, or custom dbt tests |
| Compute | Running transformation jobs without managing clusters | Serverless SQL (BigQuery, Athena), managed Spark (Databricks) |
The "One Config File" Goal
The north star for a Self-Serve Platform is that a domain engineer should be able to define a new Data Product in a single declarative configuration file — similar to how a docker-compose.yml defines a multi-container application — and the platform handles everything else: provisioning storage, registering the schema, setting up access policies, and wiring up monitoring. That one file names the product, its domain and owner, where its output lands and how often it refreshes, the freshness/availability SLA it promises, where its schema is registered, who's allowed to read it (and which of its columns are PII that need masking), and the automated quality checks it must pass before publishing. The platform reads this file, provisions the infrastructure, and enforces the constraints — without the domain team writing a single line of Terraform or IAM policy.