When CRDs aren't enough
CRDs are the right tool for the vast majority of Kubernetes extension use cases. But they have architectural constraints that occasionally make them the wrong choice:
- etcd as the only storage backend: All custom resource objects are stored in the main cluster etcd. etcd is optimized for small, frequently-read configuration data — not for storing millions of time-series data points or large binary blobs. Storing high-volume metrics data in etcd would degrade the entire cluster's control plane.
- Standard REST semantics only: CRDs support the standard Kubernetes CRUD operations (create, read, update, delete, list, watch). If you need non-standard verbs, streaming responses, or protocol-level customization, CRDs can't express that.
- Validation limits: While OpenAPI v3 schema validation is powerful, some complex cross-field validation logic is difficult to express declaratively.
How Aggregated API Servers work
An Aggregated API Server (AA Server) is an entirely separate HTTPS web server that you write, deploy, and run alongside the main Kubernetes API server. You register it by creating an APIService object naming the target group/version and the in-cluster service to proxy to, which tells the main API server: "For any request to URLs under /apis/metrics.k8s.io/v1beta1/, proxy that traffic to my separate server." The most well-known example is metrics-server, which implements the metrics.k8s.io API. When you run kubectl top pods, kubectl queries /apis/metrics.k8s.io/v1beta1/pods — which the main API server proxies to the metrics-server, which in turn queries the kubelet on each node for live CPU and memory data. None of that data ever touches etcd.
CRD vs. Aggregated API Server: when to choose which
| Concern | CRD | Aggregated API Server |
|---|---|---|
| Implementation complexity | Low — just YAML | High — full Go web server |
| Storage backend | etcd only | Any (PostgreSQL, InfluxDB, etc.) |
| API semantics | Standard CRUD + watch | Fully custom |
| Authentication/authorization | Handled by main API server | Must implement or delegate |
| Best for | Custom application config, Operators | High-volume data, custom protocols |
In practice, unless you have a specific requirement that CRDs cannot satisfy, start with a CRD. Aggregated API Servers are the right tool for infrastructure-level extensions (like metrics, custom schedulers, or multi-cluster APIs) where the constraints of etcd-backed storage or standard REST semantics are genuinely limiting.