The Limits of Etcd
Think of Kubernetes as a city hall. When citizens (users) want something — "I want 3 replicas of my app" — they file a form (a Custom Resource Definition, or CRD), and that form gets stored in the city's filing cabinet (Etcd). Controllers periodically check the cabinet and make the city match what the forms say.
This works beautifully for configuration data that changes infrequently. But what if a citizen walks up to the counter and asks: "What is the current CPU usage of every pod, right now?" You can't answer that by reading a filing cabinet. The answer changes every second, and writing it to the cabinet that fast would jam the entire city hall.
CRDs have a hard architectural constraint: all data lives in Etcd. Etcd is a distributed key-value store optimized for strong consistency and durability — not for high-frequency writes or on-the-fly computation. Consider the math: 1,000 pods, each with CPU and memory metrics updated every second, means 2,000 writes per second to Etcd. Etcd's recommended write throughput is in the hundreds of operations per second for a healthy cluster. Exceeding that causes write latency to spike, which cascades into API server slowness, controller loops stalling, and eventually a cluster that can't reconcile anything.
The Aggregation Layer solves this by letting the Kubernetes API Server act as a smart proxy. Instead of storing data in Etcd, you register a backend server that handles specific API paths. Requests for those paths are forwarded to your server, which can compute answers on the fly, query an external database, or stream data from memory — all while appearing to the user as a native Kubernetes API.
Interviewers frequently ask: "What's the difference between a CRD and an Aggregated API, and when would you choose one over the other?" The crisp answer: CRDs store declarative configuration in Etcd and are reconciled by controllers — use them for desired-state objects. Aggregated APIs serve dynamic, computed, or externally-sourced data that doesn't belong in Etcd — use them when you need custom request handling logic, non-Etcd storage, or sub-resource endpoints like /scale or /status with custom semantics.