How kubectl top Works
The Metrics Server is the reference implementation of an Aggregated API and the example every Kubernetes engineer should understand deeply.
When you run kubectl top pods, you are not reading from Etcd. Here is exactly what happens:
-
Data collection: The Metrics Server Pod runs a lightweight scraper that calls the
/metrics/resourceendpoint on thekubeletof every node. The kubelet exposes real-time CPU and memory usage for every container it manages. The Metrics Server holds this data in memory — it is intentionally ephemeral and not persisted anywhere. -
API registration: The Metrics Server registers an
APIServicefor the group-versionmetrics.k8s.io/v1beta1. This tells the main API server: "Any request for/apis/metrics.k8s.io/v1beta1/...belongs to me." -
Request handling: When you run
kubectl top pods, kubectl sends a request to/apis/metrics.k8s.io/v1beta1/namespaces/default/pods. The main API server proxies this to the Metrics Server Pod. The Metrics Server looks up its in-memory cache and returns the current values. -
HPA integration: The Horizontal Pod Autoscaler controller uses this same API path to read CPU metrics when deciding whether to scale a Deployment. This is why HPA based on CPU utilization requires the Metrics Server to be installed.
Why this matters architecturally: The Metrics Server demonstrates that the Aggregation Layer is not just a developer convenience — it is load-bearing infrastructure. Removing it breaks autoscaling. This is why production clusters must monitor Metrics Server health as a first-class concern, not as an optional add-on.
The custom metrics extension: The Metrics Server only exposes resource metrics (CPU, memory). For custom metrics (HTTP request rate, queue depth, etc.), Kubernetes defines a separate API group: custom.metrics.k8s.io. Adapters like the Prometheus Adapter implement this group, allowing HPA to scale on arbitrary Prometheus metrics. This is a second aggregated API layered on top of the same mechanism.