Keep the Signal, Drop the Noise
Sometimes you genuinely need high-cardinality data for debugging, but you don't need it stored in your long-term metrics database forever. The stream aggregation pattern lets you preserve aggregate accuracy while dramatically reducing series count.
The idea: instead of storing one series per pod, store one series per deployment. You lose the ability to ask "what is pod X's CPU usage?" but you retain the ability to ask "what is the total CPU usage of the payment-service deployment?" — which is usually the question that matters for alerting and capacity planning.
Stream Aggregation in the OTel Collector
This means chaining two processors: one that deletes the high-cardinality pod_name label, and a metricstransform processor that then aggregates cpu_usage_seconds_total by summing across whatever label set remains (deployment, namespace, node).
Before aggregation: cpu_usage_seconds_total{pod_name="payment-abc123", deployment="payment", namespace="prod"} — one series per pod, potentially thousands.
After aggregation: cpu_usage_seconds_total{deployment="payment", namespace="prod"} — one series per deployment, dozens.
Pre-aggregation with Recording Rules (Prometheus-Native)
If you're already storing high-cardinality data and want to create low-cardinality summaries for dashboards and alerts, Prometheus recording rules let you pre-compute aggregations — a rule named deployment:cpu_usage_seconds_total:sum, evaluated every minute, that sums the per-pod metric by deployment and namespace.
Recording rules write new, low-cardinality series that dashboards and alerts can query cheaply. The original high-cardinality series can then be given a shorter retention period or dropped entirely.
Interviewers sometimes ask: "When would you use stream aggregation in the collector versus recording rules in Prometheus?" Stream aggregation (collector-side) prevents high-cardinality data from ever reaching the TSDB — it's the right choice when you want to protect the database from the load entirely. Recording rules are post-ingestion — the high-cardinality data is already stored, and you're creating derived summaries for query efficiency. If cardinality is causing OOM kills, recording rules alone won't save you; you need to stop the data at the source or collector.