Who Is Eating My RAM?
When your Prometheus server starts struggling — slow queries, high memory, scrape timeouts — you need to find the metric causing the explosion. The good news: Prometheus exposes its own internal metrics, and you can use PromQL to investigate (assuming the server is still responsive enough to answer queries).
Diagnostic Query 1: Find the Highest-Cardinality Metrics
Taking the top 10 metric names ranked by their active series count returns exactly this list. How to read the output: Each result shows a metric name and its series count. A healthy metric might have dozens to thousands of series. If you see a metric with 500,000+ series, that's your suspect.
Diagnostic Query 2: Find the Offending Label
Once you've identified the metric, counting series grouped by each of its labels individually reveals which one is causing the explosion — if user_id returns millions of distinct results while the others return a handful, you've found the culprit.
Diagnostic Query 3: Use the TSDB Status Endpoint
Prometheus exposes a built-in cardinality analysis page at its /tsdb-status endpoint. This page shows:
- Top 10 metrics by series count — which metrics have the most time series
- Top 10 label names by series count — which label names appear across the most series
- Top 10 label values by series count — which specific label values create the most series
This is often faster than writing PromQL when the server is under stress, because it reads from a pre-computed index rather than scanning all series.
A common SRE interview scenario: "Your Prometheus server is OOM-killing every 6 hours. Walk me through your investigation." The answer should follow this sequence: (1) Check /tsdb-status to identify the high-cardinality metric without running expensive PromQL queries on a struggling server. (2) Use topk(10, count by (__name__) ({__name__=~".+"})) to confirm. (3) Drill into the specific metric's labels to find the offending dimension. (4) Trace the label back to the instrumentation code or the scrape target. Interviewers want to see a systematic approach, not guessing.