Too Much of a Good Thing
Labels are powerful, but they can be catastrophically dangerous if misused. Every unique combination of label key-value pairs creates a brand new time-series that your monitoring database must store, index, and serve queries against.
Cardinality refers to the number of unique values a label can take.
- Low Cardinality (Good): A label like
http_methodhas low cardinality — it only has a handful of possible values:GET,POST,PUT,DELETE,PATCH. This is safe and useful. - High Cardinality (Dangerous): A label like
user_id,session_id,order_id, ortrace_idhas extremely high cardinality. There could be millions or billions of unique values.
The Cost of High Cardinality
If you add a user_id label to your http_requests_total metric and you have 10 million users, you just created 10 million separate time-series for that one metric. Multiply that across dozens of metrics and you have hundreds of millions of time-series.
This is the number one cause of performance issues and exploding costs in monitoring systems. It is called a cardinality explosion. The symptoms are:
- Prometheus running out of memory and crashing.
- Queries taking minutes instead of milliseconds.
- Storage costs growing 10x month-over-month.
- Dashboards timing out or returning errors.
Rule of Thumb: Never put unbounded data — IDs, email addresses, full URLs with query parameters, or any value that grows with your user base — into metric labels. If you need to track per-user behavior, that belongs in logs or traces, not metrics.
A common SRE interview scenario: "Your Prometheus instance is running out of memory and you haven't added any new services. What would you investigate?" The answer starts with cardinality. Use prometheus_tsdb_head_series to see the total number of active time-series, and topk(10, count by (__name__)({__name__=~".+"})) to find which metric names have the most series. Then inspect the labels on those metrics for high-cardinality culprits like IDs or dynamic URL paths.