Free — no signup required

Introduction: The Multiplication Problem

3 min read

What is Cardinality?

Think of a filing cabinet. If you organize files by continent (7 drawers), that's low cardinality — a small, manageable number of categories. If you organize by individual person's name, you might need millions of drawers. That's high cardinality. In metrics systems, cardinality describes how many unique "drawers" your database needs to maintain.

In the context of time-series databases and metrics, cardinality refers to the number of unique values a label (or dimension) can take. Labels are the key-value pairs attached to metrics that let you slice and filter data — things like method="GET", region="us-east-1", or status_code="200".

  • Low cardinality: A status_code label might have values 200, 404, 500 — a cardinality of ~10.
  • Medium cardinality: A region label across a global deployment might have ~30 values.
  • High cardinality: A user_id label in a system with 5 million users has a cardinality of 5,000,000.

The Multiplication Problem

Time-series databases like Prometheus or VictoriaMetrics store data as individual time series. Each unique combination of metric name and label values is a separate series — its own stream of data points stored in memory and on disk.

The total number of series is the product of all label cardinalities:

total_series = cardinality(label_A) × cardinality(label_B) × cardinality(label_C)

For example, a metric http_requests_total with three labels:

Label Values Cardinality
method GET, POST, PUT, DELETE 4
status_code 200, 404, 500, 503 4
region us-east-1, eu-west-1, ap-south-1 3

Total series: 4 × 4 × 3 = 48. Perfectly manageable.

Now add one label: user_id with 500,000 unique users.

Total series: 4 × 4 × 3 × 500,000 = 24,000,000. Your Prometheus server just ran out of RAM.

Cardinality explosion is what happens when a high-cardinality label — like user_id, ip_address, trace_id, or session_token — is attached to a metric. The database must create and track a new series for every unique value it sees. Memory fills up, ingestion slows, and eventually the process is killed by the operating system (OOM Kill — Out of Memory Kill).

The Churn Problem: A Hidden Multiplier

Cardinality has a second dimension that makes it worse: churn. Churn is the rate at which series are created and abandoned. In Kubernetes, containers restart constantly. Each restart generates a new container_id. Even if you only ever have 100 containers running at once, after a week of hourly restarts you've created 100 × 24 × 7 = 16,800 unique series. Prometheus must keep stale series in its index for a configurable retention window (default: 5 minutes for staleness, but the index entry persists longer). High churn inflates the in-memory index even when active series counts look reasonable. This is why ephemeral identifiers — container IDs, pod names with random suffixes, request IDs — are particularly dangerous as metric labels.

Interview Tip

Interviewers frequently ask: "What's the difference between high cardinality and high churn, and why does each matter?" High cardinality means many simultaneously active series — this directly inflates memory usage right now. High churn means many series are created and discarded over time — this inflates the index and causes write amplification even if the instantaneous active series count is low. A system can have low cardinality but high churn (e.g., short-lived batch jobs with sequential IDs), or high cardinality but low churn (e.g., a stable set of 1 million users who never change). Both are dangerous, but for different reasons.

This is one of 18 chapters

Every chapter is free — Kubernetes, Terraform, SRE, distributed systems, and more — with fast daily review built in.

Sign up free