Mitigation Strategy 1 — Remove the Label

Fix It at the Source (Preferred)

The cleanest fix is to remove the high-cardinality label from the instrumentation code. This requires a code change, but it eliminates the problem permanently.

Bad instrumentation (Python example):

# This creates one series per user — catastrophic at scale
http_requests_counter.labels(
    method="GET",
    status_code=200,
    user_id=request.user.id   # ← HIGH CARDINALITY
).inc()

Good instrumentation:

# Track what matters at the metric level; use logs/traces for per-user data
http_requests_counter.labels(
    method="GET",
    status_code=200,
    region=request.headers.get("X-Region", "unknown")  # ← LOW CARDINALITY
).inc()

The rule of thumb: if a label value could be unique per request, it doesn't belong on a metric. Per-request data belongs in logs (structured) or traces (distributed). Metrics are for aggregated, population-level signals.

Fix It at the Collector (When You Can't Change the Code)

If the high-cardinality label comes from a third-party library, a vendor agent, or a service you don't control, you can strip the label in your OpenTelemetry Collector before it reaches the database.

The attributes processor with action: delete removes a label key from every data point passing through the pipeline — listing user_id, client_ip, and trace_id as keys to delete strips all three from every metric that passes through.

What happens to the data: When Prometheus receives http_requests_total{method="GET", status_code="200"} (without user_id), all the requests that previously created separate per-user series now merge into a single series. You lose the per-user breakdown in metrics (which you shouldn't have had there anyway), but the total counts remain accurate.

The metricstransform processor offers more surgical control — you can rename labels, merge label values, or apply regex transformations — but for simple deletion, attributes is sufficient.

Prometheus metric_relabel_configs (Drop at Scrape Time)

If you're scraping a target directly with Prometheus (no OTel Collector in the path), you can drop labels using relabeling rules in your scrape config — a metric_relabel_configs entry with a labeldrop action and a regex matching user_id|client_ip|trace_id strips all three from every scraped series.

This runs after scraping but before storage — the label is dropped before any series are written to the TSDB.