You Can't Keep Everything
In a high-traffic system handling tens of thousands of requests per second, generating and storing a full trace for every single request is prohibitively expensive. A single trace might contain hundreds of spans, each with kilobytes of attribute data. At scale, this becomes terabytes of data per day.
Sampling is the practice of selectively keeping only a fraction of traces. The goal is to retain enough data to understand system behavior and catch problems, while discarding the rest to control cost.
The fundamental challenge: you want to keep the interesting traces (errors, slow requests, rare events) and discard the boring ones (successful, fast, typical requests). The two major strategies differ in when they make this decision.
1. Head-Based Sampling
The decision to keep or drop a trace is made at the very beginning of the request — at the "head" — before any processing occurs. Typically this happens at the ingress point (API gateway or the root service).
The most common form is probabilistic (rate-based) sampling: "Keep 5% of all incoming requests, chosen randomly." The sampling decision is encoded in the traceparent header's trace flags bit and propagated to all downstream services, so every service in the chain either records or discards spans for that request consistently.
Pros:
- Extremely low overhead. 95% of traces are discarded immediately, before any spans are even created.
- Simple to implement and reason about.
- Consistent: all spans for a given trace are either kept or dropped together.
Cons:
- Blind to outcomes. You decide to keep or drop a trace before you know whether it will be interesting. A 5% sample will miss 95% of errors, including rare but critical ones.
- Poor for debugging low-frequency issues. If an error occurs in 1 out of 10,000 requests and you're sampling at 1%, you'll statistically miss it almost entirely.
2. Tail-Based Sampling
The decision is made at the very end of the request — at the "tail" — after the entire trace has been collected and the outcome is known.
How it works: The OTel Collector buffers all incoming spans in memory, grouped by Trace ID. Once the trace is complete (or a timeout expires), it evaluates the complete trace against a set of policies:
- "Did any span in this trace have
status=ERROR? → KEEP" - "Did the total trace duration exceed 2 seconds? → KEEP"
- "Did any span have attribute
http.status_code=500? → KEEP" - "Otherwise → DROP (or keep at 1% for baseline statistics)"
Pros:
- Guarantees capture of all interesting traces — errors, latency outliers, and rare events are never missed.
- Highly configurable policies based on actual trace content.
Cons:
- High memory and CPU overhead on the Collector, which must buffer all spans before deciding.
- Requires careful Collector sizing and stateful routing (all spans for a given Trace ID must reach the same Collector instance).
- Adds latency to the export pipeline.
Choosing a Strategy
Most production systems use a hybrid approach: head-based sampling at a low rate (e.g., 1%) for baseline traffic, combined with tail-based sampling rules that override the head decision and force-keep any trace containing an error or exceeding a latency threshold. This gives you cost control for normal traffic and guaranteed capture for anomalies.