Imagine a busy airport security checkpoint. Every passenger who walks through generates a log: their name, bag weight, scan result, wait time. On a quiet Tuesday, logging every passenger is fine. But on Thanksgiving eve with 50,000 passengers, you'd drown in paper. The smart move is to log selectively — always log the flagged passengers, and sample a small fraction of the routine ones. That's exactly what distributed tracing sampling does.
The Volume Problem
In a high-throughput microservices system processing 10,000 requests per second, generating a full distributed trace for every single request creates an astronomical amount of data. Consider the math:
- 10,000 req/sec × 10 spans/trace × 1 KB/span = 100 MB/sec of raw trace data
- Over 24 hours: ~8.6 TB/day
That volume creates three compounding cost pressures:
- Storage Cost: Storing billions of spans per day in backends like Jaeger, Tempo, or Honeycomb is expensive — often the dominant observability cost line.
- Network Cost: Sending this data out of your cloud environment incurs egress fees. At cloud provider rates, 8.6 TB/day can cost thousands of dollars per month in egress alone.
- Processing Cost: Your analysis backend needs massive compute to index, compress, and make this data queryable. Query latency degrades as data volume grows.
The Value Discrepancy
99.9% of your requests are likely "HTTP 200 OK" completing in under 200ms. They all look nearly identical. They provide almost no debugging value — you already know your happy path works.
The valuable traces are the 0.1%: the errors, the latency outliers, the requests that triggered a database deadlock, the ones that hit a cold cache. These are the traces you must keep.
Sampling is the discipline of keeping the 0.1% of "signal" while discarding the 99.9% of "noise" — without accidentally throwing away the very data you need most.
Key Point: Sampling is not about losing data — it's about making a deliberate, policy-driven decision about which data has enough value to justify its cost. The goal is to make that decision correctly, not just cheaply.