Tail-based sampling inverts the head-sampling model entirely. Instead of deciding at the start whether to keep a trace, it waits until the trace is complete — then decides based on what actually happened.
How It Works
The workflow has four stages:
- Collect everything: Every span from every service is forwarded to a central collector. Nothing is dropped yet.
- Buffer: The collector holds all spans in memory, grouped by TraceID, waiting for the trace to complete.
- Evaluate: Once the trace is complete (or a timeout is reached), the collector applies policies against the full trace data.
- Keep or drop: If the trace matches a "keep" policy (e.g., contains an error, exceeded a latency threshold), all its spans are forwarded to storage. Otherwise, the entire buffer for that TraceID is discarded.
All Spans → [Buffer: 30 seconds] → Policy Evaluation → Keep (errors/slow) or Drop (routine)
What Policies Can Evaluate
Because tail sampling sees the complete trace before deciding, it can make decisions based on:
- HTTP status codes: Keep all traces where any span has
http.status_code >= 500 - Latency: Keep all traces where root span duration > 2000ms
- Specific attributes: Keep all traces where
user.tier = "enterprise"orpayment.amount > 10000 - Exception presence: Keep all traces where any span recorded an exception
- Composite logic: Keep if (error OR slow) AND NOT (synthetic monitoring)
This is the "time machine" quality: you're making a decision in the present based on information from the future (the completed request).
Trade-offs
| Property | Tail-Based Sampling |
|---|---|
| CPU overhead | High — all spans must be processed |
| Memory overhead | High — all in-flight traces must be buffered |
| Captures all errors | ✅ Yes — policy can be 100% for error traces |
| Implementation complexity | High — requires specialized infrastructure |
| Suitable for | Critical services, billing, payments, SLA-bound workloads |
A common interview question: "Why not just use tail-based sampling everywhere?" The answer has two parts. First, cost: tail sampling requires buffering 100% of your traffic in memory before dropping any of it — you pay the memory cost for data you'll ultimately discard. Second, complexity: tail sampling requires a stateful, centralized collector architecture (covered in the next concept). Head sampling is stateless and trivially horizontally scalable. The right answer is usually a combination: head-sample aggressively for routine traffic, and use tail sampling only for services where missing errors is unacceptable.
Key Point: Tail sampling doesn't reduce the collection cost — every span is still generated by your application and transmitted to the collector. It reduces the storage cost by dropping data before it reaches your backend. If your bottleneck is network egress from your application pods, tail sampling doesn't help. You need head sampling (or not instrumenting at all) to reduce that cost.