The in-memory buffer is the heart of tail-based sampling — and its most dangerous component. Sizing it incorrectly causes either excessive memory usage (leading to OOM crashes) or data loss (from spans arriving after the decision window closes).
The Memory Math
Every span in flight must be held in memory until its trace is complete. The required buffer size is:
RAM = Throughput (spans/sec) × Avg Span Size (bytes) × Buffer Window (seconds)
Example calculation:
| Parameter | Value |
|---|---|
| Request rate | 10,000 req/sec |
| Spans per trace | 10 |
| Total spans/sec | 100,000 spans/sec |
| Average span size | 1 KB |
| Buffer window | 30 seconds |
| Required RAM | ~3 GB per collector |
This is RAM that cannot be used for anything else. If your traffic spikes to 2× normal, your buffer doubles. If your collector runs out of memory, it crashes — and all buffered traces are lost.
Mitigation strategies:
- Span compression: Compress spans in the buffer (LZ4 or Snappy). Reduces effective size by 3-5×.
- Overflow policies: Configure the collector to fall back to probabilistic head sampling when the buffer exceeds a threshold, rather than crashing.
- Horizontal scaling: Add more Layer 2 collectors. Consistent hashing distributes the buffer load.
The Late Arrival Problem
Setting the buffer window is a balancing act. Consider a 30-second window:
t=0s: Root span (Service A) arrives → buffer starts
t=28s: Most spans arrive and are buffered
t=30s: Buffer window closes → policy evaluated → "No error" → DROP decision made
t=31s: Final span arrives (Service C, with the error) → ORPHANED
The orphaned span has no trace to join (the buffer was already flushed). It either gets discarded or stored as an isolated, unlinked span — useless for debugging.
Factors that cause late arrivals:
- Network jitter: Spans transmitted over UDP (common in high-throughput systems) can be reordered or delayed.
- Slow services: A downstream service that takes 25 seconds to respond will generate its span at t=25s.
- Retry logic: A service that retries a failed call three times before giving up may generate spans over a 60-second window.
- Clock skew: If service clocks are not synchronized (NTP drift), span timestamps may appear out of order.
Tuning guidance:
Set your buffer window to the 99th percentile trace duration of your slowest service, plus a 20% safety margin. Monitor the otelcol_processor_tail_sampling_late_span_goes_to_expired_policy metric in the OpenTelemetry Collector — a non-zero value means you're losing spans to late arrivals and need to increase your window (at the cost of more RAM).
Key Point: The late arrival problem is not just a configuration issue — it's an architectural signal. If your p99 trace duration is 45 seconds, you need a 54-second buffer window, which at 100,000 spans/sec requires ~5.4 GB of RAM per collector. At some point, the cost of tail sampling exceeds the cost of just storing all your traces. This is the calculation that drives many teams toward hybrid strategies: tail sample only the services where it matters most, and head sample (or store everything) for the rest.