Tail sampling sounds straightforward in theory. In practice, it introduces a fundamental distributed systems problem: how do you make a decision about a complete trace when its spans arrive at different collector instances?
The Fan-Out Problem
In a production system, you typically run multiple OpenTelemetry Collector instances for redundancy and throughput. Your application services send spans to these collectors using a load balancer — which distributes spans across instances by default.
Consider a trace with three spans:
- Span A (Service 1, root) → arrives at Collector #1
- Span B (Service 2, middle) → arrives at Collector #3
- Span C (Service 3, leaf — contains the error) → arrives at Collector #5
Collector #1 has the root span but doesn't know an error occurred downstream. Collector #5 has the error but doesn't have the root span to reconstruct the trace. Neither collector has enough information to make a correct tail-sampling decision.
The Solution: Two-Layer Collector Architecture
The standard solution is a two-tier collector deployment where the first layer's only job is routing, and the second layer's job is buffering and decision-making.
Layer 1 — The Router (Gateway Collectors):
These collectors receive spans from your application services. They perform one operation: extract the TraceID from each span, hash it, and use consistent hashing to route the span to a specific Layer 2 collector.
destination = hash(TraceID) % num_layer2_collectors
Because the hash function is deterministic, every span with the same TraceID — regardless of which service generated it or which Layer 1 collector received it — will always be routed to the same Layer 2 collector.
Layer 2 — The Brain (Sampling Collectors):
These collectors receive spans that are already pre-grouped by TraceID. Each instance is guaranteed to receive all spans for the traces assigned to it. They maintain the in-memory buffer, wait for trace completion, apply policies, and forward kept traces to the storage backend.
Consistent Hashing and Collector Failures
A critical operational concern: what happens when a Layer 2 collector crashes?
With simple modulo hashing (hash(TraceID) % N), removing one collector changes the routing for all TraceIDs (because N changed). This causes trace fragmentation during rollouts and failures.
Consistent hashing solves this: collectors are arranged on a virtual ring, and each TraceID maps to the nearest collector on the ring. When a collector is removed, only the TraceIDs that were assigned to that collector are redistributed — all others remain stable. This is why the OpenTelemetry Collector's loadbalancingexporter uses consistent hashing internally.
Key Point: The two-layer architecture trades horizontal scalability for operational complexity. You now have two fleets of collectors to manage, size, and monitor. The Layer 2 collectors become stateful — they hold in-memory buffers that cannot be lost without data loss. This means you cannot freely restart them, and you must size them carefully for your traffic volume. Many teams underestimate this operational burden and reach for managed tail-sampling services (like Honeycomb's or Datadog's) instead of running their own.