Real production systems rarely use a single sampling strategy. The most effective architectures layer multiple strategies to balance cost, coverage, and operational complexity.
Strategy 1: Tiered Sampling by Service Criticality
Not all services need the same sampling treatment:
| Service Type | Strategy | Rationale |
|---|---|---|
| Payment processing | Tail: 100% errors, 1% success | Missing a payment error is unacceptable |
| User authentication | Tail: 100% errors, 5% success | Security events must be captured |
| Product catalog (read-only) | Head: 1% | Errors are visible in metrics; traces are for perf profiling |
| Background jobs | Head: 0.1% | High volume, low debugging value |
| Synthetic monitoring | Head: 0% (drop entirely) | Known-good traffic; no debugging value |
Strategy 2: Priority Sampling
Priority sampling assigns a sampling priority attribute to each trace at the head, which downstream tail samplers use to override their default policies:
sampling.priority = 2: Always keep (user explicitly requested tracing, e.g., viaX-Debug-Trace: trueheader)sampling.priority = 1: Normal sampling rules applysampling.priority = 0: Never keep (synthetic traffic, health checks)
This allows individual requests to opt into full tracing — useful for debugging specific user sessions or testing in production.
Strategy 3: Adaptive Sampling
Adaptive sampling dynamically adjusts the sampling rate based on current conditions:
- Traffic spike: Automatically reduce sampling rate to prevent buffer overflow.
- Error rate spike: Automatically increase sampling rate for the affected service.
- SLO breach: Increase sampling rate for the service contributing to the breach.
The OpenTelemetry Collector's tail_sampling processor supports a rate_limiting policy that implements a form of adaptive sampling by capping the number of traces kept per second, regardless of the configured percentage.
Key Point: The most important hybrid strategy insight is this: use metrics and logs to detect that something is wrong, and use traces to understand why. This means your sampling strategy should prioritize keeping traces that are different from the norm — errors, outliers, and novel patterns — rather than trying to keep a representative statistical sample of all traffic. Metrics give you the statistics; traces give you the stories.