Before choosing when to sample, you must understand the constraint that governs how sampling must work. This constraint is called coherency, and violating it renders your traces useless.
"All or Nothing"
A distributed trace is a directed acyclic graph (DAG) of connected Spans. Each span represents one unit of work (an HTTP call, a database query, a function). Spans are linked by a shared TraceID and parent-child SpanID relationships.
The golden rule: either keep all spans for a trace, or keep none of them.
Consider a three-service call chain: Service A → Service B → Service C.
- Bad Sampling: Service A records its span. Service B independently decides not to record. Service C records its span.
- Result: You have a "broken trace." You see the entry point and the exit point, but the middle is missing. The parent-child links are severed. You cannot reconstruct the call graph, measure inter-service latency, or identify where time was spent. This data is actively misleading.
Decision Propagation
To guarantee coherency, the sampling decision must be made exactly once — typically at the trace's origin — and propagated to every downstream service via the trace context headers.
The W3C TraceContext standard (the industry standard for trace propagation) defines a traceparent header that includes a trace-flags byte. Bit 0 of that byte is the sampled flag:
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
^^
trace-flags: 01 = sampled
The propagation contract works like this:
- Service A (the root) makes the sampling decision: "Keep this trace."
- Service A sets
trace-flags=01in thetraceparentheader it sends to Service B. - Service B reads the header, sees
sampled=1, and is obligated to record its span — regardless of its own local sampling configuration. - Service B propagates the same flag to Service C, and so on.
If the root decides "Drop this trace" (trace-flags=00), every downstream service sees that flag and skips recording. No spans are generated anywhere in the chain. The decision is atomic across the entire distributed system.
Interviewers often ask: "What happens if two services have different sampling rates configured?" The correct answer is: the root's decision wins. Once a trace is marked sampled (or not) at the head, all downstream services must honor that flag. Local sampling configuration only applies when a service is acting as a root (i.e., it receives no incoming trace context). This is a common misconfiguration — engineers set a 10% rate on a leaf service thinking it will reduce their data volume, but it has no effect on traces that arrive already marked as sampled.
Key Point: Coherency is enforced by the propagation protocol, not by coordination between services at runtime. This is why the W3C TraceContext standard matters — it gives every service a shared language for communicating the sampling decision without requiring them to talk to each other directly.