Free — no signup required

The Instrumentation Layer: Connecting Traces to Metrics

2 min read

The Missing Link

Exemplars don't appear automatically just because you have both metrics and tracing instrumented. The metrics library and the tracing library are independent — they don't know about each other by default. You must explicitly wire them together so that when the metrics library records an observation, it can ask the tracing library: "What is the current Trace ID?"

This connection is called a SpanContext supplier or Exemplar sampler, depending on the library.

How It Works

When a request arrives, the tracing library creates a Span and stores the current Trace ID in thread-local storage (or a context propagation mechanism). When the metrics library records a histogram observation for that same request, it calls the SpanContext supplier to retrieve the active Trace ID and attaches it as an Exemplar label.

The key requirement: the metric observation must happen within the same execution context as the active trace span. If you record the metric outside the span's scope, the SpanContext supplier returns nothing, and no Exemplar is attached.

Language Examples

The wiring pattern is consistent across ecosystems, differing only in API shape. In Go (prometheus/client_golang with OpenTelemetry), you read the active span from the request context and, if it's valid, call the histogram's exemplar-aware observe method with the trace and span IDs attached as exemplar labels — a standard .Observe() call has no exemplar support, so the SDK must expose this dedicated method. In Java (Micrometer with OpenTelemetry), the wiring happens once at registry construction — a DefaultExemplarSampler backed by an OpenTelemetrySpanContextSupplier is passed into the PrometheusMeterRegistry, after which every timer observation automatically captures the active trace ID with no per-call changes to business logic. In Python (prometheus_client with OpenTelemetry), you read the current span's context and, if valid, pass an exemplar dict with the formatted trace ID into .observe() alongside the measured duration.

The Full Requirements Checklist

For Exemplars to work end-to-end, every layer of the stack must be configured correctly:

Layer Requirement
Application Metrics library wired to tracing context (SpanContext supplier)
Exposition format App exposes /metrics in OpenMetrics format (application/openmetrics-text)
Prometheus Started with --enable-feature=exemplar-storage
Grafana Prometheus data source has exemplarTraceIdDestinations configured
Tracing backend Tempo/Jaeger is configured as a Grafana data source

If any layer is missing, Exemplars silently fail — the metrics still work, but the trace links don't appear.

This is one of 18 chapters

Every chapter is free — Kubernetes, Terraform, SRE, distributed systems, and more — with fast daily review built in.

Sign up free