The User Experience in Grafana

From Graph to Root Cause in One Click

The real power of Exemplars is in the visualization layer. Grafana has native support for rendering Exemplars when querying Prometheus or Tempo as a data source.

Here is the end-to-end experience:

Step 1 — The Alert: Your P99 latency graph shows a spike at 14:32.

Step 2 — The Overlay: Grafana detects Exemplar data in the Prometheus query response. It renders small diamond or star icons directly on the graph, overlaid on the metric line. Each icon represents one sampled request.

Step 3 — The Investigation: You see a diamond icon right at the peak of the spike. You hover over it — it shows trace_id=abc123, value=1.87s.

Step 4 — The Jump: You click the diamond. Grafana opens a split panel (or redirects you) to your configured tracing backend (Tempo, Jaeger) and loads the full distributed trace for trace_id=abc123.

Step 5 — The Root Cause: The trace shows a 1.6-second database query on the orders service. You have your answer.

Total time from "I see a spike" to "I know the cause": under 60 seconds.

Configuring the Data Source Link in Grafana

For Grafana to know where to send you when you click an Exemplar, you configure a data source link in the Prometheus data source settings — an exemplarTraceIdDestinations entry naming the label to read (trace_id) and the UID of the target Tempo data source.

This tells Grafana: "When you find an Exemplar with a label named trace_id, use its value to look up the trace in the Tempo data source."