"For Example..."
The word "exemplar" comes from Latin meaning "a typical example or model." In observability, an Exemplar is a specific, sampled data point — containing a Trace ID and an exact measurement value — that is attached to an aggregated metric bucket.
Think of it like a receipt stapled to a monthly expense report. The report shows the total ($847 spent on cloud compute). The receipt is one specific transaction ($23.50 for a particular job run) that you can inspect in detail. The receipt doesn't represent every transaction, but it gives you a concrete example to investigate if the total looks wrong.
How It Works Mechanically
When your application records a metric observation — say, an HTTP request that took 1.7 seconds — the Prometheus client library does two things simultaneously:
- Increments the histogram bucket for
le="2.0"(requests under 2 seconds). - Attaches an Exemplar to that bucket: the Trace ID of the current request, plus the exact measured value (1.7s).
The Exemplar is not stored for every single request. That would be prohibitively expensive and would defeat the purpose of aggregated metrics. Instead, the library keeps one Exemplar per bucket per scrape interval (typically 15–60 seconds), replacing the previous one each time a new observation arrives. This means you get a representative sample — enough to investigate — without blowing up your storage.
What an Exemplar Contains
An Exemplar is a small structure with three components:
| Field | Example | Purpose |
|---|---|---|
| Labels | {trace_id="abc123", span_id="def456"} |
Links to the specific trace |
| Value | 1.7 |
The exact measurement that triggered this exemplar |
| Timestamp | 1686000000.000 |
When the observation was recorded |
The trace_id is the critical field. It is the key that unlocks the full distributed trace in your tracing backend.