Why You Should Not Trace Everything
Tracing every single request in a high-traffic system is expensive and unnecessary. If your API handles 10,000 requests per second and 99.9% of them succeed in under 50ms, recording all 10,000 traces per second generates enormous storage costs and buries the interesting traces in noise. You only need a statistically significant sample to identify patterns and catch errors.
X-Ray uses a sampling algorithm to decide which requests to record. The algorithm is controlled by sampling rules.
The Default Rule: "1 + 5%"
If you configure nothing, X-Ray applies this default:
- Reservoir: Record the first 1 request per second unconditionally. This guarantees you always have at least one trace per second even during low-traffic periods — critical for catching intermittent bugs.
- Fixed Rate: Record 5% of all additional requests beyond the first one.
So at 100 req/sec: you record 1 (reservoir) + 4.95 (5% of 99) ≈ 6 traces per second.
At 10,000 req/sec: you record 1 + 499.95 ≈ 501 traces per second.
Custom Sampling Rules
You can create rules that override the default for specific traffic patterns. Rules are evaluated in priority order (lower number = higher priority):
Rule: "HighValueCheckout"
Priority: 1
Host: *
HTTP Method: POST
URL Path: /checkout
Reservoir: 10 # Always record at least 10/sec
Fixed Rate: 50% # Record half of additional traffic
This means your /checkout endpoint gets dense tracing (you care deeply about conversion failures) while your /healthcheck endpoint uses the default 1+5% (you do not need 10,000 health check traces per second).
Centralized vs. Local Rules
| Mode | How Defined | Management |
|---|---|---|
| Local | JSON file on each server | Must redeploy to change rules |
| Centralized | X-Ray Console / API | SDK polls X-Ray every 10 seconds for updates |
Always use Centralized rules in production. Local rules require a deployment to change sampling rates — exactly when you are in the middle of an incident and need to increase sampling to capture more data, you cannot afford a deployment cycle.
Senior Insight: Sampling rules are also a security consideration. If an attacker can force your system to trace every request (by manipulating the X-Amzn-Trace-Id header to always request sampling), they can drive up your X-Ray costs. The SDK respects the Sampled=1 flag in the incoming header by default, but you can configure it to ignore client-requested sampling and rely solely on your server-side rules.