How Tracing Actually Works
You do not just "turn on" X-Ray with a checkbox. You must instrument your code — meaning you integrate the X-Ray SDK into your application so it can observe and record what your code is doing.
Here is the full data flow, step by step:
1. Interception (The SDK Wraps Your Calls)
The SDK patches standard libraries. When your code calls dynamodb.putItem() or makes an outbound HTTP request, the SDK intercepts that call transparently. You do not rewrite your business logic — the SDK wraps it.
2. Propagation (The Trace Header)
When your service makes an outbound call to another service, the SDK injects a special HTTP header: X-Amzn-Trace-Id. This header carries the Trace ID and the current Segment ID. The receiving service's SDK reads this header and knows it is part of an existing trace rather than a new one. This is how a single Trace ID follows a request across service boundaries.
3. The Daemon (The Local Buffer)
The SDK does not send data directly to the X-Ray API. Doing so would add network latency to every operation your code performs. Instead, the SDK sends small UDP packets to a local X-Ray Daemon process listening on 127.0.0.1:2000. UDP is fire-and-forget — your application thread does not wait for an acknowledgment.
4. Batching (The Daemon Uploads)
The Daemon collects these UDP packets, buffers them in memory, and uploads them to the X-Ray API in batches over HTTPS. This decouples your application's performance from the observability pipeline.
Where Does the Daemon Run?
| Environment | Daemon Status |
|---|---|
| AWS Lambda | Built-in, managed by AWS. You only need to enable Active Tracing. |
| Amazon ECS | Run as a sidecar container in the same task definition. |
| Amazon EC2 | Install and run as a background process (systemd service). |
| AWS Fargate | Run as a sidecar container (same as ECS). |
Critical Detail for ECS/Fargate: The sidecar container must share the network namespace with your application container so that 127.0.0.1:2000 resolves correctly. In ECS task definitions, containers in the same task share a network namespace by default — this is why the sidecar pattern works without extra configuration.
Interviewers frequently ask: "Why does the X-Ray SDK use UDP to communicate with the daemon instead of HTTP?" The answer has two parts: (1) UDP is non-blocking — your application thread does not wait for acknowledgment, so tracing adds near-zero latency to your hot path. (2) If the daemon is unavailable, UDP packets are silently dropped rather than causing your application to throw exceptions. Observability infrastructure should never be a single point of failure for your application.