The Signal-to-Noise Problem
In a typical production system, the vast majority of telemetry is low-value noise. Consider a service handling 10,000 requests per minute:
- ~9,800 requests succeed (HTTP 200) — routine, repetitive, low diagnostic value
- ~180 requests are slow (HTTP 200, latency > 1s) — worth sampling
- ~20 requests fail (HTTP 500) — critical, must keep every one
If you store all 10,000 request traces at $0.10/GB, you're paying mostly to store evidence that things are working fine. The goal is to store the 20 failures at full fidelity and a statistical sample of the rest.
Filtering: Drop the Obvious Noise
Filtering permanently discards data that matches a rule. This is appropriate for data that has zero diagnostic value — a Vector filter transform, for example, can drop any log line containing a Kubernetes liveness-probe signature, and a second chained filter can drop anything below INFO level so DEBUG noise never reaches storage.
Common candidates for filtering:
- Health check / liveness probe requests
- DEBUG and TRACE level logs in production
- Metrics scrape requests from Prometheus itself
- Internal service-to-service heartbeats
Sampling: Keep a Representative Slice
Sampling retains a percentage of matching events. Unlike filtering (which discards based on content), sampling discards based on volume while preserving statistical representativeness. A Vector sample transform configured to keep 1 in 10 events, keyed consistently by status code, implements the pattern of keeping 100% of errors, 10% of successful requests, and 1% of health checks.
Head-based vs. Tail-based sampling is an important distinction:
- Head-based sampling decides at the start of a request whether to sample it. Simple and cheap, but you might drop a request that later fails.
- Tail-based sampling waits until the entire trace is complete, then decides. This lets you always keep traces that contain errors, even if you decided to sample them at the start. Requires buffering the entire trace in memory — more complex and resource-intensive.
For most teams, head-based sampling with a rule to always keep errors is a practical starting point.
Real-World Impact
A team running 50 microservices on Kubernetes, each emitting logs at moderate volume, can realistically reduce their Datadog log ingestion bill by 60–80% by:
1. Dropping health check logs (often 30–40% of total volume)
2. Dropping DEBUG logs (another 20–30%)
3. Sampling INFO logs at 10% (another 50% reduction on the remainder)
Key Point: Filtering removes zero-value data permanently. Sampling retains statistical representativeness at a fraction of the volume. Together, they are the most impactful cost-reduction levers in observability.