You Can't Keep Everything
In a high-traffic system, logging every single INFO and DEBUG event can generate terabytes of data per day. At $0.50/GB for storage plus indexing costs, this becomes a significant budget line. More importantly, the sheer volume makes it harder to find the signal in the noise.
You need a strategy to keep the events that matter and drop the ones that don't.
Sampling Strategies
Head-Based Sampling: The decision to keep or drop a request's logs is made at the beginning of the request, before any processing happens.
- Example: "Keep 1% of all incoming requests, but always keep 100% of requests that are flagged as errors."
- Pro: Simple to implement. Low overhead — you never buffer logs you're going to drop.
- Con: You make the keep/drop decision before you know the outcome. A request that starts normally but fails 30 seconds later might be dropped.
Tail-Based Sampling: All logs for a request are held in a temporary buffer. The keep/drop decision is made after the request completes, based on its outcome.
- Example: "If the request completed successfully in under 200ms, drop its logs. If it errored or took over 1 second, keep everything."
- Pro: You always keep logs for interesting events (errors, slow requests). Your retained data is maximally useful.
- Con: Requires a buffering layer that holds all in-flight request logs in memory until the request completes. More complex and resource-intensive to implement. Works best when integrated with a distributed tracing system that already tracks request completion.
Adaptive Sampling: Dynamically adjust the sampling rate based on current system load or error rate. When errors spike, increase the sampling rate to capture more detail. When everything is healthy, reduce it to save costs. This is the most sophisticated approach and is typically implemented at the observability platform level rather than in application code.