SRE Interview Questions: SLI, SLO, and Error Budgets
10 real questions, pulled straight from HamChops's Observability & SRE chapter — scenario-based, not trivia.
A team manages an e-commerce payment service. They have an internal engineering goal to ensure 99.95% of payment requests are processed successfully over a rolling 30-day window. They also have a contract with their largest merchant that states if the success rate drops below 99.9% in any calendar month, they will issue a 20% service credit. The team's monitoring dashboard currently shows that 99.97% of requests have succeeded in the last 30 days. Match each of the following values to the correct concept: (1) 99.97%, (2) 99.95%, (3) 99.9%, (4) 0.05%.
(1) SLI — the actual measured performance right now. (2) SLO — the internal engineering target. (3) SLA — the external contractual commitment with financial consequences. (4) Error Budget — calculated as 100% − SLO = 100% − 99.95% = 0.05%.
Your service has a 99.9% availability SLO over a rolling 30-day window. Your monitoring system shows that over the last 2 hours, your error rate has been 1.5%. Your on-call engineer is trying to decide whether to escalate this to a full incident response. Using burn rate analysis, what is the burn rate, how long until the error budget is exhausted at this rate, and should the engineer escalate?
The SLO error rate is 0.1%. The burn rate is 1.5% ÷ 0.1% = 15×. At this rate, the monthly error budget will be exhausted in 30 days ÷ 15 = 2 days. Yes, the engineer should escalate immediately — a burn rate of 15× is well above the critical threshold of 14.4× that would exhaust 2% of the budget in a single hour.
Your team runs a three-node PostgreSQL cluster with automatic failover. A junior engineer proposes the following two alerts: Alert A fires when any single PostgreSQL node's CPU exceeds 85% for 5 minutes. Alert B fires when the application's database query error rate exceeds 2% for 2 minutes. The junior engineer argues that Alert A is more proactive because it catches problems before users are affected. Your tech lead disagrees. Who is right, and why? What role, if any, should Alert A play in your observability strategy?
The tech lead is right. Alert B is the correct paging alert because it directly measures user impact. Alert A should be demoted to a dashboard metric or a low-severity warning routed to Slack, not a page.
A Prometheus alert rule is configured with expr: rate(http_errors_total[5m]) > 0.05 and for: 10m. Your SLO dashboard shows the error rate has been above 0.05 for the past 8 minutes, but no alert has fired and no one has been paged. A colleague says the alert rule must be broken. What are the two most likely explanations, and how would you verify each?
First, the alert may be in "pending" state — the condition has been true for 8 minutes but the for: 10m duration has not elapsed yet. Second, the alert may have fired but been routed to a destination that was silenced or misconfigured. Check the Prometheus /alerts endpoint for pending state, and check Alertmanager's routing and silence configuration.
Your team runs a payment processing service deployed as 20 replicas in Kubernetes. A product manager asks for a dashboard showing the 95th percentile latency of payment API calls across the entire fleet, updated every minute. A junior engineer proposes using a Summary metric because "it directly gives us percentiles." What is wrong with this approach, and what should you use instead?
Summaries calculate percentiles on each individual application instance and cannot be mathematically aggregated across instances. You cannot average 20 pre-calculated p95 values and get the true fleet-wide p95. Use a Histogram instead — it ships raw bucket counts to Prometheus, which can aggregate all 20 instances' buckets and then calculate the true fleet-wide p95 using histogram_quantile(0.95, ...).
A platform team receives an alert that their Prometheus server's memory usage has grown from 4GB to 28GB over the past two weeks, despite no new services being onboarded. A developer recently added "enhanced debugging" to the payment service by adding a request_id label to all payment metrics. The payment service handles 50,000 requests per hour. What is the most likely cause of the memory spike, and what is the correct fix?
The request_id label is causing a cardinality explosion. Each unique request ID creates a new time-series. At 50,000 requests per hour, the payment service is generating 50,000 new time-series every hour — millions per day. The fix is to remove the request_id label from the metric entirely. Request-level detail belongs in distributed traces or structured logs, not metrics.
Your team runs Grafana Loki for log aggregation across 200 microservices in Kubernetes. An on-call engineer writes the following LogQL query to find all database timeout errors across the entire platform: {} |= "db_timeout". The query times out every time. A senior engineer says the query is "fundamentally wrong." What is wrong with it, and how would you fix it?
The query has no label filter, forcing Loki to decompress and grep through every log chunk across all 200 services. Loki's architecture requires label filters to narrow the search space first. The fix is to add the most selective label filter available — for example, {env="prod"} |= "db_timeout" or, better, {app="payment-service", env="prod"} |= "db_timeout" if the error is service-specific.
Your team runs a payment processing platform handling 50,000 requests per second. The SLA requires that all failed transactions (HTTP 5xx or payment gateway errors) must be traceable for post-incident analysis. Your current head-based sampling at 2% is causing you to miss roughly 98% of error traces, making post-incident debugging nearly impossible. However, switching to 100% sampling would increase your tracing infrastructure costs by 40x. A colleague proposes switching entirely to tail-based sampling with a policy of "keep all traces where any span has status=ERROR, drop everything else." What is the primary operational risk of this approach, and what would you recommend instead?
The primary risk is Collector memory exhaustion. Tail-based sampling requires buffering all spans in memory before making a keep/drop decision. At 50,000 RPS, even a 5-second buffer window means holding 250,000 requests worth of spans in memory simultaneously. If the Collector is undersized, it will OOM-crash, causing a complete loss of trace data. The recommended approach is a hybrid strategy: keep head-based sampling at 1-2% for normal traffic (for baseline performance statistics), but add tail-based sampling rules that override the head decision and force-keep any trace containing an error status or exceeding a latency threshold. This captures all interesting traces while keeping infrastructure costs manageable.
Your team runs a chaos experiment: you inject 2 seconds of network latency between your API gateway and your user-profile service. Your hypothesis was that the API gateway's 3-second timeout would trigger, return a cached response, and keep the overall error rate below 0.1%. Instead, you observe that within 90 seconds of starting the experiment, the API gateway itself becomes unresponsive and the error rate climbs to 40%. The latency injection is only affecting the user-profile service, which handles 10% of total API traffic. What is the most likely root cause, and what architectural fix does this experiment suggest?
The most likely cause is thread pool exhaustion on the API gateway. Requests to the user-profile service are blocking threads while waiting for the 2-second timeout to fire. Because the gateway uses a shared thread pool for all services, the blocked threads consume the pool, starving requests to all other services — not just user-profile. The fix is to implement bulkhead isolation: give each downstream dependency its own bounded thread pool or connection pool so that one slow dependency cannot exhaust resources needed by others.
Your organization runs a mixed infrastructure: 60% of workloads are on Kubernetes (EKS on AWS), 30% are on legacy EC2 instances running traditional Java applications (no containers), and 10% are managed AWS services like RDS and SQS. Your team wants to start a Chaos Engineering program. A junior engineer proposes using Chaos Mesh for everything. A senior engineer pushes back. Who is right, and what tool strategy would you recommend?
The senior engineer is right to push back. Chaos Mesh is Kubernetes-native and cannot inject faults into bare EC2 instances or managed AWS services like RDS. For this mixed environment, AWS Fault Injection Simulator (FIS) is the better primary tool because it natively supports EC2, EKS, RDS, and SQS from a single platform. Chaos Mesh could be used as a supplementary tool for Kubernetes-specific fault types that FIS doesn't support (like fine-grained pod-level network partitions), but a single-tool strategy with FIS covers the broadest surface area.
Want the rest of Observability & SRE?
This is 10 of hundreds of concept reviews in this chapter alone — plus 17 more chapters, with fast daily review built in.
See pricing