The "Zombie" Query Problem
In a microservices architecture, a single user request often fans out through multiple services before hitting the database. Consider this chain:
User → Service A → Service B → Database
Now consider these timeouts, configured independently by each team:
| Layer | Timeout |
|---|---|
| User | 5 seconds patience |
| Service A | 2 seconds |
| Service B | 10 seconds |
| DB query | 8 seconds to complete |
Here's what happens at T+2 seconds: Service A times out and returns an error to the user. The user sees a failure and moves on.
But Service B doesn't know this. It's still waiting. The database is still executing the query. At T+8 seconds, the query completes, Service B processes the result, and sends a response to Service A — which has already closed the connection. All of that work was wasted. The database spent 6 seconds of CPU and I/O on a result that nobody will ever read.
At scale, these "zombie queries" accumulate. They consume connection pool slots, CPU, and I/O — crowding out queries that are still serving live users.
Propagating the Deadline
Deadline Propagation solves this by treating the original request's deadline as a shared resource that flows through the entire call chain.
- The user's request arrives. The system records an absolute deadline:
now() + 5s. - Service A passes this deadline in its outgoing request to Service B (typically as an HTTP header like
X-Request-Deadline: 2024-01-15T14:23:07.000Zor via gRPC context). - Service B receives the deadline, calculates the remaining time (
deadline - now()), and sets a statement timeout on its database connection to exactly that remaining duration. - If the query runs too long, the database kills it automatically — at the exact moment the client would have timed out anyway.
In PostgreSQL, this means Service B calling SET LOCAL statement_timeout = '4800ms' right before running the real query. The SET LOCAL applies only to the current transaction, so it doesn't affect other connections in the pool.
The result: no zombie queries. Resources are freed the instant they stop being useful. The database only works on queries that have a live requester waiting for the answer.
A common SRE interview question is: "How do you prevent a slow downstream service from cascading failures upstream?" Most candidates answer with circuit breakers (correct but incomplete). The complete answer adds deadline propagation: circuit breakers protect against sustained failure, while deadline propagation prevents wasted work on individual requests that have already been abandoned. Mentioning both — and explaining that they're complementary, not alternatives — signals senior-level thinking.