Embracing Uncertainty as a First-Class Concept
Most systems pretend clocks are accurate and then fail in subtle ways when they are not. Google took the opposite approach: they built a system that explicitly models clock uncertainty and uses that model to make safety guarantees.
The result is TrueTime, the timekeeping API at the heart of Google Spanner.
Instead of asking "What time is it right now?" — a question that has no precise answer on distributed hardware — TrueTime asks "What is the range of times it could possibly be right now?" The API call TrueTime.now() returns not a single timestamp but an interval:
[Earliest_Possible_Time, Latest_Possible_Time]
The actual current time is guaranteed to fall somewhere inside that window. Because Google installs GPS receivers and Atomic Clocks in every data center, the hardware is precise enough to keep this window extremely small — typically less than 7 milliseconds.
The Commit Wait Protocol
Knowing the uncertainty window is only half the solution. Spanner still needs to guarantee that if $T_1$ finishes before $T_2$ starts, then $Timestamp(T_1) < Timestamp(T_2)$.
The mechanism is Commit Wait:
- When a transaction is ready to commit, Spanner picks a write timestamp $T_{write}$ equal to the current
TrueTime.now().latest— the upper bound of the uncertainty window. - Spanner then pauses the transaction and waits until the real clock has definitely advanced past $T_{write}$.
- Only after this wait does Spanner acknowledge the commit to the client.
The wait duration is approximately $2\varepsilon$ where $\varepsilon$ is the average uncertainty (roughly 7ms), making the total wait around 14ms.
Why does this work? Because after the wait, any future transaction that starts will observe a TrueTime.now().earliest that is already greater than $T_{write}$. The ordering rule is guaranteed by physics — the clock has literally moved past the old timestamp.
Timeline:
T_write = 1000ms (picked at commit time)
Commit Wait: pause until real time > 1000ms
Commit acknowledged at ~1014ms
Any T2 that starts after this point:
TrueTime.now().earliest > 1000ms
Therefore: Timestamp(T2) > Timestamp(T1) ✓
Pros: Absolute correctness. No software heuristics. The guarantee is backed by atomic physics.
Cons: Requires custom hardware infrastructure that only Google operates. Every write has a minimum latency floor of approximately 14ms that cannot be optimized away — it is a fundamental property of the approach. Cross-region writes add propagation latency on top of this floor.
A common interview question is: "Why can't you just use NTP for Spanner?" The answer is that NTP uncertainty on commodity hardware is 50–500ms. If Spanner used NTP, Commit Wait would need to be 100–1000ms per write — making the system unusably slow. TrueTime's atomic clocks reduce uncertainty to ~7ms, making the 14ms Commit Wait acceptable. The hardware investment directly buys write performance.