Brute Force Physics
Google's Spanner database wanted strict external consistency: if Transaction A commits before Transaction B starts (in real physical time), then A's commit timestamp must be less than B's. This is a stronger guarantee than causal consistency — it means Spanner's transaction order matches real-world time order, making it safe to use as a globally consistent database.
The problem: you cannot achieve this with NTP because the uncertainty window is too large. Google's solution was to not pretend the uncertainty doesn't exist — instead, make the uncertainty explicit and reason about it.
TrueTime: Uncertainty as a First-Class Citizen
Google built TrueTime, a time API available to all Spanner processes. Unlike a normal clock that returns a single value, TrueTime returns an interval:
TT.now() → TTInterval { earliest: t_early, latest: t_late }
This interval is a guarantee: the true current time is somewhere within [t_early, t_late]. The width of the interval (called epsilon, ε) is typically 1–7 milliseconds in Google's datacenters, achieved by placing GPS receivers and atomic clocks in every datacenter and using a carefully engineered NTP-like protocol to bound the uncertainty.
TrueTime also provides two helper methods:
- TT.before(t) → true if the current time is definitely before t (i.e., t_late < t)
- TT.after(t) → true if the current time is definitely after t (i.e., t_early > t)
Commit Wait: Waiting Out the Uncertainty
To guarantee that Transaction A's timestamp is less than Transaction B's timestamp when A commits before B starts, Spanner uses Commit Wait:
- When a transaction is ready to commit, Spanner calls
TT.now()and picks a commit timestamps = t_late(the latest possible current time). - Spanner then waits until
TT.after(s)is true — meaning the entire uncertainty window has passed and the true current time is definitely greater thans. - Only then does Spanner apply the commit and release the transaction.
The wait is typically 2ε — about 2–14 milliseconds. This is the price of external consistency.
Why this works: After the commit wait, any new transaction that starts will call TT.now() and get a t_early that is greater than s (because we waited until the true time is definitely past s). So the new transaction's timestamp will be greater than the committed transaction's timestamp. External consistency is guaranteed.
The Infrastructure Behind TrueTime
TrueTime's small epsilon (1–7ms) is not magic — it requires significant infrastructure investment:
- GPS receivers in every Google datacenter, providing time accurate to ~100 nanoseconds.
- Atomic clocks as a backup for GPS outages (GPS can be jammed or fail).
- Two independent time masters per datacenter (one GPS, one atomic) to detect and reject faulty sources.
- Armageddon masters: Google's name for the time master servers that serve TrueTime to Spanner processes.
This infrastructure is not available to most organizations. For teams outside Google, the practical alternatives are:
- CockroachDB: An open-source Spanner-inspired database that uses NTP with a larger uncertainty window (500ms by default) and implements the same commit wait logic.
- YugabyteDB: Similar approach.
- Centralized sequencer: Route all writes through a single node that assigns monotonically increasing sequence numbers — simpler but creates a bottleneck.
Interviewers at companies building distributed databases often ask: "How does Google Spanner achieve external consistency?" The key points to hit: (1) TrueTime returns an uncertainty interval, not a single value. (2) Commit Wait forces a transaction to wait until its timestamp is definitely in the past before committing. (3) This requires GPS/atomic clock infrastructure to keep epsilon small enough that the wait is acceptable (milliseconds, not seconds). Bonus: mention that CockroachDB implements the same algorithm with NTP, accepting a larger epsilon.