CockroachDB: Hybrid Logical Clocks (The Software Solution)

The Open-Source Constraint

CockroachDB is designed to be "Spanner for the rest of us" — a globally distributed, strongly consistent SQL database that runs on commodity hardware: AWS EC2 instances, Azure VMs, bare-metal servers, even a laptop. No GPS receivers. No atomic clocks. Just standard Linux system clocks synchronized by NTP.

This constraint rules out Commit Wait as a practical mechanism. With NTP uncertainty of 250–500ms, a Commit Wait would add half a second to every write. The system would be correct but unusable.

CockroachDB's answer is Hybrid Logical Clocks (HLC), a technique that combines physical wall-clock time with a logical counter to track causality without requiring precise clock synchronization.

Anatomy of an HLC Timestamp

Every HLC timestamp is a pair:

(Physical Component, Logical Component)

• Physical Component: The wall-clock time from the local Linux system clock, in nanoseconds. This gives timestamps a real-world time anchor.
• Logical Component: A simple integer counter (0, 1, 2, ...). This breaks ties when two events happen within the same physical clock tick, and — critically — allows the system to advance time logically when it receives a message from a node with a higher timestamp.

A timestamp of (1700000000000, 3) means "wall clock nanosecond 1700000000000, third event at that physical instant."

The HLC Algorithm

The rules are simple and applied on every message exchange between nodes:

When Node A sends a message:
Node A attaches its current HLC timestamp to the message.

When Node B receives a message with timestamp (p_incoming, l_incoming):

p_local = Node B's current wall clock
p_new   = max(p_local, p_incoming)

if p_new > p_local:
    # Incoming message is from the future relative to local clock
    # Jump physical time forward, reset logical counter
    HLC = (p_new, l_incoming + 1)
elif p_new == p_incoming and p_new == p_local:
    # Both clocks agree on physical time
    # Increment logical counter to preserve ordering
    HLC = (p_new, max(l_local, l_incoming) + 1)
else:
    # Local clock is ahead
    HLC = (p_new, l_local + 1)

The key insight: HLC timestamps only ever move forward. A node can never receive a message that causes it to go backward in time. This is the causality guarantee — if event B causally depends on event A (because B received a message from A), then Timestamp(B) > Timestamp(A).

Handling Uncertainty: The Max Clock Offset

Because CockroachDB cannot do Commit Wait, it uses a different mechanism to handle the uncertainty window: the max_offset setting (default: 500ms).

This setting declares the maximum amount of clock skew the cluster will tolerate. CockroachDB uses this value in two ways:

1. Uncertainty Reads: When a transaction reads a value, it checks whether any write timestamps fall within the uncertainty window. If a write happened at a timestamp that might be in the future relative to the reader's clock (within max_offset), CockroachDB restarts the transaction with a higher timestamp to ensure it sees the correct state. This is called an uncertainty restart.

2. Clock Skew Enforcement: Each node continuously monitors its clock offset relative to other nodes — exposed in the crdb_internal.gossip_nodes table — and if it detects that its own clock has drifted beyond max_offset, it considers itself "insane" — unsafe to participate in transactions — and shuts itself down to protect cluster consistency.

The tradeoff: CockroachDB sacrifices some availability (a node with a bad clock shuts down) to preserve consistency. This is the correct tradeoff for a database — silent data corruption is far worse than a node restart.