The "Negative Duration" Bug
Imagine you are timing how long it takes to drive from city A to city B. You note the departure time on your phone's clock. Halfway through the trip, your phone automatically adjusts for a time zone change and jumps back one hour. When you arrive and subtract departure from arrival, you get a negative travel time. This is not a hypothetical — it is a real class of bug that crashes production systems, and it happens because developers use the wrong clock.
Linux (and POSIX in general) exposes multiple distinct clocks through the clock_gettime() system call. The two you must understand are CLOCK_REALTIME and CLOCK_MONOTONIC.
The Wall Clock (CLOCK_REALTIME)
This answers the question: "What time is it right now, in the real world?" It is the clock you would read off a wall — hence the name. It is synchronized with UTC and reflects the actual calendar time.
- Dangerous property: It is discontinuous. NTP can step this clock forward or backward at any moment. A sysadmin running
date -scan jump it by hours. A VM migration can cause it to skew wildly. - The bug in code:
start = time.time() # CLOCK_REALTIME under the hood
do_work()
end = time.time()
duration = end - start # Can be NEGATIVE if NTP stepped back during do_work()
If NTP corrects the clock backward by 5 seconds while do_work() runs for 3 seconds, duration is -2.0. Any code that treats this as a timeout, a rate limiter, or a metric will behave incorrectly — and the failure is silent and intermittent, making it extremely hard to debug.
The Monotonic Clock (CLOCK_MONOTONIC)
This answers the question: "How much time has elapsed since some arbitrary fixed point?" On Linux, that fixed point is system boot. The monotonic clock starts near zero at boot and counts upward, forever.
- Safe property: It never jumps backward. NTP adjustments do not step this clock. If NTP needs to correct a large drift, it slews the monotonic clock — adjusting its rate slightly (e.g., running at 1.0005× speed) until it converges, rather than jumping.
- Limitation: It has no relationship to wall time. You cannot use it to answer "what time is it?" — only "how long has it been?"
import time
start = time.monotonic() # CLOCK_MONOTONIC under the hood
do_work()
elapsed = time.monotonic() - start # Always >= 0, always accurate
The Rule
Use CLOCK_REALTIME (wall clock) only when you need to display or record a human-readable timestamp. Use CLOCK_MONOTONIC for every timeout, interval, duration measurement, rate limiter, and retry backoff in your code. This is not a preference — it is a correctness requirement.
There is a third clock worth knowing: CLOCK_BOOTTIME. It behaves like CLOCK_MONOTONIC but also counts time while the system is suspended (sleeping). Use this when measuring intervals that might span a laptop lid-close or a VM suspend/resume.
A common SRE interview question is: "Your monitoring shows a metric with a negative rate — the counter appears to go backward. What are the possible causes?" One strong answer is: the metric collection code used wall clock time for duration measurement, and an NTP correction caused a negative delta. The correct fix is to use a monotonic clock for all elapsed-time calculations. Interviewers also ask about the difference between time.time() and time.monotonic() in Python — know that time.time() wraps CLOCK_REALTIME and time.monotonic() wraps CLOCK_MONOTONIC.