The MVCC Problem
Etcd uses Multiversion Concurrency Control (MVCC) — a storage strategy where updating a key never overwrites the old value. Instead, Etcd writes a new version alongside the old one. This allows readers to see a consistent snapshot of the database without blocking writers, which is critical for a high-concurrency system like Kubernetes.
The consequence: every write operation accumulates historical versions on disk. A single Secret that gets rotated 100 times has 100 versions stored. Left unchecked, this would grow forever.
Compaction is the process that marks old versions as deleted. Kubernetes triggers compaction automatically (by default, every 5 minutes via the kube-apiserver --etcd-compaction-interval flag). Compaction tells Etcd: "You no longer need versions older than revision X." But — and this is the critical point — compaction does not return disk space to the operating system. It creates holes inside the database file, exactly like deleting files on a hard drive without defragmenting it. The file stays the same size on disk even though most of its contents are logically empty.
Why This Becomes a Crisis
Etcd enforces a hard storage quota (default: 2 GiB, configurable up to 8 GiB via --quota-backend-bytes). When the database file hits this quota — even if most of it is holes — Etcd enters alarm mode. In alarm mode:
- All write operations are rejected with
etcdserver: mvcc: database space exceeded. - The Kubernetes API server can no longer create or update resources.
- The cluster effectively becomes read-only.
This is a production outage. Recovering from it requires defragmentation.
The Solution: Defrag
Defragmentation rewrites the entire database file from scratch, packing all live data together and releasing the holes back to the filesystem. etcdctl exposes this as a single defrag operation, run against one member at a time (specifying that member's endpoint and TLS credentials).
Important operational rules:
- Defrag is an expensive I/O operation. The member being defragmented is briefly unavailable while the file is being rewritten. During this window, it cannot serve reads or participate in consensus.
- Always defrag followers first, then the leader last. If you defrag the leader first, Raft will elect a new leader, causing unnecessary churn.
- Never defrag all members simultaneously. You will lose quorum and the cluster will stop accepting writes.
- After defrag, if the cluster was in alarm mode, you must manually clear the alarm (
etcdctl alarm disarm) — defragmentation does not clear it automatically.
Interviewers frequently ask: "Your Etcd cluster went read-only. What do you do?" The expected answer covers three steps in order: (1) confirm the alarm with etcdctl alarm list, (2) defrag members one at a time starting with followers, (3) disarm the alarm with etcdctl alarm disarm. Candidates who jump straight to "restart Etcd" reveal they don't understand that the database file size is the root cause — a restart changes nothing about the file.
Key Point: In high-churn clusters (frequent HPA scaling, rolling deployments, Secret rotation), defragmentation should be a scheduled maintenance task — not a reactive emergency procedure. Many teams run it weekly via a CronJob. Monitor the etcd_db_total_size_in_bytes and etcd_db_total_size_in_use_in_bytes Prometheus metrics; when the ratio diverges significantly, it's time to defrag.