The Metrics That Matter
Reactive Etcd operations — defragging after the cluster goes read-only, replacing a member after it crashes — are the worst way to operate. The goal is to detect problems before they become outages. Etcd exposes a rich set of Prometheus metrics that make this possible.
Database size metrics are the most critical for preventing the quota alarm: etcd_db_total_size_in_bytes (physical file size on disk, including holes) and etcd_db_total_size_in_use_in_bytes (actual live data size). When the former approaches your quota (default 2 GiB) and significantly exceeds the latter, fragmentation is high and defragmentation is overdue. A healthy ratio is within 20% of each other.
Leader election and consensus metrics reveal instability: etcd_server_leader_changes_seen_total should stay near zero — frequent changes signal instability. etcd_server_proposals_failed_total spiking indicates failed Raft proposals. etcd_server_proposals_pending shows the backlog of uncommitted proposals. etcd_network_peer_round_trip_time_seconds running high means members are at risk of timing out on each other.
Disk performance metrics are often the root cause of Etcd instability: etcd_disk_wal_fsync_duration_seconds (time to flush the write-ahead log to disk — a p99 above 10ms is concerning) and etcd_disk_backend_commit_duration_seconds (time to commit a batch — a p99 above 25ms is a warning).
Etcd is extremely sensitive to disk latency. Running Etcd on a shared disk with other I/O-heavy workloads (logs, container images) is a common cause of intermittent leader elections and write timeouts. Production Etcd should run on dedicated SSDs.
A Minimal Alert Set
Three Prometheus alerts catch the most common Etcd failure modes before they become outages: one firing when database size crosses 80% of quota (sustained for 5 minutes, giving time to schedule defragmentation before it becomes urgent), one firing when leader changes exceed 3 within an hour (a sign of disk latency or network instability), and a critical alert firing the moment any member has been unreachable for a minute.
Key Point: Disk I/O latency is the single most common root cause of Etcd instability in production. Before blaming Etcd itself, always check etcd_disk_wal_fsync_duration_seconds — if the p99 is above 10ms, the disk is the problem. Moving Etcd to a dedicated NVMe SSD or a separate disk from the OS and container runtime often resolves chronic instability without any Etcd configuration changes.