Keeping Docs Fresh: The "Bit Rot" War

Bit rot is the phenomenon where documentation becomes incorrect over time simply because the system it describes keeps changing. A runbook written six months ago may reference a server that was renamed, a script that was moved, or a team that was reorganized. Wrong documentation is worse than no documentation — it sends an engineer confidently down a path that leads nowhere.

Strategy 1: Docs as Code

Do not store runbooks in a separate wiki that lives outside the engineering workflow. Store them as Markdown files in the same Git repository as the application code.

When a developer changes a port number in config.yaml, they open a pull request. The runbook file runbooks/high-latency.md is right there in the same repository. The PR reviewer can ask: "Did you update the runbook?" This makes runbook updates a natural part of the code review process rather than an afterthought.

Practical structure:

my-service/
├── src/
├── deploy/
├── runbooks/
│   ├── high-latency.md
│   ├── disk-space-low.md
│   └── database-cpu-high.md
└── README.md

Strategy 2: The "Broken Link" Test

Treat runbooks like software — test them automatically.

Every alert in your monitoring system should have a runbook_url metadata tag pointing to the relevant runbook. Write a CI/CD pipeline job that:

1. Queries your alerting system (e.g., Prometheus Alertmanager, PagerDuty) for all active alert rules.
2. Extracts the runbook_url from each alert's annotations.
3. Makes an HTTP GET request to each URL.
4. Fails the build if any URL returns a 404, a redirect to a generic "page not found" wiki page, or an empty document.

This turns runbook maintenance from a manual discipline into an automated gate. A missing runbook blocks deployment.

Example Prometheus alert with runbook annotation:

groups:
  - name: checkout-service
    rules:
      - alert: HighHTTP500Errors
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
        for: 3m
        labels:
          severity: critical
        annotations:
          summary: "High 5xx error rate on checkout service"
          runbook_url: "https://github.com/myorg/checkout-service/blob/main/runbooks/high-http-500-errors.md"

Strategy 3: GameDay Verification

During a GameDay (a controlled failure drill), the primary artifact being tested is not just the system — it is the runbook. A runbook is only "verified" if an engineer who did not write it can follow it successfully during a drill without asking clarifying questions.

Establish a formal verification lifecycle:

If an engineer has to ask "Wait, is this step still valid?" during a drill, the runbook has failed its verification and must be updated before it earns VERIFIED status.