Free — no signup required

Rollback Strategies: The Myth of "Undo

3 min read

Why "Undo" Is Harder Than It Sounds

When a bad application deployment happens, rollback is simple: redeploy the previous container image. The application is stateless, so there's nothing to lose. Database rollbacks are fundamentally different because the schema change may have already modified or deleted data.

Consider this migration:

-- V5__remove_legacy_column.sql
ALTER TABLE users DROP COLUMN legacy_notes;

The "undo" script would be:

-- V5__undo.sql
ALTER TABLE users ADD COLUMN legacy_notes TEXT;

Running the undo script restores the column, but every value that was in legacy_notes is gone forever. The rollback script creates a structurally correct but semantically empty column. Any application code that depended on that data will now behave incorrectly in a different way.

The "Down" Script Approach

Liquibase supports explicit rollback (called "undo") scripts. Flyway Pro supports them. In Liquibase, each changeset pairs its forward change (like dropping a column) with an explicit <rollback> block describing the inverse operation — and a rollback to a specific tagged version is a single command that walks the changesets back in reverse.

The problems with automated rollbacks in production:

  1. Data loss is silent. The rollback succeeds (exit code 0), but data is gone. No alarm fires.
  2. Rollback scripts are undertested. Teams run their forward migrations hundreds of times in CI. Rollback scripts are often run zero times before a production incident — exactly when you need them to work perfectly.
  3. Partial rollbacks create inconsistency. If V5, V6, and V7 are deployed and V6 has a bug, rolling back V6 while keeping V7 may be impossible if V7 depends on V6's schema.

The Fix-Forward Strategy

Most high-velocity engineering teams — including those at companies like GitHub, Shopify, and Stripe — prefer Fix Forward as their primary strategy.

The principle: when a migration causes a problem, you do not run a rollback script. You write a new migration that corrects the issue — for example, a follow-up migration that restores the column a prior one removed — and push it through the standard pipeline like any other change.

Why Fix Forward is safer:

  • The pipeline is the only path to production. No manual emergency SQL.
  • The history remains linear and auditable.
  • The fix goes through the same review, testing, and approval process as any other change.
  • No risk of data loss from a poorly-written rollback script.

When rollbacks are appropriate: For non-destructive changes (adding a nullable column, adding an index), automated rollbacks are safe and reasonable. The risk is specifically with destructive changes (dropping columns, dropping tables, changing column types). A mature database CI/CD policy distinguishes between these two categories.

Interview Tip

"How do you handle a bad database migration in production?" is a standard SRE interview question. The answer that impresses: distinguish between schema rollback (safe for additive changes) and data rollback (requires a backup restore). Explain the Fix Forward strategy and why it's preferred for destructive changes. Mention that the real defense is pre-production testing and the Expand-Contract pattern — not rollback scripts.

This is one of 18 chapters

Every chapter is free — Kubernetes, Terraform, SRE, distributed systems, and more — with fast daily review built in.

Sign up free