Why the Database Is the Hard Part
Code is easy to version and roll back — git revert takes seconds. Data is different. A database holding 500 million rows has "gravity": you can't just swap it out like a container image. Every schema change must be applied to live data, and that operation has real consequences.
If your CI/CD pipeline automatically runs flyway migrate or liquibase update before deploying new application pods, you expose yourself to two categories of failure:
1. Operational Failures (Locking)
Many DDL (Data Definition Language) operations — ALTER TABLE, ADD INDEX, MODIFY COLUMN — acquire a table-level lock while they execute. On a 100 GB table, adding an index can take 20–40 minutes. During that time, every write to that table blocks. Your application hangs. Users see timeouts. This is a self-inflicted outage triggered by a routine deployment.
Modern databases like PostgreSQL and MySQL 8+ support online DDL for some operations (e.g., CREATE INDEX CONCURRENTLY in Postgres), but not all. You must know which operations are safe and which are not.
2. Semantic Failures (Breaking Changes)
Suppose you rename a column from user_name to username. You run the migration, then start rolling out the new application pods. For the next 3–5 minutes, the old pods (still running) are executing SELECT user_name FROM users — a column that no longer exists. Every query fails. Your deployment window becomes an outage window.
The Rule of Two Versions
To achieve zero-downtime deployments, your database schema must always support two consecutive versions of the application code simultaneously:
- The Old Version — currently running in production, handling live traffic.
- The New Version — being rolled out pod by pod.
This constraint rules out a large class of "simple" migrations. You cannot rename a column in one step. You cannot drop a column the moment you stop using it. You cannot add a NOT NULL constraint to a column that existing code might not populate. Every schema change must be decomposed into a sequence of safe, additive steps.