Where Should the Migration Script Run?
In Kubernetes, you need to run your schema migration tool (Flyway, Liquibase, Rails db:migrate, Alembic) as part of your deployment pipeline. You have two common options, and choosing the wrong one creates subtle but serious problems.
Option 1: Kubernetes Job (Preferred)
A Job is a Kubernetes object that runs a container to completion. Your CD pipeline creates the Job, waits for it to succeed, and only then triggers the Deployment rollout. The Job's spec runs the migration tool once (restartPolicy: Never) with backoffLimit: 0, so a failure is never silently retried, and passes it the database connection details via environment variables sourced from a Secret.
Why backoffLimit: 0 matters: If a migration fails partway through, retrying it automatically can cause double-application of partially-idempotent scripts. Fail fast, alert a human, and let them assess the state before retrying.
Option 2: InitContainers
An initContainer runs inside every Pod before the main application container starts. It's tempting because it's self-contained — the Pod handles its own migration by simply running the same migration command as one of its init steps.
Why this is dangerous at scale: If your Deployment scales to 50 replicas, Kubernetes will launch 50 Pods simultaneously. Each Pod's initContainer will attempt to run flyway migrate at the same time. Flyway uses a lock table to prevent concurrent migrations, so 49 of those containers will block waiting for the lock, hammering your database with connection attempts. This "thundering herd" can exhaust your database connection pool and cause the very outage you were trying to prevent.
Additionally, if the migration fails, every Pod fails to start. You have no running application at all — not even the old version, if the old Pods were already terminated.
Helm Hooks and ArgoCD Sync Waves
The production-grade approach is to use your CD tool's native sequencing primitives:
Helm Hook: annotating the migration Job with helm.sh/hook: pre-upgrade (plus a weight controlling ordering relative to other hooks, and a delete policy controlling cleanup) tells Helm: run this Job before upgrading any other resources, and only proceed if it succeeds.
ArgoCD Sync Wave: giving the migration Job a lower argocd.argoproj.io/sync-wave annotation value than the Deployment achieves the same effect declaratively. ArgoCD will apply all wave-0 resources and wait for them to become healthy before applying wave-1 resources. This gives you declarative, GitOps-native migration sequencing.
A common interview question is: "How do you run database migrations in Kubernetes?" Many candidates say "initContainers." The correct answer is a pre-deployment Job orchestrated via Helm hooks or ArgoCD sync waves. Explain why initContainers are dangerous at scale (thundering herd, connection pool exhaustion, no running fallback on failure) — this distinction signals real production experience.