Your preparation
0 of 0 safeguards readyIncident worksheet
Make the next decision with evidence
Protect data and availability, determine the migration's actual state, and complete, compensate, or restore without rerunning unsafe steps.
Evidence→Decision→Action→Proof
Capture before evidence disappears
- Record migration ID, code, checksum, actor, start and stop time, transaction state, locks, rows changed, schema diff, and application versions.
- Preserve logs, database snapshots, replication lag, failed statements, query plans, and a sample of affected records.
- List nontransactional effects such as index builds, backfills, triggers, queues, search indexes, files, and external calls.
Decisions that change the response
| Question | Act when | Action |
|---|---|---|
| Rollback or finish forward? | The migration has committed partial or nontransactional state. | Inspect actual state first; use an idempotent completion or explicit compensation instead of blind rerun. |
| Keep serving traffic? | Both deployed application versions tolerate the observed schema and write behavior. | Continue only compatible paths; otherwise stop writes or the affected feature. |
Proof that recovery worked
- Expected schema, constraints, indexes, row counts, null rates, and business invariants match the migration plan.
- Old and new application versions behave according to the compatibility window.
- Replication, backups, queries, jobs, and customer workflows remain healthy after completion.
Controls to put in place
- Use expand-and-contract changes, small batches, checkpoints, and idempotent backfills.
- Test against production-shaped volume with lock, timeout, replication, and rollback observation.
- Take and verify a recovery point before irreversible changes; monitor business invariants during rollout.
Tabletop drill
Interrupt a staged backfill halfway. Identify committed state, resume from a checkpoint, run the compatibility matrix, and verify counts and application behavior.
Escalate when
Use database specialists when locks, replication, corruption, or irreversible type and constraint changes exceed the tested recovery plan.
What this means
Rolling back application code may not roll back the database. Partial, long-running, or irreversible changes can make both versions unsafe.
Warning signs
- Migration status differs across instances or environments.
- Database locks, replication lag, or query errors spike during release.
- Old code fails against the new schema.
- Row counts or transformed values are incomplete.
Recover now
First 15 minutes
- Stop additional deployments and migration runners.
- Record applied steps, locks, logs, row counts, and current schema.
- Choose roll forward or rollback using tested procedures and data-loss impact.
- Restrict writes if mixed versions can corrupt data.
Today
- Complete or reverse the migration in bounded, observable steps.
- Reconcile every affected row against source rules.
- Restore compatible application versions gradually.
- Add an expand-contract path for the corrected migration.
Verify recovery
- Schema matches the intended version on every database.
- Old and new data satisfies business invariants.
- Replication, locks, latency, and error rates are normal.
- Restarting a migration cannot repeat destructive work.
Prepare now
Access
- Production migrations run through a restricted, reviewed identity.
Backups and evidence
- A restorable backup or snapshot exists before destructive changes.
- Migration progress and row counts are recorded.
Contacts and ownership
- One person owns go/no-go and another verifies data.
Practice
- The migration and rollback or roll-forward path were rehearsed on production-like data.
Common mistakes
- Assuming application rollback reverses schema changes.
- Running a huge data rewrite in one transaction.
- Retrying a non-idempotent migration.
Sources
Last reviewed July 19, 2026Guidance changes. Confirm provider-specific actions in the linked official sources.