DisasterLab
PlaybooksProduction database corrupted
Datacritical50 minutes to prepare

Production database corrupted

Production records still exist but values, relationships, indexes, or internal storage are no longer trustworthy.

01DetectConfirm the signal
02ContainStop more damage
03RecoverRestore control
04VerifyProve it works

What do you need right now?

Prepare before it happensRecover from this now

Your preparation

0 of 0 safeguards ready
0%
Continue the prevention checklist
Incident worksheet

Make the next decision with evidence

Stop corruption from spreading, preserve the earliest clean recovery paths, and choose repair or restore based on verified business consistency.

EvidenceDecisionActionProof

Capture before evidence disappears

  • Capture engine errors, checksums, failed queries, affected pages or tables, replication state, storage health, schema, version, and the first known bad record.
  • Preserve snapshots, replicas, logs, write-ahead logs, exports, and application events before automated retention or failover changes them.
  • Map affected records to customers, transactions, derived indexes, caches, files, and external systems that may contain corroborating state.

Decisions that change the response

QuestionAct whenAction
Stop writes?New writes can propagate or obscure corruption and no safe partition can be isolated.Enter read-only or maintenance mode and protect replicas from promotion or replay.
Repair or restore?Repair can prove logical consistency without discarding valid history.Repair only in a clone; otherwise restore to a clean point and reconcile the missing interval.

Proof that recovery worked

  • Engine integrity checks, schema constraints, relationships, counts, totals, and sampled customer workflows pass.
  • Replicas and backups are recreated from verified clean state rather than inheriting corruption.
  • Every reconstructed or lost transaction in the gap has a documented business outcome.

Controls to put in place

  • Run integrity checks and restore tests that include application-level totals and invariants.
  • Keep point-in-time recovery and isolated copies beyond the likely corruption-detection delay.
  • Use checksums, constraints, idempotent event records, and reconciliation against payment or vendor systems.
Tabletop drill

Corrupt synthetic rows in a staging copy. Detect the invariant failure, freeze writes, compare repair with point-in-time restore, reconcile the gap, and rebuild a replica.

Escalate when

Use database recovery specialists before in-place repair, replica promotion, or destructive maintenance when the corruption boundary is unknown.

What this means

The application may appear online while reading or writing incorrect data. Continuing normal traffic can spread corruption and make recovery harder.

Warning signs

  • Integrity checks, replication, or database logs report corruption.
  • Totals, relationships, or customer records contradict source events.
  • Reads differ across replicas or repeated queries.
  • Errors begin after a migration, restore, storage failure, or faulty job.

Recover now

First 15 minutes

  1. Stop or restrict writes that could spread the damage.
  2. Capture database, application, and infrastructure logs plus exact detection time.
  3. Identify the last known-good point and affected tables or records.
  4. Clone data for investigation instead of experimenting on the only production copy.

Today

  1. Choose repair, point-in-time recovery, or replay based on measured scope.
  2. Restore into an isolated environment and run integrity and business checks.
  3. Reconcile changes made after the restore point from trusted event sources.
  4. Reopen writes gradually and monitor invariant failures.

Verify recovery

  • Database-native integrity checks pass.
  • Business totals and relationships match independent records.
  • Replicas agree and new writes remain consistent.
  • A sample of affected customers is verified.

Prepare now

Access

  • Destructive database actions require restricted roles and review.

Backups and evidence

  • Point-in-time recovery and logical exports cover critical data.
  • Business invariants are tested independently of database health.

Contacts and ownership

  • A decision owner can pause writes and approve restoration.

Practice

  • A recent restore test included application-level reconciliation.

Common mistakes

  • Trusting a successful restore without checking business meaning.
  • Running repair commands directly on the only copy.
  • Leaving writes open while measuring scope.

Sources

Last reviewed July 19, 2026Guidance changes. Confirm provider-specific actions in the linked official sources.