DisasterLab
PlaybooksProduction deploy breaks the site
Infrastructureserious30 minutes to prepare

Production deploy breaks the site

A release causes errors, missing pages, failed checkouts, corrupt writes, or unexpected customer behavior.

01DetectConfirm the signal
02ContainStop more damage
03RecoverRestore control
04VerifyProve it works

What do you need right now?

Prepare before it happensRecover from this now

Your preparation

0 of 0 safeguards ready
0%
Continue the prevention checklist
Incident worksheet

Make the next decision with evidence

Stop the faulty rollout, choose rollback or forward repair from evidence, and restore service without losing incompatible data changes.

EvidenceDecisionActionProof

Capture before evidence disappears

  • Record deploy ID, commit, artifact digest, feature flags, configuration, migration version, start time, actor, canary result, and the last known-good release.
  • Preserve errors, traces, health checks, saturation, customer reports, and request samples split by version, region, tenant, and endpoint.
  • List schema, queue, cache, search-index, object, and third-party side effects that a code rollback will not reverse.

Decisions that change the response

QuestionAct whenAction
Rollback or repair forward?The old artifact remains compatible with current data and configuration.Rollback immediately; otherwise stop writes or affected features and repair forward.
Drain or kill new instances?Requests or jobs have non-idempotent side effects.Drain safely, reconcile in-flight work, and kill only after ownership of each operation is clear.

Proof that recovery worked

  • Golden-path and failure-path checks pass against the production release and real dependencies.
  • Error, latency, saturation, queue, and business metrics return to the pre-deploy range.
  • Data migrations and side effects are reconciled; no mixed-version workers remain.

Controls to put in place

  • Use immutable artifacts, progressive delivery, automated health gates, and one-command rollback.
  • Design expand-and-contract migrations that tolerate old and new application versions.
  • Test rollback with queues, caches, flags, and data changes—not only application code.
Tabletop drill

Deploy a controlled failure to staging, pause rollout automatically, decide rollback compatibility, restore the prior artifact, and reconcile one simulated side effect.

Escalate when

Bring in database or provider specialists when data changed irreversibly, rollback is incompatible, the failure crosses regions, or customer transactions need manual correction.

What this means

Restore service before debugging every detail. A code rollback is usually fast, but it may not reverse database migrations, queued jobs, changed secrets, deleted storage, or external side effects.

The safe target is the last version known to work with the current data and infrastructure.

Warning signs

  • Errors, latency, support messages, or failed conversions rise immediately after a release.
  • Health checks pass while checkout or sign-in fails.
  • A migration, queue worker, or webhook starts changing data unexpectedly.
  • Only one browser, region, tenant, or feature flag is affected.
  • The deployment cannot be linked to a commit and change owner.

Recover now

First 15 minutes

  1. Stop the rollout. Pause automatic promotion, queued migrations, and further deploys.
  2. Measure customer impact. Test the critical path and compare error, latency, payment, and database metrics with the release time.
  3. Choose rollback or feature disablement. Prefer the smallest proven action that restores service.
  4. Check irreversible changes. Review schema migrations, background jobs, secrets, storage, DNS, and external API calls before rolling code back.
  5. Record the active and target versions. Keep timestamps, commit IDs, deploy IDs, and actions.

Today

  1. Roll back to the last compatible production deployment or disable the faulty feature.
  2. Verify sign-in, checkout, payments, webhooks, writes, email, and support.
  3. Reconcile failed or duplicated customer actions during the bad release.
  4. Fix the issue in a preview or staging environment.
  5. Redeploy with focused monitoring and a clear abort threshold.
  6. Tell customers when orders, payments, or data may need confirmation.

Verify recovery

  • The intended version serves all production routes.
  • Critical customer journeys pass from more than one network.
  • Error, latency, queue, and conversion metrics return to baseline.
  • Database schema and application version are compatible.
  • No duplicate charges, orders, messages, or jobs remain.
  • The faulty version cannot be promoted accidentally.

Prepare now

Deployment

  • Every deployment maps to a commit, owner, and timestamp.
  • Preview or staging checks cover the critical customer path.
  • Production can roll back without rebuilding from a developer laptop.
  • Risky features have independent flags or kill switches.

Data

  • Database migrations are backward-compatible or have a tested recovery plan.
  • Jobs and webhooks are idempotent.
  • A backup or recovery point exists before destructive migrations.
  • Release monitoring includes business actions, not only server health.

Practice

  • A recent drill rolled back a test deployment and verified data compatibility.

Common mistakes

  • Debugging in production while customers wait. Restore service first.
  • Assuming code rollback reverses data changes. It does not.
  • Deploying another untested fix immediately. You add uncertainty to the incident.
  • Checking only the homepage. Sign-in and checkout may still be broken.
  • Forgetting queued work. Old jobs can continue damage after rollback.

Sources

Last reviewed July 19, 2026Guidance changes. Confirm provider-specific actions in the linked official sources.