DisasterLab
PlaybooksCloud region outage
Infrastructurecritical60 minutes to prepare

Cloud region outage

A cloud region or availability zone hosting critical workloads becomes unavailable or severely degraded.

01DetectConfirm the signal
02ContainStop more damage
03RecoverRestore control
04VerifyProve it works

What do you need right now?

Prepare before it happensRecover from this now

Your preparation

0 of 0 safeguards ready
0%
Continue the prevention checklist
Incident worksheet

Make the next decision with evidence

Confirm the regional failure, protect data consistency, and move only the services whose dependencies and recovery objectives support failover.

EvidenceDecisionActionProof

Capture before evidence disappears

  • Capture provider status, synthetic checks, regional control-plane errors, dependency failures, replication lag, queue depth, and last successful writes.
  • Map each service to databases, object stores, identity, DNS, secrets, certificates, queues, email, and third-party regional dependencies.
  • Record traffic, write ownership, data recovery point, failover actions, and any split-brain protection.

Decisions that change the response

QuestionAct whenAction
Fail over?The outage will exceed the service objective and the target region has current data and tested capacity.Freeze or fence writes, confirm replication, promote deliberately, then shift traffic in stages.
Run read-only?Reads are safe but write consistency or downstream processing cannot be guaranteed.Expose an explicit read-only mode and queue only operations proven safe to replay.

Proof that recovery worked

  • The target region handles production load with one authoritative writer and measured data lag.
  • DNS, sessions, jobs, webhooks, and asynchronous work resolve to the intended region.
  • Failback is a separate reviewed operation; recovered source-region traffic cannot create split brain.

Controls to put in place

  • Document regional dependencies and classify services as active-active, failover, read-only, or restore-only.
  • Replicate data and configuration with monitored recovery points and isolated regional credentials.
  • Exercise failover and failback under realistic load, including queues and third-party callbacks.
Tabletop drill

Block a staging region, fence writes, promote the secondary, shift synthetic traffic, verify asynchronous work, and fail back only after reconciling data.

Escalate when

Open urgent provider support when control planes or replication are impaired; use database specialists when write ownership or data divergence is uncertain.

What this means

Compute, databases, storage, identity, and control-plane actions can fail together. A theoretical second region does not help unless data, secrets, DNS, and capacity are ready.

Warning signs

  • The provider status page and your monitoring show regional failures.
  • New resources cannot be created or control-plane calls time out.
  • Multiple unrelated services fail in the same location.
  • Cross-zone recovery does not restore service.

Recover now

First 15 minutes

  1. Confirm affected region, services, and provider guidance.
  2. Freeze risky deployments and data migrations.
  3. Choose wait, degrade, or fail over using the documented recovery objective.
  4. Communicate what works, what does not, and the next update time.

Today

  1. Fail over only to an environment with verified data, secrets, dependencies, and capacity.
  2. Prevent split-brain writes between old and new regions.
  3. Reconcile queued and delayed work after recovery.
  4. Plan failback as a separate controlled change.

Verify recovery

  • Customer traffic uses a healthy environment.
  • Data freshness and write ownership meet the declared objective.
  • Critical dependencies and background work operate.
  • Failback will not duplicate or lose writes.

Prepare now

Access

  • Recovery identities work independently of the affected region.

Backups and evidence

  • Data replication and backups have measured recovery objectives.
  • Infrastructure configuration can recreate the service elsewhere.

Contacts and ownership

  • One person owns the failover decision and another verifies data.

Practice

  • A regional recovery drill has tested DNS, capacity, data, and secrets.

Common mistakes

  • Failing over an empty shell without current data or secrets.
  • Allowing two regions to accept conflicting writes.
  • Rushing failback because the status page is green.

Sources

Last reviewed July 19, 2026Guidance changes. Confirm provider-specific actions in the linked official sources.