Critical SaaS provider outage

Your preparation

0 of 0 safeguards ready

Incident worksheet

Make the next decision with evidence

Keep the business's essential workflow moving, preserve requests for replay, and distinguish provider recovery from your own integration failures.

EvidenceDecisionActionProof

Capture before evidence disappears

Capture provider status, request IDs, response codes, timeouts, webhook gaps, queue depth, affected features, and the last known-good transaction.
List data held only by the vendor, pending writes, callbacks, exports, tokens, and manual work performed during the outage.
Measure customer and internal impact by workflow rather than treating every vendor error as equal.

Decisions that change the response

Question	Act when	Action
Queue, degrade, or disable?	The operation's idempotency, ordering, expiry, and customer expectation are known.	Queue safe work, degrade read paths, and disable actions that could duplicate or lose state.
Use a substitute?	The fallback has compatible data, security, legal, and operational behavior.	Switch only the pre-approved workflow; avoid improvised data uploads to a new vendor.

Proof that recovery worked

New requests follow the declared degraded behavior and customers see honest status.
Queued work replays once, in order where required, with duplicate and failure reconciliation.
Provider recovery is confirmed through your own end-to-end transaction, not status alone.

Controls to put in place

Define timeout, retry, idempotency, queue, and circuit-breaker behavior for each critical vendor operation.
Export irreplaceable vendor data and keep alternate contact and status routes.
Practice the degraded workflow and vendor recovery reconciliation.

Tabletop drill

Return timeouts and delayed webhooks from a vendor stub. Enter degraded mode, queue safe work, communicate impact, restore the dependency, and reconcile duplicates.

Escalate when

Use provider support and account contacts when the public status lacks your failure; involve legal or procurement when contractual service, data access, or exit rights are at risk.

What this means

The provider may recover quickly, but your business still needs a decision: wait, degrade gracefully, switch to manual work, or activate a fallback. Avoid repeatedly changing your system while the provider is unstable.

Separate the provider’s public status from your actual customer impact. One may recover before the other.

Warning signs

Provider requests time out, fail, or return partial results.
Queues and retries grow while your own infrastructure appears healthy.
The provider status page reports an incident or maintenance.
Customers report missing email, payments, files, authentication, or support replies.
Your integration starts duplicating work after retries.

Recover now

First 15 minutes

Confirm the dependency. Compare provider status, your logs, a direct test, and reports from another network.
Stop retry amplification. Add backoff, pause nonessential jobs, and cap queues before they overload your systems.
Protect customer actions. Preserve accepted requests locally and show a truthful pending state. Do not say something succeeded when only the provider call started.
Activate the smallest fallback. Use cached reads, manual processing, a secondary channel, or a temporary feature disablement.
Open one incident timeline. Record provider updates, your impact, queue size, customer messages, and changes.

Today

Decide how long you can wait before switching to a deeper fallback.
Keep provider-bound work idempotent so replay does not duplicate charges, emails, orders, or files.
Update customers when the outage blocks an important action.
Reconcile queued and partially completed work after the provider recovers.
Remove temporary bypasses carefully and watch for delayed webhooks or duplicate events.

Verify recovery

The provider status is healthy and direct integration tests succeed.
Queues are draining at a controlled rate.
Accepted customer actions are completed exactly once.
Missing, delayed, and duplicate records have been reconciled.
Temporary manual work and bypasses are closed.
Customer-facing status matches actual service.

Prepare now

Dependency

Every critical provider has an owner, status page, support route, and incident threshold.
The product has explicit behavior for provider timeout, partial success, and retry.
Critical operations are idempotent.
A local record exists before sending irreplaceable work to a provider.

Fallback

The business has a manual or degraded path for the most important customer action.
Queues, retry limits, and circuit breakers are tested.
Provider data needed for continuity is exported regularly.
Customer status can be updated without the failed provider.

Practice

A drill has disabled the provider integration and safely replayed queued work.

Common mistakes

Retrying every request immediately. You overload both systems and create duplicates.
Trusting the public status page alone. Your account or region may still be broken.
Switching providers during a short outage. A rushed migration can create a second incident.
Losing accepted customer actions. Save them before the provider call.
Replaying without idempotency. Recovery creates duplicate charges or messages.

Sources

Last reviewed July 19, 2026Guidance changes. Confirm provider-specific actions in the linked official sources.