Your preparation
0 of 0 safeguards readyIncident worksheet
Make the next decision with evidence
Protect core transactions, understand whether demand is valuable or abusive, and scale each bottleneck without multiplying retries or cost.
Evidence→Decision→Action→Proof
Capture before evidence disappears
- Graph requests, users, conversion, cache hit rate, latency, errors, saturation, queue age, database load, third-party calls, and cost by endpoint and tenant.
- Record marketing, launch, bot, attack, webhook, crawler, and retry sources with correlation IDs.
- Capture current autoscaling limits, quotas, pool sizes, concurrency, rate limits, and the first saturated dependency.
Decisions that change the response
| Question | Act when | Action |
|---|---|---|
| Scale or shed? | Demand is legitimate but a downstream system cannot scale safely or economically. | Prioritize core paths, queue safe work, cache reads, and disable noncritical features before scaling blindly. |
| Queue or reject writes? | Operations are idempotent and customers can see delayed status. | Queue with bounded age; otherwise reject clearly instead of accepting work you may lose. |
Proof that recovery worked
- Core-path latency, errors, queue age, database load, and cost stay within declared limits at peak.
- Retries are bounded with jitter and do not create a second wave during recovery.
- Deferred work completes once, and customers can reconcile every accepted transaction.
Controls to put in place
- Load-test the complete dependency chain and publish capacity per critical path.
- Predefine degradation order, queue limits, per-tenant fairness, and cost alarms.
- Coordinate launches with providers and prewarm only measured bottlenecks.
Tabletop drill
Generate 20× staged demand through realistic paths. Enter each degradation level, protect one core transaction, cap a tenant, drain the queue, and calculate cost.
Escalate when
Use provider capacity support when quotas or regional resources constrain a legitimate event; involve security when traffic signals indicate abuse.
What this means
More traffic is useful only while the critical customer path works. Protect sign-in, checkout, payment webhooks, and existing customers before serving every page or feature.
Do not scale blindly. The real bottleneck may be the database, connection pool, third-party API, queue, function limit, or spending cap.
Warning signs
- Latency, error rates, database connections, queue depth, or CPU rise sharply.
- Checkout works slowly while marketing pages remain available.
- Autoscaling reaches a configured maximum or new instances fail health checks.
- A third-party API starts rate limiting requests.
- Cloud cost alerts rise faster than revenue.
Recover now
First 15 minutes
- Confirm whether traffic is legitimate. Compare CDN, WAF, application, campaign, and analytics data. Block obvious abuse without blocking the launch audience.
- Protect the critical path. Disable expensive secondary features, background refreshes, exports, previews, and nonessential integrations.
- Cache safe public responses. Increase CDN caching for static pages and repeated anonymous requests. Do not cache personalized or authenticated data accidentally.
- Queue or rate-limit excess demand. Use a waiting room, request limits, or a temporary invite flow to keep the origin within a known capacity.
- Watch the bottleneck and spend. Track errors, latency, database saturation, queue depth, provider limits, and cost at short intervals.
Today
- Scale the constrained component within tested limits.
- Increase connection pools and concurrency only when the database and downstream services can accept them.
- Move slow work to queues where retry and idempotency are safe.
- Pause paid traffic if customers cannot complete the main action.
- Publish a short status update when delays affect customers.
- Keep a record of peak demand, protective changes, errors, lost actions, and cost.
Verify recovery
- Sign-in, checkout, payment confirmation, and support work at an acceptable latency.
- Error rates and queues return to normal without dropping customer actions.
- Database, compute, and third-party limits have safe headroom.
- Duplicate retries did not create duplicate orders, charges, or emails.
- Cost remains inside an explicit emergency ceiling.
- Temporary limits and disabled features have owners and removal dates.
Prepare now
Capacity
- The critical customer path is identified and monitored separately.
- CDN caching is configured for safe public content.
- A waiting-room, rate-limit, or queue-all control is ready.
- Provider quotas, autoscaling limits, and spending alerts are documented.
- Third-party API limits and fallback behavior are known.
Safety
- Payment and order operations are idempotent.
- Expensive optional features have kill switches.
- Load tests cover the database and downstream services, not only the homepage.
- A status page can be updated while production is overloaded.
Practice
- A test has routed excess users to a queue and verified that accepted orders are not duplicated or lost.
Common mistakes
- Scaling only the web server. The database or third-party service may fail first.
- Caching personalized responses. A rushed cache rule can expose one customer’s data to another.
- Leaving retries unlimited. Retries amplify overload and duplicate work.
- Keeping ads running through a broken checkout. You pay to create frustrated customers.
- Removing protection too early. Demand often returns in waves.
Sources
- Cloudflare: Waiting Room
- Cloudflare: Control waiting-room traffic
- Cloudflare: Monitor waiting-room status
Last reviewed July 19, 2026Guidance changes. Confirm provider-specific actions in the linked official sources.