Your preparation
0 of 0 safeguards readyIncident worksheet
Make the next decision with evidence
Stop backlog growth, identify the blocking class, and drain work without duplication, broken ordering, or overload.
Evidence→Decision→Action→Proof
Capture before evidence disappears
- Capture queue depth and age, job type, state, attempts, lease, worker version, error, payload ID, dependency latency, and dead-letter counts.
- Record deploys, autoscaling, quotas, database locks, provider failures, poison messages, and the last completed job for each partition.
- Classify jobs by idempotency, ordering, expiry, customer visibility, and side effects before retrying anything.
Decisions that change the response
| Question | Act when | Action |
|---|---|---|
| Retry, skip, or quarantine? | The job's outcome and idempotency can be established. | Retry unresolved idempotent work; quarantine poison jobs; reconcile ambiguous side effects first. |
| Add workers? | The bottleneck is worker capacity and dependencies can absorb the concurrency. | Scale gradually with rate and database limits; more workers can deepen contention. |
Proof that recovery worked
- Oldest age and depth fall steadily without duplicate business effects or new dependency saturation.
- Poison jobs are isolated with owners, evidence, and a replay or correction plan.
- New jobs complete within the objective while the backlog drains.
Controls to put in place
- Make consequential jobs idempotent with explicit business operation IDs.
- Alert on age, retries, dead letters, lease expiry, and throughput by job type.
- Provide pause, quarantine, replay, and reconciliation tools that operators can test safely.
Tabletop drill
Insert a poison job and slow a dependency in staging. Find the blocking class, quarantine it, scale within limits, drain the queue, and prove each business action occurred once.
Escalate when
Use database or vendor support when locks and dependency limits cause the stall; involve customer operations when delayed jobs expire or require manual correction.
What this means
The website can appear healthy while essential work accumulates. Blind replay can duplicate charges, emails, or state changes.
Warning signs
- Queue age and depth rise while throughput falls.
- Workers repeatedly crash on one message.
- Customers see pending states that never finish.
- Dead-letter or retry counts increase.
Recover now
First 15 minutes
- Measure oldest job, queue depth, throughput, failures, and worker health.
- Pause producers if backlog growth threatens recovery.
- Isolate poison messages rather than retrying them endlessly.
- Protect non-idempotent jobs such as charges and outbound messages.
Today
- Fix worker capacity, dependency failure, schema mismatch, or poison job.
- Resume in small batches with deduplication and rate limits.
- Reconcile externally visible actions against provider records.
- Inform affected customers about delays and corrected states.
Verify recovery
- Queue age and depth trend to normal.
- Failed work has an explicit disposition.
- Replayed jobs do not create duplicates.
- New work completes within the expected time.
Prepare now
Access
- Operators can pause producers and workers separately.
Backups and evidence
- Job ID, attempts, payload reference, and outcome are traceable.
- Dead-letter work is retained safely.
Contacts and ownership
- Every critical queue has an owner and maximum tolerable age.
Practice
- A poison job and safe replay have been tested.
Common mistakes
- Purging the queue before measuring business impact.
- Adding workers when a dependency is failing.
- Replaying payments or emails without idempotency.
Sources
Last reviewed July 19, 2026Guidance changes. Confirm provider-specific actions in the linked official sources.