Your preparation
0 of 0 safeguards readyIncident worksheet
Make the next decision with evidence
Preserve provider truth, stop unsafe downstream actions, and replay each event exactly once after the receiver is trustworthy.
Evidence→Decision→Action→Proof
Capture before evidence disappears
- Capture event ID, type, object, creation time, delivery attempts, signature result, response, latency, endpoint version, deploy, and provider request ID.
- Graph receiver errors, queue depth, database failures, secret rotation, DNS, certificates, firewall, and account endpoint settings.
- List business state derived from webhooks: orders, subscriptions, access, refunds, invoices, emails, and ledger entries.
Decisions that change the response
| Question | Act when | Action |
|---|---|---|
| Replay now? | The receiver is fixed, idempotency exists, ordering assumptions are known, and provider object state can be queried. | Replay in bounded batches and reconcile each event to current provider truth. |
| Poll the provider? | Webhook delay threatens important state and API limits support a temporary reconciliation. | Poll affected objects by time window; do not create a permanent second source of truth. |
Proof that recovery worked
- The provider sees successful signed deliveries and your receiver records each event ID once.
- Provider objects, internal ledger, subscriptions, access, refunds, and invoices reconcile for the incident window.
- Out-of-order, duplicate, delayed, and unknown events pass tests without duplicate effects.
Controls to put in place
- Persist raw verified events before business processing and make handlers idempotent.
- Alert on provider delivery failure and last-success age, not only endpoint uptime.
- Keep a documented replay and time-window reconciliation tool.
Tabletop drill
Reject, duplicate, and reorder sandbox events. Repair the endpoint, replay a bounded window, query provider truth, and prove each business effect occurs once.
Escalate when
Use payment-provider support when event history or signatures are inconsistent; involve finance when access, invoices, refunds, or ledger state diverges.
What this means
Money may have moved even when your application did not update. Retrying without idempotency can create duplicate fulfillment or state changes.
Warning signs
- The payment provider reports failed endpoint deliveries.
- Paid customers remain pending or canceled customers retain access.
- Signature, timeout, version, or response-code errors rise.
- Provider state disagrees with local invoices or subscriptions.
Recover now
First 15 minutes
- Check provider event and delivery logs against endpoint responses.
- Fix reachability, signature verification, timeout, or version errors.
- Return success quickly and move slow processing to a queue.
- Do not bulk replay until handlers are idempotent.
Today
- Fetch provider events and authoritative object state for the missed window.
- Replay or reconcile in bounded batches keyed by event and object IDs.
- Correct customer access, fulfillment, and accounting differences.
- Monitor automatic retries so manual replay does not duplicate work.
Verify recovery
- New signed events receive timely successful responses.
- Provider and local payment state reconcile.
- Duplicate delivery produces no duplicate side effect.
- Missed events have explicit processed outcomes.
Prepare now
Access
- Two people can inspect endpoints, events, and signing secrets.
Backups and evidence
- Event IDs, object IDs, attempts, and outcomes are retained.
Contacts and ownership
- Payment reconciliation has an owner and support route.
Practice
- Failed, duplicate, delayed, and out-of-order events are tested.
Common mistakes
- Assuming webhooks arrive once or in order.
- Treating local state as the payment system of record.
- Doing slow business logic before returning success.
Sources
Last reviewed July 19, 2026Guidance changes. Confirm provider-specific actions in the linked official sources.