Your preparation
0 of 0 safeguards readyIncident worksheet
Make the next decision with evidence
Stop incompatible calls, preserve customer operations, and adapt against observed vendor behavior rather than assumptions.
Evidence→Decision→Action→Proof
Capture before evidence disappears
- Capture request and response bodies with secrets removed, headers, version, schema, status, request IDs, latency, and first failure time.
- Compare vendor changelog, status, SDK version, contract tests, webhook events, and last known-good fixtures.
- List affected operations, tenants, retries, queued work, stored vendor IDs, and irreversible side effects.
Decisions that change the response
| Question | Act when | Action |
|---|---|---|
| Pin, adapt, or disable? | A known-good API or SDK version remains supported. | Pin it while implementing contract changes; otherwise disable unsafe writes and preserve requests. |
| Replay failed operations? | Their outcome can be proven and an idempotency mechanism exists. | Replay only unresolved operations; reconcile ambiguous responses first. |
Proof that recovery worked
- Contract tests cover success, error, pagination, webhook, authentication, and rate-limit behavior against the live vendor sandbox.
- Stored IDs and state reconcile with the vendor for every operation in the incident window.
- Old and new versions cannot process the same queued item twice.
Controls to put in place
- Wrap vendors behind a small adapter and keep recorded contract fixtures.
- Monitor deprecations, schema drift, SDK releases, and webhook failures.
- Use idempotency and a reconciliation job for every consequential external write.
Tabletop drill
Change a vendor stub's field, error, and pagination behavior. Detect the contract failure, disable unsafe writes, adapt the integration, and reconcile queued operations.
Escalate when
Use vendor engineering support when behavior contradicts documentation or data outcomes are ambiguous; involve counsel when the change violates contractual notice or availability commitments.
What this means
Payments, imports, messages, identity, or core product features may fail even though both systems remain online.
Warning signs
- Response codes, schemas, or validation errors change suddenly.
- Only new accounts, regions, or objects fail.
- A vendor deprecation or version notice was missed.
- Retries increase without successful outcomes.
Recover now
First 15 minutes
- Capture exact sanitized requests, responses, headers, versions, and times.
- Check vendor status, changelog, deprecation, and support channels.
- Pause destructive retries and customer-visible side effects.
- Switch to the last tested API version or a documented degraded mode.
Today
- Update parsing, authentication, limits, or version configuration.
- Test against representative production-like responses.
- Replay failed work idempotently and reconcile external state.
- Add contract tests and deprecation ownership.
Verify recovery
- Critical API journeys work with current vendor behavior.
- Failed work is reconciled without duplicates.
- Unsupported versions or fields are removed.
- Alerts identify future contract drift.
Prepare now
Access
- Vendor developer and support accounts have two owners.
Backups and evidence
- Sanitized contract fixtures and failure responses are retained.
Contacts and ownership
- Every critical API has an owner, version, changelog, and fallback.
Practice
- Contract tests simulate missing fields, new fields, errors, and limits.
Common mistakes
- Retrying a permanent contract error.
- Logging secrets while capturing requests.
- Depending on undocumented response behavior.
Sources
Last reviewed July 19, 2026Guidance changes. Confirm provider-specific actions in the linked official sources.