Your preparation
0 of 0 safeguards readyIncident worksheet
Make the next decision with evidence
Restore the intended audience and state, reverse unsafe side effects, and prevent a flag change from bypassing release and authorization controls.
Evidence→Decision→Action→Proof
Capture before evidence disappears
- Capture flag key, value, targeting rules, prerequisites, environment, actor, change time, SDK state, cache TTL, and evaluated cohorts.
- List exposed users, sessions, requests, experiments, writes, emails, charges, migrations, and background jobs.
- Preserve audit events, application evaluations, deployment changes, and customer reports around the activation window.
Decisions that change the response
| Question | Act when | Action |
|---|---|---|
| Disable globally? | The feature changes data, money, permissions, privacy, or stability and the affected cohort is uncertain. | Turn it off at the authoritative service and verify caches and clients converge. |
| Reverse side effects? | The flag caused persistent writes or external actions. | Build a scoped reconciliation from event evidence; do not blindly rerun or delete. |
Proof that recovery worked
- Server, workers, clients, edge caches, and background jobs evaluate the intended value.
- Affected cohorts and persistent side effects are enumerated and reconciled.
- A user cannot gain authorization merely by changing a client-visible flag.
Controls to put in place
- Keep flags separate by environment with owners, expiry dates, descriptions, and audited changes.
- Require review for high-impact targeting and make kill switches server-authoritative.
- Test on/off, stale-cache, missing-provider, and partial-rollout behavior.
Tabletop drill
Enable a harmless staging flag for the wrong cohort. Detect it, disable across caches and workers, identify evaluated users, and reconcile a synthetic side effect.
Escalate when
Use privacy, security, or finance review when the flag exposed restricted data, changed authorization, charged customers, or triggered irreversible external actions.
What this means
The deployed code may be unchanged, but runtime behavior can shift instantly. Flags controlling billing, authorization, migrations, or data writes deserve incident-level care.
Warning signs
- Behavior changes without a deployment.
- Exposure percentage, audience, or environment differs from the plan.
- Error, conversion, cost, or support metrics move after a flag edit.
- Audit logs show an unexpected flag update.
Recover now
First 15 minutes
- Capture the flag value, targeting, editor, and change time.
- Return to the last known-safe configuration or disable the feature.
- Pause related writes, billing, or outbound actions if side effects may continue.
- Avoid deleting the flag while old application versions still reference it.
Today
- Identify every user, request, or job exposed during the window.
- Reverse incorrect side effects through reviewed, idempotent operations.
- Fix targeting and add an approval rule for high-risk flags.
- Test behavior when the flag service is unavailable.
Verify recovery
- Intended and unintended cohorts behave as expected.
- Incorrect side effects are reconciled.
- Audit history and ownership are intact.
- Safe defaults apply during provider failure.
Prepare now
Access
- Production flag changes use named accounts and least privilege.
Backups and evidence
- Flag values, targeting, and audit history are exportable.
Contacts and ownership
- Every high-risk flag has an owner, expiry date, and rollback value.
Practice
- A staged rollout and emergency disable have been tested.
Common mistakes
- Treating configuration as lower risk than code.
- Rolling back code while leaving the flag active.
- Keeping stale flags indefinitely.
Sources
Last reviewed July 19, 2026Guidance changes. Confirm provider-specific actions in the linked official sources.