Your preparation
0 of 0 safeguards readyIncident worksheet
Make the next decision with evidence
Keep trusted release and repair work possible without bypassing the controls that make artifacts trustworthy.
Evidence→Decision→Action→Proof
Capture before evidence disappears
- Capture provider status, API and job errors, queue times, webhook delivery, runner health, affected repositories, and the last successful artifact digest.
- List releases, hotfixes, security patches, scheduled jobs, previews, and infrastructure changes blocked by CI.
- Record which build inputs, secrets, signing steps, approvals, caches, and provenance exist only inside the provider.
Decisions that change the response
| Question | Act when | Action |
|---|---|---|
| Wait or use a fallback? | A production or security deadline is shorter than the provider's credible recovery window. | Use the documented fallback pipeline with the same reviewed source, tests, signing, and approval evidence. |
| Manual deploy? | No fallback exists and customer harm from waiting exceeds the release risk. | Require two-person review, record commands and hashes, use short-lived credentials, and schedule a normal rebuild. |
Proof that recovery worked
- The deployed artifact maps to reviewed source and passed the required tests and approvals.
- No permanent credential or protection bypass remains after the provider recovers.
- Queued provider jobs are cancelled or reconciled so they cannot redeploy stale code.
Controls to put in place
- Keep build definitions portable and critical artifacts reproducible outside one provider.
- Document an emergency release path with isolated credentials, signing, and two-person approval.
- Store source, dependency locks, keys that can be reissued, and runbooks outside CI.
Tabletop drill
Disable the main pipeline in a sandbox. Build and sign one release through the fallback, deploy it to staging, record evidence, then reconcile queued jobs when CI returns.
Escalate when
Contact provider support when status is unclear or queues risk duplicate releases; involve security when an outage coincides with suspicious workflow or credential activity.
What this means
Production may remain online, but urgent fixes and rollback automation are unavailable. An improvised laptop deployment can bypass the controls that make releases safe.
Warning signs
- Jobs remain queued, runners disconnect, or artifacts cannot be retrieved.
- Provider status shows a control-plane or regional incident.
- Repository checks never report or deployment environments remain locked.
- Tokens work but every workflow fails similarly.
Recover now
First 15 minutes
- Confirm provider scope and preserve failed job details.
- Freeze nonessential releases and identify whether an urgent change is truly required.
- Use the tested emergency release path only with peer verification.
- Do not weaken branch rules permanently to bypass a temporary outage.
Today
- Build from a trusted commit in a controlled environment.
- Verify artifact provenance, tests, configuration, and rollback before release.
- Restore normal branch and environment controls after provider recovery.
- Compare any manually deployed state with the canonical pipeline.
Verify recovery
- Normal pipelines build and deploy a harmless change.
- Production matches a known source commit and artifact.
- Temporary credentials and rule exceptions are removed.
- Queued jobs cannot deploy stale versions unexpectedly.
Prepare now
Access
- Emergency deployment access is separate, restricted, and audited.
Backups and evidence
- Build instructions, lockfiles, and deployment configuration live with source.
Contacts and ownership
- Release and rollback decisions have named owners.
Practice
- A controlled emergency build and deploy has been rehearsed.
Common mistakes
- Deploying from an untrusted developer machine.
- Disabling protections and forgetting to restore them.
- Letting old queued jobs deploy after recovery.
Sources
Last reviewed July 19, 2026Guidance changes. Confirm provider-specific actions in the linked official sources.