AI Workflow Incident Response Runbook
A repeatable incident response runbook for small teams running AI workflows: when to declare an incident, how to contain it, and what to change so it does not happen again.
Why AI workflows need their own playbook
Automation failures do not always look like server outages. An AI workflow can fail silently: sending wrong data downstream, generating misleading drafts, or exhausting a quota without stopping. Traditional ops runbooks often miss these failure modes because the system still reports 200 OK while the output is wrong.
Small teams feel this faster than large ones. There is no dedicated SRE team. The same people who built the workflow also have to fix it, often while it is still running. A short, explicit runbook reduces that cognitive load.
Two useful reference points are observability patterns for AI workflows and circuit breakers, retries, and fallbacks. Together they form the detect-and-stop layer this runbook assumes already exists.
When to declare an AI workflow incident
The trigger should be written down, not debated live. Use the following thresholds:
- Wrong output reaches a user or customer. A draft, summary, classification, or decision was incorrect and was not caught before delivery.
- Upstream data is corrupted. The workflow rewrote a record, updated a CRM field, or sent a webhook based on bad AI output.
- Automation is now causing manual work. The team is spending more time fixing the workflow than it saved.
- Quota, cost, or rate limit risk. The workflow is looping, retrying without backoff, or generating traffic that will hit limits within the hour.
If any one of these is true, declare the incident. Do not wait for a second symptom.
The runbook: four steps
1. Triage
Answer three questions in five minutes:
- Is the workflow still running, or has it stopped?
- Did bad output reach outside the system, or is it contained?
- What is the blast radius: one user, one queue, or the full pipeline?
If the workflow is still running and the blast radius is growing, skip straight to containment.
2. Contain
Stop the spread before you start fixing. The right mechanism depends on how the workflow is hosted:
| Host | Containment action |
|---|---|
| Cron job | Disable the cron entry or move the script out of the active path. |
| Webhook listener | Return 503 or disable the route. |
| Queue consumer | Pause the queue or set visibility timeout high. |
| Third-party automation | Turn off the zap or flow. |
If the workflow already wrote bad data, flag the affected records for review instead of silently reverting them. Reverting without review can hide the scope of the incident.
3. Fix
Fix the workflow, not just the immediate output. A common mistake is patching one bad result and restarting the same workflow with the same prompt, tools, and data path. If the failure was caused by ambiguous instructions, vague success criteria, or missing validation, the same failure will repeat.
Make the smallest change that removes the failure mode, then test with one representative case before restoring full traffic.
4. Review
Write a short postmortem within 48 hours. It does not need to be formal. Include:
- What triggered the incident
- Where detection failed or was delayed
- What changed in the workflow before the failure
- One concrete change to prevent recurrence
The review is not complete until the team knows what to change in the workflow itself, not just what to do when it breaks.
Example: AI coding assistant ships a bad patch
A small team uses an AI coding assistant to generate migration scripts. The workflow reads the schema, generates SQL, and runs it in staging automatically. One morning the assistant generates a destructive migration: it drops a column that is still referenced by active features.
The incident unfolds quickly. Staging breaks within minutes. Because the workflow runs without human review, the bad migration propagates to the next scheduled deploy. The team declares an incident and pauses the automation.
In triage, the blast radius is limited to staging and one deploy queue. Containment is straightforward: disable the cron job and block the migration pipeline. Fix requires two changes: add a read-only validation step that checks for destructive operations, and require a human approval step before any migration is applied outside development.
The postmortem shows the assistant had been fine-tuning its output style recently, which made the schema parsing less literal. The team updates the prompt to require an explicit migration plan review before code generation.
Limits and notes
This runbook covers workflow-level incidents, not model outages or provider downtime. For availability problems, use standard infrastructure playbooks. For AI-specific reliability gaps, combine this runbook with circuit breakers and fallback paths so that failures degrade safely instead of failing loudly.