Is your AI incident response plan ready for agent failure?

An AI incident is not merely a model giving a bad answer. It starts when an AI system creates, or has a credible path to create, harm that your team must contain: exposed customer data, an unauthorized payment, deleted records, published false advice, or a compromised agent that can keep acting. Your response plan has to follow that harm across the model, tools, identities, memory, vendors, and people who approved the workflow.

Most existing security plans cover stolen credentials and vulnerable servers. Most reliability plans cover timeouts and failed queues. An agent can fail while every server stays healthy and every credential remains technically valid. It may use an allowed tool, under an approved identity, to perform the wrong action at machine speed. That gap is why an AI incident response plan needs its own triggers, evidence, authority, and runbooks.

The plan does not need a new department. It needs a small group that knows who can stop the agent, who decides whether customers or regulators must hear about it, and how to preserve enough context to explain what happened. If those decisions wait for a live incident, the agent will keep working while executives debate ownership.

Define an AI incident by harm, not model behavior

A useful incident definition starts with impact and loss of control. A hallucination inside an internal brainstorming chat may be ordinary model error. The same unsupported claim inserted into medical guidance, a customer invoice, or a public status update can be an incident because a person may act on it. Severity depends on the consequence, reach, and reversibility, not on whether the model output looks strange.

Teams routinely blur three different events. A model-quality defect is a repeatable shortfall against an evaluation target. A security event involves unauthorized access, manipulation, disclosure, or action. A safety or business-integrity event causes harmful advice, discrimination, contractual error, financial loss, or reputational damage even when nobody broke a security control. One event can occupy all three categories, but the distinction decides who joins the response and what containment means.

Write triggers that an on-call engineer can recognize without interpreting intent. Open an incident when an agent exposes protected data, executes an unauthorized or wrongly authorized action, produces harmful content that reaches a user or downstream system, loses required human oversight, behaves differently after a model or prompt change with material impact, or shows evidence of prompt injection, poisoned memory, or tool compromise. A near miss also deserves a record when only a failed permission check or a cautious employee prevented harm.

Use a severity matrix based on four questions: what data or operation was reachable, how many people or records may be affected, whether the action can be reversed, and whether the agent can continue. An agent with write access and an active credential deserves faster escalation than an offline answer awaiting review. Avoid a special low severity just because the root cause includes a hallucination. False information can create a high-severity incident when it enters production decisions.

NIST AI 600-1 tells organizations to align generative AI incident plans with mapped impacts, define ownership, rehearse regularly, learn from retrospectives, and check relevant reporting duties. That is stronger advice than creating a separate AI risk register that nobody on call can use. Put AI triggers into the same paging, case management, and executive escalation paths used for other material incidents, then add the evidence and containment steps unique to agents.

Detection has to watch control boundaries, not prose quality. Page on denied high-impact tool calls, repeated requests for broader access, new destinations, approval bypasses, cross-tenant retrieval, unusual memory writes, and bursts of side effects. Route softer signals, such as rising citation failures or user corrections, into review queues. A single awkward sentence should not wake the incident team, but an agent trying five ways to send the same record should.

Fixed thresholds work for controls with known limits. Transaction size, recipient count, tool-call rate, and data classification should have hard boundaries enforced outside the model. Behavioral baselines help find novelty, but do not let an anomaly score make the containment decision alone. A newly launched workflow is anomalous by definition, while a slow and steady leak may resemble normal traffic.

Connect user reports to execution traces. A support ticket that says “the assistant sent somebody else's address” should let the responder find the answer, retrieval records, and tool calls without searching several unrelated systems. Give employees and customers a clear reporting route, and train support staff to preserve the conversation identifier and time rather than asking the user to reproduce sensitive content in a new message.

Monitor the controls themselves. Alert when trace events stop arriving, when an agent uses an unrecognized prompt or model version, when a tool call lacks an authorization event, or when approval metadata disappears. Missing evidence during active agent execution is a response condition because the team has lost its ability to distinguish safe work from harmful work.

Create incident records in the language of business impact. “Agent returned status 200 with malformed reasoning” will not help privacy, legal, or operations decide what to do. “Billing agent may have sent one customer's invoice to another customer between these times; sending is paused and recipient scope is under investigation” states the known harm, boundary, containment, and uncertainty.

Test alert routing with synthetic events. Send a harmless denied tool call, a fake cross-tenant retrieval marker, and a trace with missing approval metadata through the production monitoring path. Confirm that each alert reaches the expected person with the trace ID and containment control attached. Dashboard visibility is not enough when nobody receives a page or knows which switch to use.

Classification can change as evidence arrives. Start at the highest credible severity supported by reach and capability, then lower it deliberately with a recorded reason. Teams get into trouble when the first reporter calls an event a minor hallucination and every later responder inherits that framing, even after discovering that the answer triggered a real transaction.

Give one incident commander authority to stop agents

One incident commander must be able to pause agent actions without waiting for the team that owns the model. The commander coordinates decisions and keeps a timeline; they do not personally debug every layer. Name a primary and backup for each production period, and give them tested access to disable tool execution, revoke agent credentials, stop queued jobs, and switch the affected workflow to a safe fallback.

The technical lead traces prompts, models, tools, memory, and deployment changes. The security lead handles suspected intrusion, credential exposure, forensics, and breach assessment. The product or operations owner judges user harm and selects the manual fallback. Legal and privacy staff decide notification and preservation duties. Communications owns consistent messages to customers, partners, and staff. An executive sponsor accepts material business risk when recovery requires a tradeoff the commander cannot make.

Do not assign the model vendor as the owner of your incident. A vendor can investigate its service, preserve provider-side logs, and explain model or policy changes. Your company still owns the workflow, the tool permissions, the data sent, and the decision to keep operating. Record vendor support routes, contract identifiers, emergency contacts, log-retention terms, and the exact information you can share without leaking more sensitive data.

A simple responsibility rule prevents committee response: the commander decides pace and containment, specialists advise within their fields, and the business owner decides whether a degraded manual process is acceptable. Anyone may report or escalate suspected harm. Only designated roles may declare recovery, notify affected people, or approve deletion of evidence.

The awkward case is an agent owned jointly by engineering and a business team. Resolve it before launch. Assign one operational owner for the whole workflow, even if separate teams own the model gateway, retrieval store, and connected applications. Shared ownership during normal work often becomes absent ownership during an outage.

Keep a decision log with timestamps, the decision maker, known facts, rejected options, and the next review time. This is not administrative decoration. When evidence changes, the log shows why the team revoked one credential but left another workflow running, and it stops the same argument from restarting on every handoff.

Preserve the evidence an agent actually leaves

Agent evidence is a chain of inputs, state, decisions, and side effects. Application logs that record only the final answer cannot show whether the model selected a tool, inherited poisoned memory, received malicious retrieved text, or acted after a human approval. Capture the chain before an incident, because you cannot reconstruct missing context afterward.

For every run, record a stable trace ID; user, tenant, and agent identities; model and version; system and user prompt versions; retrieved document identifiers and versions; memory reads and writes; tool names, arguments, results, and authorization decisions; approval events; output destinations; policy checks; token and cost limits; timestamps; and deployment identifiers. Store sensitive values according to their classification. A useful log can reference a protected payload rather than duplicating secrets into a broad analytics system.

This compact event shape gives responders something they can query and join:

{
  "trace_id": "tr_01J...",
  "agent_id": "billing-assistant",
  "tenant_id": "t_2048",
  "model": "provider/model-version",
  "prompt_version": "sha256:...",
  "memory_refs": ["mem_771:v4"],
  "tool_call": {"name": "issue_refund", "args_ref": "vault://events/991"},
  "authorization": {"principal": "agent-billing-prod", "decision": "allow", "policy": "refund-v7"},
  "approval": {"required": true, "actor": "user_18", "decision": "approved"},
  "result": {"status": "submitted", "external_id": "rf_8831"},
  "occurred_at": "2026-08-09T14:03:22Z"
}

The expected output is one event per meaningful transition, joined by trace_id, not one enormous transcript. Preserve raw prompts and responses in a restricted store when policy permits, and place hashes and references in the operational event stream. This separation lets responders establish sequence without giving every dashboard viewer access to customer content.

Treat prompt templates, tool schemas, policies, evaluation sets, and memory snapshots as versioned production artifacts. A responder should be able to answer which exact instructions and permissions applied to a run. A Git commit alone is insufficient when an administrator can change a hosted prompt, a vendor silently updates an alias, or memory persists across deployments.

Set retention from the longest credible detection delay and your legal duties, then test retrieval. Also synchronize clocks across the model gateway, application, identity provider, queues, and connected tools. A five-minute clock skew can reverse the apparent order of approval and action, which turns a straightforward investigation into an argument about causality.

Containment starts with identity and tools

Containment should remove the agent's ability to cause further harm while preserving the safest useful service. Killing the user interface is often insufficient. Queued jobs may continue, delegated agents may retain credentials, webhooks may retry, and connected tools may accept calls after the visible application goes offline.

Build a containment ladder before launch. The first rung disables a dangerous tool or changes it to read only. The next requires human approval for every side effect. Then pause a tenant, workflow, agent identity, model route, or all agent execution. The last rung revokes credentials and isolates connected systems. The commander chooses the narrowest rung that stops the credible harm, then widens it when uncertainty remains.

OWASP describes excessive agency as damaging action caused by excessive functionality, permissions, or autonomy. That distinction matters during response. If the agent misused a legitimate refund tool within its granted limit, rotating the credential will not fix the control failure. If an attacker stole the identity, changing prompts will not contain access. Responders must identify whether the unsafe capability came from the available function, the granted privilege, or the missing approval boundary.

Every production agent needs its own workload identity. Do not share a broad service account across agents or environments. Issue short-lived credentials, restrict them to named operations and resources, cap transaction values and call rates outside the model, and make revocation independent of a deployment. The model should request an action; deterministic code should authorize it.

Keep kill controls outside the agent's planning loop. An agent must not be able to re-enable its own tool, edit the policy that constrains it, suppress its audit events, or approve its own recovery. Test the controls under queue backlog, provider outage, partial network failure, and expired administrator sessions. A button that depends on the failing agent stack is theater.

After isolation, capture volatile state before deleting memory or clearing queues. Snapshot active plans, pending tool calls, retrieved context, and current policy versions. Quarantine suspected documents and memory records so new runs cannot consume them, but keep protected copies for analysis. Erasing the poisoned input too early may stop the symptom while destroying the only evidence of how it spread.

Separate data leaks from hallucination harm

Data exposure and harmful hallucination require different investigations, even when one output contains both. A leak asks what protected information left an authorized boundary and who received it. Hallucination harm asks which unsupported claim reached a decision or audience, what action it influenced, and how far that claim propagated. Combining them under “bad output” hides notification duties and downstream correction work.

For a suspected leak, stop further disclosure, preserve the exact input and output, identify data classification and subjects, trace every destination, revoke exposed secrets, and inspect retrieval and tool access. Determine whether the model provider retained the content under your agreement and whether plugins, logs, caches, analytics, or human reviewers received copies. Security and privacy leads should assess reporting deadlines from verified facts, not wait for perfect root cause.

A hallucination does not become harmless because the model displayed a disclaimer. Find where the claim went: a private draft, a support reply, a database field, generated code, a financial decision, or public content. Identify people and systems that relied on it, freeze automated propagation, and replace or retract the claim at every destination. If generated code shipped, examine its behavior and reachable data instead of treating the incident as a documentation error.

The investigation also needs provenance. Record the prompt, retrieved sources, tool results, model version, sampling settings, validation checks, and human review. Do not label every unsupported answer a “model issue.” Stale retrieval, a truncated tool response, contradictory instructions, a parser that dropped citations, or an approval screen that hid uncertainty can produce the same visible error. Root cause belongs at the control that should have prevented harm.

Do not ask the affected model to judge whether its own output was false or sensitive. Use authoritative records, deterministic scanners where appropriate, and qualified people. A second model may help sort a large corpus, but its verdict needs validation. Otherwise the response creates a new unmeasured AI dependency inside the incident.

Customer communication should name observed impact, affected period, actions taken, and any step the recipient must take. Avoid blaming “the AI,” which tells customers nothing about accountability. Your company selected the workflow and controls. Say what happened in terms of data or decisions, and correct false content in the same channel where people received it.

Use three runbooks with explicit decisions

A runbook earns its place when it converts an ambiguous alert into decisions. Write separate paths for uncontrolled agent action, suspected data disclosure, and harmful false output. Each path should name entry conditions, immediate containment, evidence to preserve, owners to page, decision points, recovery criteria, and notification inputs.

For uncontrolled action, the first decision is whether the agent still has a path to side effects. Disable the affected tool or identity, stop queued and delegated work, preserve active state, and list completed external actions. Then choose reversal, compensation, or acceptance for each action. Do not restore autonomy until deterministic authorization and approval boundaries prevent the same action, even if a new prompt appears to behave.

For suspected disclosure, contain every copy path before narrowing scope. Pause the response channel and retrieval source, protect logs, revoke any secret in the content, and identify recipients. Decide whether the data was merely processed inside an approved boundary or disclosed outside it. That distinction depends on contracts, configuration, and actual routing, not on a vendor's generic privacy page.

For harmful false output, stop publication and downstream automation, preserve the claim and its provenance, and appoint a subject-matter owner to establish truth. Search for repeated or derived versions using trace IDs and destination records. Decide which recipients need correction, which decisions need review, and whether earlier outputs from the same configuration require sampling. Recovery requires corrected information to reach the places where the false claim had influence.

Every runbook needs stop conditions. “Monitor the model” is not one. Good conditions are observable: the agent identity is disabled, the queue is drained into quarantine, the exposed token no longer authenticates, every known public copy is corrected, or a policy denies the reproduced unsafe tool call. Attach an owner and timestamp to each condition.

Include an uncertainty branch. If responders cannot tell whether data escaped, treat it as potentially disclosed while they preserve evidence and restrict access. If they cannot enumerate agent actions, isolate the identity and connected tools before attempting reconstruction. If they cannot establish truth quickly, suspend the affected advice rather than publishing a confident substitute.

Keep the runbook short enough to use under pressure and place deep diagnostic commands in linked internal procedures. Since this public article contains no links, the practical rule is simple: the first screen should tell the on-call person whom to page, what to stop, what not to destroy, and which decision comes next. A twenty-page policy that starts with definitions will be ignored at 2 a.m.

Recover through replay, not optimism

Recovery requires evidence that the harmful path is closed and that suspended work can resume safely. A prompt edit plus a few successful chats proves little. Replay the incident input and nearby variants against the proposed controls in an isolated environment, verify tool authorization separately, and compare every side effect with an expected record.

Set recovery criteria during planning. For an action incident, require a reproduced denial or required approval at the policy layer, reviewed pending actions, rotated credentials where exposure is possible, and reconciled external state. For a leak, require closed routes, known recipient scope, completed secret rotation, and approved notifications. For false output, require an authoritative correction, propagation review, and a validation control suited to the claim.

Restore in stages. Start with shadow execution that records intended actions without performing them. Move to a limited tenant or low-impact operation with mandatory approval. Increase scope only when monitoring shows expected decisions and no unexplained tool calls. Keep the containment controls ready; recovery is a controlled experiment, not a declaration that the model is fixed.

Manual fallback deserves engineering attention. NIST AI 600-1 explicitly recognizes that fallback may include manual processing. Document which work can wait, which work staff can perform without the agent, how to avoid duplicate actions when automation returns, and what capacity limit triggers a broader service suspension. A manual process that nobody has staffed is not a fallback.

After service returns, reconcile intent, agent output, and external reality. Check payments, messages, repository changes, tickets, database writes, and delegated jobs. Agents often produce partial success: one tool call commits while the next fails, then a retry repeats the first action. Recovery must find those mismatches instead of trusting the final conversational response.

Declare recovery only when the incident commander, technical owner, and affected business owner agree on evidence. Record residual risk and a deadline for any temporary control. Permanent “temporary” human approval is a warning that the workflow had too much autonomy for its design.

Rehearse the failure before production does

A plan works only after people use it against a plausible failure. Run a tabletop at least whenever the agent gains a powerful tool, a new data class, persistent memory, a new vendor, or a wider audience. Regular cadence matters, but material capability changes should trigger practice rather than wait for the calendar.

Use a scenario that crosses boundaries. A support agent retrieves a malicious instruction from a customer attachment, writes it into shared memory, sends fragments of another customer's record through an approved messaging tool, and queues follow-up actions. Inject incomplete evidence and a vendor outage. The exercise should force choices about containment scope, customer notice, memory quarantine, manual fallback, and who can accept downtime.

Measure actions, not attendance. Can the on-call engineer identify the agent identity and disable it? Can the commander find pending jobs and tool calls? Can privacy staff determine which data fields and recipients are involved? Can communications draft an accurate notice from the decision log? Can the team replay the path without touching production? Record the time and blockers, but do not invent a universal target that ignores your system's impact.

A blameless review should still name faulty decisions and controls. “The model hallucinated” is not a root cause. Ask why unsupported output could reach the destination, why the agent had that tool, why approval missed the risk, why logs lacked provenance, or why the fallback failed. Assign every correction to an owner with a due date and verify it in the next exercise.

Keep the plan proportional to the autonomy you deploy. A read-only internal assistant needs different paging than an agent that changes production infrastructure or contacts customers. If your team cannot describe and test the kill path, evidence chain, and manual fallback, reduce the agent's permissions until it can. Capability without response capacity is borrowed risk.

For founders who need an outside review, oleg.is offers a five-day Team & AI Audit focused on engineering operations and identifiable savings, which can include examining where agent controls and ownership break under real workflows. The useful outcome is not a polished policy. It is a team that can stop the right capability, preserve the right evidence, correct the harm, and prove the workflow is safe enough to resume.