AI Incident Response Playbook

Why AI incidents break the normal runbook

Ordinary IT incident response assumes you can reproduce the failure. A page returns a 500, you pull the stack trace, you find the line. AI systems do not cooperate with that method. Run the same prompt twice and you can get two different answers, so "I could not reproduce it" tells you almost nothing about whether the problem is fixed. The behavior depends on sampling, on whatever context got retrieved that time, on which tools happened to be available, and on a model version the vendor can change without telling you. Absence of a repeat is not evidence of a fix.

The second difference is logging. Most small-company AI deployments record the final output and nothing else. The system prompt in effect, the documents pulled into context, the tool calls the agent made and what they returned, the user identity behind the session: usually not captured at all, or captured into a vendor dashboard you have never opened. That is the exact material you need to answer whether customer data was exposed, and it is the material you are least likely to have.

Third, the vendor holds much of what is left, and the vendor does not work on your timeline. Your contract may or may not entitle you to detailed logs. Support tickets take days. Retention windows on hosted consoles are often short and are not designed around your legal obligations. Plan on the assumption that anything you do not capture in the first day may be gone by the time you know you need it.

Detection: the alert is usually a person

You will probably not get a page. The realistic detection channels are a support ticket, an employee saying that looked wrong, a customer screenshot, or an unexplained spike on a usage bill. That means the highest-value detection control at your size is social, not technical: one standing instruction that anything strange from an AI tool goes to one named person the same day, with no judgment call about whether it is serious. People sit on things they are not sure about, and the delay is what costs you the logs.

Once a report lands, spend ten minutes on triage before anything else. Which system misbehaved, is it still running, who was affected, and what can it reach. Then classify the impact into one of three buckets, because the rest of the response follows from it: data went somewhere it should not have, the system took an action in the real world, or the output was simply wrong. The third bucket still counts when the wrong output was advice a customer acted on. Also ask whether someone caused it on purpose. A successful prompt injection has a signature: content from outside your control, a web page, an uploaded file, an inbound email, entered the model's context and changed what it did next.

Containment: the smallest switch that actually stops it

Containment pulls in two directions. You want it stopped now, and you want the business to keep running. Resolve that by ranking your options ahead of time from least to most disruptive: revoke one credential, disable one integration, drop the agent from write access to read-only, insert a human approval step, take the feature offline entirely. Reach for the smallest switch that reliably stops the behavior, not the first one you think of. If you have never written this list down for your AI tools, that is a twenty minute task worth doing this week.

For an agent with write access to a real system, the reliable switch is usually the credential, not the application. Revoke the token in the downstream system, the CRM, the database, the cloud account, rather than trusting a toggle in a product you did not build. Apps have caches, retries, and queued jobs. Then confirm the stop rather than assuming it: watch the target system for a few minutes and check for scheduled or queued actions the agent already submitted that have not fired yet.

When the incident is an exposed API key, revoke before you rotate and assume everything that key could reach was reachable. Then answer two questions. What did the key do during the window it was exposed, which you can often reconstruct from the downstream provider's own usage logs, and where else does that key live. Keys copied into an AI tool have a habit of also sitting in a CI job, a second SaaS integration, and someone's local environment file.

Evidence expires, and the vendor holds most of it

Capture before you fix. The list is specific: the full conversation or session with user messages and timestamps, the session or trace IDs, the model name and version, the system prompt and settings in effect at the time, any documents or records retrieved into context, every tool call with its arguments and result, and the account the AI system was acting as. If you also know which customer or user was on the other end, record that, because it is the difference between telling one person and telling everyone.

Then stop the clock on deletion. Turn off automatic cleanup where the product allows it. Tell staff not to delete threads, tickets, or emails connected to the incident, which people do reflexively when they feel responsible. Export whatever you can see into a file stored outside the affected system, and screenshot anything that will not export. Open a vendor support ticket that asks in writing for logs covering the affected window to be preserved. A preservation request on day one is a very different conversation from the same request on day twenty.

Keep one running timeline document from the moment the report arrives. What you observed and when, what you changed and when, who you told and what you said. It feels like overhead while you are busy, and it is the single artifact your counsel, your insurer, and any enterprise customer will ask for. Reconstructing it from memory two weeks later is where otherwise honest accounts fall apart.

Deciding what to disclose, then closing the loop

The notification question is not one you should answer alone. In general terms, duties to notify attach to unauthorized access to or disclosure of personal data, but the triggers, the definitions, the recipients, and the deadlines differ by jurisdiction, by industry, and by what you signed with your business customers. Some of those clocks are short and they can start before you finish investigating. The failure mode for small companies is quiet reasoning toward "probably fine." Get counsel in early with the facts you have, and let the ambiguity be their problem rather than yours.

Order matters more than speed. The owner or decision maker first, then counsel and your cyber insurer, since policies commonly require prompt notice and late notice can affect coverage. Then contractual notification to business customers, which is frequently stricter and faster than any statute. Then affected individuals and regulators on counsel's advice, then any public statement. Tell the vendor in parallel if their product contributed, because they may have obligations of their own and their logs are still ticking. Whatever you send, keep it factual: what happened, what you know and do not yet know, what you have done, what the reader should do. Do not speculate about cause mid-investigation and do not call it isolated until you have checked whether it was.

Hold the review within about two weeks, while the evidence is still on hand and the memory is fresh. Keep it on the system rather than the person, because a team that gets blamed reports the next one late. Almost every AI incident review lands on the same two findings: you could not see what the system did, and the system could do more than the job required. That gives you your work. Add the logging you wished you had, with a retention period you chose on purpose. Cut the permissions back to the minimum. Then turn the incident itself into a test you run after every prompt, model, or configuration change, because with a nondeterministic system the only proof you have fixed something is that you keep checking.