AI incident response planning: the plan you write before the model fails

An AI incident is not a security incident, and the difference matters on day one

Most engineering organizations already have an incident response plan. It was written for a cybersecurity event: an intrusion, a ransomware detonation, a credential leak. It assumes an adversary, it assumes the system was working correctly until someone attacked it, and it assumes containment means cutting off access. None of those assumptions hold when a model starts being wrong. There is usually no adversary. The system is working exactly as built. And cutting off access is often the last thing anyone wants to do, because the model is now load-bearing in a workflow that has no manual path back.

That gap is the reason so many teams discover they have no AI incident plan at the worst possible moment: after a customer forwards a screenshot. The security plan runs, the security team correctly determines that no breach occurred, the ticket closes, and nobody has answered the actual question, which is whether the model should still be making decisions right now.

The federal framework community has been explicit that this is a separate discipline. NIST published the AI Risk Management Framework 1.0 (NIST AI 100-1) on January 26, 2023, and two of its MANAGE subcategories are entirely about the after. MANAGE 4.1 requires that post-deployment monitoring plans include "appeal and override, decommissioning, incident response, recovery, and change management." MANAGE 4.3 goes further: "Incidents and errors are communicated to relevant AI Actors including affected communities. Processes for tracking, responding to, and recovering from incidents and errors are followed and documented." That is a written plan requirement, not a monitoring requirement, and it is the one most teams skip.

On the security side, the reference document changed shape too. NIST SP 800-61 Revision 3, published in April 2025, retired the twelve-year-old four-phase lifecycle from Revision 2 and rebuilt incident response around the six Cybersecurity Framework 2.0 Functions: Govern, Identify, Protect, Detect, Respond, Recover. Its stated reason is worth reading carefully by anyone planning for AI failures. In 2012, the publication says, incidents were rare, narrow, and usually resolved within a day or two, which made it realistic to treat incident response as a separate activity performed by a separate team. That is no longer true, so response now lives inside ordinary risk management rather than beside it. Model failures behave the same way. They are frequent, they are diffuse, and they do not resolve in an afternoon.

The four ways a deployed model actually fails

Planning starts with an honest failure taxonomy, because each of these four needs a different first move and only one of them is what most people picture.

Silent degradation. Nothing breaks. Accuracy slides because the world moved, the upstream data changed shape, or a vendor quietly updated a model behind an API. Nobody notices for weeks. This is the failure your drift monitoring is supposed to catch, and it is the failure that most drift channels have been trained out of reporting by their own false-alarm rate.

Loud wrongness. A single output is confidently, visibly, embarrassingly incorrect in front of a person who matters. The aggregate metrics are fine. There is no statistical anomaly to find, because one bad answer out of fifty thousand does not move a mean. The severity comes entirely from what that one answer touched.

Operational failure. Latency spikes, the inference cluster saturates, a rate limit trips, a vendor has an outage, or the token bill triples overnight. These look like ordinary availability incidents and your existing runbook mostly handles them, with one AI-shaped tail: when the model is unavailable, does the product return an error, or does it silently return something worse?

Boundary failure. The model saw data it should not have seen, or emitted data it should not have emitted. Training data surfaced in an output. A retrieval index pulled a document the requesting user had no right to. A prompt log captured controlled information into a system that is not accredited to hold it. This is the one class that crosses back into security and privacy, and it is the class that starts contractual clocks running.

Grade the incident by consequence, never by metric

The single most useful artifact in an AI incident plan is a severity ladder, and the ladder has to be defined by what happened to the person on the other end, not by how far a number moved. A four-point AUC drop on a low-stakes recommendation is not an emergency. A perfectly calibrated model applied to a population it was never validated on is an emergency, even though every metric on the dashboard is green.

Two rules make this ladder work. First, anyone can raise a grade and only the accountable owner can lower one, which removes the incentive to under-report at three in the morning. Second, the grade is assigned from the consequence column alone. Teams that grade from the metric column end up with a SEV-4 sitting on a decision that has already gone out the door.

The rollback path is the plan

Everything above is preparation for a single question, and it is a question about authority rather than engineering.

If the on-call engineer has to wake a director to get permission to fall back, the fallback will not happen for two hours, and those two hours are the whole incident. Standing authority to degrade the service has to be written down, delegated to the person holding the pager, and stated in a way that removes career risk from using it. That authority is worth more than any monitoring investment on the list.

Authority is useless without somewhere to fall back to, and this is where most plans are hollow. A deployed model needs a defined fallback tier chosen before launch, in this order of preference: the previous model version, still deployed and warm; a deterministic rule-based path that is worse but predictable; a human review queue with the staffing to absorb the volume; or the feature turned off with an honest message to the user. Every deployed model should have one of those four named in writing, along with how long the system can survive on it.

The failure mode is picking a tier nobody has tested. A previous version that is not warm takes forty minutes to bring up. A human queue sized for ten cases a day collapses under four hundred. A rule-based path written eighteen months ago references a schema that no longer exists. Rolling back once, deliberately, during a quiet week, converts a theoretical tier into a real one, and it is the cheapest insurance in the entire plan.

NIST states the requirement directly. AI RMF MANAGE 2.4 reads: "Mechanisms are in place and applied, and responsibilities are assigned and understood, to supersede, disengage, or deactivate AI systems that demonstrate performance or outcomes inconsistent with intended use." The Generative AI Profile that NIST published in July 2024 as NIST AI 600-1 turns that into concrete actions, including establishing and regularly reviewing the specific criteria that warrant deactivation, and defining an escalation path to the organizational risk authority when those criteria are met. The point of writing the criteria down in advance is that nobody has to argue about them while the system is misbehaving.

The first twenty-four hours

Sequence matters more than speed. The most common self-inflicted wound in an AI incident is a fast, well-intentioned fix that destroys the ability to explain what happened.

Pin the evidence before you fix anything

Defense contractors already live under a version of this discipline. DFARS 252.204-7012 requires a contractor to preserve and protect images of all known affected information systems and all relevant monitoring and packet-capture data for at least 90 days from submission of the cyber incident report, so that the government has the chance to request the media. The reason that clause exists is that remediation destroys evidence, and remediation is what a good engineer does first by instinct.

An AI incident has its own evidence set, and it is more fragile than a disk image because most of it is not written down anywhere by default. Before anything is redeployed, capture: the exact model artifact and version hash that served the bad output; the full input, including the system prompt and any retrieved context, not just the user's visible text; the raw output before post-processing; the feature values as computed at serve time rather than as recomputed later; the configuration, including temperature, thresholds, routing rules, and any feature flags; and the upstream data snapshot the model was reading.

The recomputation trap deserves emphasis. If your feature pipeline recalculates values from a warehouse rather than logging what was actually passed to the model, then an hour after the incident the inputs you can reconstruct are not the inputs the model saw. Investigations die there. So do the conversations with a customer who wants to know exactly what happened, because the honest answer becomes "we cannot fully reconstruct it," and that sentence costs more trust than the original failure did.

Hot-swapping the model to stop the bleeding is correct. Hot-swapping without pinning the artifact first means the thing that misbehaved no longer exists.

What actually obliges you, and when

Buyers ask whether an AI failure has to be reported to anyone. The answer depends on who the customer is and what the model touched, and the honest version is that some of these instruments are binding today while others are frameworks a contract can pull in by reference.

Three things about that table matter more than the rows themselves. First, M-25-21 rescinded and replaced M-24-10, so a governance plan written against the older memo is aimed at a document that no longer governs. It defines high-impact AI as AI whose output "serves as a principal basis for decisions or actions that have a legal, material, binding, or significant effect on rights or safety" — a definition written around consequence, exactly like the severity ladder above. Its 365-day documentation window ran from April 3, 2025, which means agencies are now inside the reporting regime rather than approaching it.

Second, the memo builds in an escape valve worth knowing about if you sell to an agency: a Chief AI Officer may waive one or more minimum practices for a specific application after a written, system-specific determination, must recertify each waiver annually, and must report the scope, justification, and supporting evidence to OMB within 30 days of granting or revoking one. Waivers are also publicly summarized. If a program office tells a vendor a practice does not apply, there should be a waiver behind that statement.

Third, the civilian-side picture for contractor reporting of security incidents is still moving, and anyone planning against it should track the final rules rather than the proposals. Our companion piece on contractor cyber incident reporting covers where that stands and what triggers the DoD clock in practice.

Detection is not the hard part, and it is not where incidents come from

Teams spend the monitoring budget on statistical detectors and then discover that the majority of real incidents arrive through a human being. A user files a ticket. An analyst mentions something odd in a standup. A customer's program lead forwards a screenshot to your account manager. None of those paths runs through the drift dashboard.

So the plan needs a report channel that a non-engineer can find in under a minute, and an escalation rule that gets a human report in front of an on-call engineer without passing through three queues. NIST AI 600-1 makes the same point in operational terms, calling for verification that the people monitoring reported issues can actually evaluate system performance and promptly escalate. The word doing the work there is promptly. A report channel with a four-day median response is a report channel that stops receiving reports.

The statistical detectors still matter, for the silent-degradation class specifically. They just need to be built so somebody still reads them in month six, which is a design problem with known answers and a subject of its own.

Communication: order, audience, and restraint

Once an incident is graded, three audiences need different things at different speeds. The affected user needs to know their case is under review and what happens next, in plain language and without technical detail. The customer's program lead needs the scope, the containment status, and an honest statement of what is not yet known. The contracting officer enters the conversation only when a clause requires it, and when that happens the report goes to the channel the clause names, not to a relationship contact.

Restraint is a discipline. The strongest early communication states what is known, what is contained, and when the next update comes, and says nothing about cause. Root cause stated in hour two is wrong often enough that it becomes its own second incident. The correction is more expensive than the silence would have been.

There is also a public layer that mature AI programs use and immature ones ignore. NIST AI 600-1 suggests establishing organizational roles and procedures for communicating incidents to downstream stakeholders through community resources, naming the AI Incident Database, AVID, CVE, NVD, and the OECD AI Incidents Monitor. Both AI-specific registries are live. The AI Incident Database is operated by the Responsible AI Collaborative to index harms and near-harms realized in the real world by deployed AI systems, with incident records now numbered past 1,600. The OECD AI Incidents Monitor tracks incidents and hazards across location, industry, harm type, and affected stakeholders. Reading them before you deploy is a cheap way to find failure modes your own tabletop would never have invented.

The post-incident review that produces a change, not a document

SP 800-61r3 makes a shift here that transfers cleanly to AI work: lessons learned during response should often be shared as soon as they are identified, rather than held until recovery concludes, and improvement is treated as a continuous activity feeding every function rather than a phase that closes the loop once. In practice that means the review starts while the incident is open and its output is a merged change with an owner and a date, not a document that circulates and dies.

The four questions that produce useful reviews are narrow. What did the system do, precisely, in terms of inputs and outputs? What was the earliest signal that existed, and why did it not reach a human faster? What did we have to guess at because it was not logged? And what would have made the fallback one step shorter? That last question generates most of the durable improvements, because the fallback path is the part of the system that only ever gets exercised under stress.

Worth stating plainly: the point of the review is not to find who broke it. Models degrade for reasons nobody chose. A review that produces blame produces underreporting, and underreporting is how a SEV-3 becomes a SEV-1 three weeks later.

Rehearse it, because a plan nobody has run is a document

NIST AI 600-1 asks organizations to rehearse third-party incident response plans at a regular cadence and improve them from what the rehearsal shows. A two-hour tabletop once a quarter is enough, and the scenarios that expose the most are the awkward ones. A vendor silently updates the model behind an API and your outputs shift with no deploy on your side. A retrieval system returns a document to a user who should not have been able to see it. A model that has been correct for eight months is applied to a new population nobody revalidated it against. Your fallback tier is invoked and the human queue receives ten times the volume it was sized for.

Run each one against real names and a real clock. The finding is almost never technical. It is that nobody knew who to call, or that the person with the authority to fall back was on a plane, or that the log the investigation needed was set to seven-day retention.

The plan on one page

• A named accountable owner per deployed model, with a named backup, reachable outside business hours
• A severity ladder graded on consequence to the person on the other end, with a rule that anyone may raise a grade
• Standing authority for the on-call engineer to fall back without asking permission, written down and stated without career risk
• A defined fallback tier for every model: previous version warm, deterministic path, human queue, or off
• A rollback executed once in production during a quiet week, so the tier is real rather than theoretical
• An evidence-pinning step that runs before remediation, capturing artifact, inputs, retrieved context, raw outputs, and config
• Serve-time feature logging rather than warehouse recomputation, with retention longer than your longest plausible investigation
• A notification list of names, not roles, annotated with which clock each name starts
• A quarterly tabletop against a scenario the team has not seen, run against a real clock
• A review whose output is a merged change with an owner and a date

If you are buying this rather than building it

Three questions separate a vendor who has operated a model from one who has only shipped one. Ask what their fallback tier is for the specific system being proposed, and whether it has been exercised in production. Ask what gets captured at serve time and how long it is retained, since that answer tells you whether an investigation is possible at all. Ask who on their side can turn the model off without escalating, and what the criteria are.

Vague answers to those three are the reliable signal. A team that has lived through a model failure answers all three in about ninety seconds, because they wrote the answers down after the last one.

Frequently asked questions