Everything is green and the system is wrong
A conventional outage announces itself. Requests fail, a queue backs up, a dashboard turns red, and somebody's phone goes off inside ninety seconds. A model failure does none of that. The service returns 200. Latency sits inside budget. The error rate is zero, because from the runtime's point of view nothing went wrong: a request arrived, a prediction was produced, a response was written. The prediction was just wrong, and it was wrong for a class of inputs nobody was sampling.
That is why the first report almost never comes from monitoring. It comes from a support ticket, an account manager forwarding a screenshot, or a finance number that moved and nobody can explain. Time to detect gets measured in days rather than minutes, and not because the team is careless. Every threshold in a normal alert stack is a statement about availability, and availability is exactly the property that stayed fine.
The second structural difference is that there is no single artifact to roll back. In an ordinary outage the last deploy is the prime suspect and reverting it is a bounded action with a known duration. In a model system the surface that can change without any deploy of yours includes the provider's model behind a floating version alias, the retrieval corpus (which changes whenever someone edits a document), the feature pipeline upstream, the input distribution as real users change what they type, a prompt edited in a configuration store, and a downstream tool whose response schema shifted. Six moving parts, and five of them leave no entry in your deploy log.
The third difference is the one that turns a bad afternoon into a bad quarter. Damage from a model incident is retroactive and persistent. When a checkout service is down for twenty minutes, the damage stops when it comes back. A classifier that was wrong for twenty minutes has already routed the tickets, sent the replies, set the prices, approved the refunds, and written the fields. Recovery is not restoring service. It is finding those decisions and undoing them, which is a data problem with a much longer tail.
You are probably here because
- You found out from a support ticket or a screenshot somebody forwarded, and every dashboard was green the whole time it was happening.
- The output changed and no deploy went out, so there is no obvious thing to roll back and no agreement on what actually moved.
- Somebody asked what fraction of outputs were wrong and how many customers were affected, and there was no way to answer either question.
- You are still finding decisions the system made during a window that closed weeks ago.
The detection layer and the output ledger below are what address these, and all four usually share one root cause: everything being collected describes the request, and nothing describes the output.
The six things that actually changed
When we get pulled into a live model incident, the investigation almost always starts in the wrong place. Everyone looks at the model first, because the model is the interesting part. It is rarely the thing that moved.
First Cause of a Model Incident — Our Ranking by Frequency
Our ordering of first causes across model incidents we have been called into. The bottom row is the one everybody investigates first.
Upstream shape changes. A column that used to be a string is now an object. A currency field started arriving in cents. An identifier gained a prefix. Nothing in the pipeline errors, because the pipeline is permissive by design, so the value simply reaches the model as an unfamiliar token and the model does what it does with unfamiliar tokens: it guesses fluently.
Input distribution moves. A marketing campaign brought in a different customer segment. A new integration started sending machine-generated text where humans used to write. A product change made a rare question common. The model did not get worse. It is being asked something it was never good at, and the mix shifted enough for that to show.
Retrieval drift. This one is uniquely nasty because the corpus is edited by people outside engineering. A policy document gets rewritten, a duplicate gets uploaded, a re-index runs with a different chunk size, and answers that were grounded yesterday are grounded in something else today. The model is unchanged. Its evidence is not.
Provider changes. If production calls a model through an alias that tracks the newest version, behavior can change without any action on your side. Formatting, refusal patterns, verbosity, and how strictly a tool schema is followed are all fair game.
Severity when the failure is a rate, not a state
Standard severity ladders are written for binary outcomes. Checkout is down. The database is unreachable. A model defect has no such state, so severity has to be assembled from four things: reversibility of the action taken, autonomy of the system that took it, exposure in volume and visibility, and the rate at which the defect occurs relative to baseline.
Rate is the one people skip, and skipping it is what makes model incidents drag. A team that cannot say whether the defect affects two percent of outputs or forty percent will argue about containment for an hour, because both the person who wants to kill the feature and the person who wants to leave it running are reasoning from the same absence of evidence.
| Level | What puts it here | Response |
|---|---|---|
| SEV1 | The system acts without a human in the loop, the action is expensive to reverse, and the defect rate is materially above baseline for an identified input class | Containment lever pulled within the hour. Reversal workstream opened in parallel with root cause, not after it. |
| SEV2 | A person reviews before the action, but volume is high enough that review is nominal, or the action is reversible but has already reached customers | Raise the confidence floor, route the affected class to real review, contain within the shift. |
| SEV3 | Output is customer-visible but advisory. A person reads it and decides. No automatic action follows. | Fix in the normal cycle. Add the failing cases to the evaluation set as a regression test. |
| SEV4 | Internal-only output, or a quality regression with no decision attached to it | Ticket it. Track whether it moves. |
| Unscoped | The rate is unknown and cannot be estimated yet | Treat as one level above your best guess until a sample has been labeled. Downgrade on evidence, never on optimism. |
The detection layer has to exist before you need it
None of this can be assembled during the incident. The signals below are cheap to build and nearly all of them are label-free, which matters because ground-truth labels for a production model usually arrive days later, if at all. During an incident you do not have labels. You have proxies, and the good ones move early.
Human override rate. The single best model monitor most teams already collect and almost nobody alerts on. The proportion of model outputs that a person edited, rejected, re-routed, or escalated. It requires no labeling effort, it reflects the judgment of people who know the domain, and it moves before anything else does. If your interface has an edit button, you have this metric whether or not you are reading it.
Refusal and fallback rate. Empty responses, schema-validation failures, null extractions, "I don't have enough information", fallback to the default branch. A jump here is almost never noise. It usually means an input changed shape or a retrieval path went dry.
Output shape distribution. Response length, token count, JSON validity, per-field fill rate, and the fraction of requests that took the tool path rather than answering directly. These distributions are stable in a healthy system and they shift well before accuracy does. A sudden drop in mean output length is a behavior change, and it is visible within minutes.
Retrieval health on canaries. Keep a fixed set of queries whose correct source documents you have written down, and check on a schedule that those documents are still being retrieved in the top results. When the canaries stop hitting, retrieval broke, and no amount of prompt work is going to repair it.
Downstream acceptance. Did the routed ticket get re-routed by a person. Did the drafted reply get sent or discarded. Did the flagged transaction get cleared. This is a delayed label, but it is the closest thing to ground truth you get for free, and it is the metric that best predicts whether anyone outside engineering is about to notice.
The version identifier is the cheapest field in the system and the most valuable during an incident
Record the exact model identifier, the prompt or template version, the embedding model version, and the retrieval index version on every single request, alongside the request id. It is four short strings. Without them, "did the provider change something" is an afternoon of guesswork. With them, it is one query. The same fields make the difference between a postmortem that can be read six months later and one that cannot.
A canary set is a monitor, not a CI gate
Most teams build an evaluation set, wire it into CI, and run it on deploy. That catches your changes and misses everyone else's, which is most of them. The same suite run on a schedule against the production configuration is a monitor, and it is the only one that measures the thing you actually care about. Hourly if the system acts without a human. Daily otherwise. Alert on the delta against the last known-good run rather than on an absolute threshold.
Sample size decides what you are able to see, and this is worth doing the arithmetic on before you trust a number. On a 200-example set where the model normally scores around 90 percent, the standard error of the measured score is about 2.1 points, so consecutive runs will bounce by roughly four points in either direction for no reason at all. A 200-example canary detects a five-point regression. It will never detect a one-point one, and if you alert on small movements you will be paging on noise inside a week. Seeing small regressions takes either thousands of examples or a paired comparison that runs both configurations over the identical inputs, which removes most of the variance because the two runs share the same hard cases.
Stratify the set by input class and score each stratum separately. An aggregate that drops two points can mean the model got slightly worse everywhere, or that it became useless for one segment representing five percent of traffic. The second is an incident and the first probably is not, and an aggregate number cannot tell them apart.
The first fifteen minutes
The instinct in a model incident is to start fixing, because the fix feels close: change the prompt, adjust a threshold, try the newer model. Every one of those actions destroys evidence you are about to need. The order below front-loads capture and scoping, and it delays root cause on purpose.
Model Incident — First Response
Step three is the one teams skip and the one that decides how the rest of the incident goes. Fifty examples pulled at random from the window and labeled by two people takes about fifteen minutes. If the observed defect rate is around ten percent, fifty examples puts roughly an eight-point band around it, which is enough to pick a severity level and not enough to put in a customer email. That is the correct trade. You need a decision now and a publishable number later, and the sample can be extended while containment runs.
Pull the sample randomly from the whole window, not from the complaints. The complaints are the reason you are here and they are the most biased sample available. A rate estimated from reported failures is not a rate.
Containment, ranked by cost
Containment for a model system is a dial rather than a switch, which is an advantage if the dial was built and a liability if the only available action is turning the feature off entirely. Build at least two of these before you need them.
| Lever | What it costs | When it is the right call |
|---|---|---|
| Kill switch to the deterministic path | Full loss of the feature. Users notice immediately. | SEV1, or any case where the action is hard to reverse and you cannot yet bound the rate |
| Roll back to a pinned prior version | Minutes, if the pin exists. Impossible if it does not. | The change is yours and the previous version is known good on the canary set |
| Raise the confidence floor | Coverage drops. More requests fall through to a human or a default. | The defect concentrates in low-confidence outputs, which you have checked rather than assumed |
| Route the affected class to review | Queue depth and staffing. Works for hours, not days. | The failing input class is identifiable by a rule you can write |
| Disable one tool or one route | Narrow capability loss, usually invisible to most users | The defect is traceable to one integration, one schema, or one document set |
| Freeze the retrieval index at the last good snapshot | Answers go stale, gradually | Retrieval canaries broke and the corpus changed inside the window |
The rollback lever deserves a specific warning. It only exists if a rollback target exists, and calling a provider through an alias that always resolves to the newest version means you have no target. Pin production to a dated version identifier, keep the previous one warm, and verify once a quarter that switching between them is a configuration change rather than a code change. A rollback path that has never been exercised is a plan, not a lever.
Send it over and we will tell you what we would change.
Email the fields you currently log on every model call — one real log line with the values scrubbed is enough — plus the evaluation set you run against it, to contact@precisionfederal.com. You get back a short written note naming the three things we would change and why. One business day. No charge, no meeting, no deck.
contact@precisionfederal.comWhat goes wrong inside the incident
- Editing the prompt live to see if it helps, which destroys the ability to reproduce the failure ten minutes later.
- Changing two things at once, so whichever one worked cannot be identified and both ship permanently.
- Judging by three hand-typed examples in a playground that does not share the production prompt, retrieval context, or sampling settings.
- Rolling forward to a newer provider model under time pressure, trading a known defect for an unmeasured one.
- Deleting bad outputs without recording them, which erases the evidence and the reversal list in the same operation.
- Declaring resolution because the dashboard came back, with no sampled measurement to confirm it.
- Defining the affected window as "since we noticed" instead of searching backward for when the signal actually turned.
- Letting the person who wrote the prompt be the only one who reads the failing outputs.
The part nobody plans for: undoing decisions already made
This is where model incidents get expensive, and it is decided entirely by whether one table exists. Call it the output ledger. One row per decision the system made, written synchronously with the decision itself, carrying: request id, timestamp, a reference or hash of the input, the model identifier and version, the prompt or template version, the retrieval document ids used, the raw output, the score or confidence, the downstream action taken, and who or what took it.
With that table, scoping the blast radius is a single statement: every decision between T1 and T2 produced by version X against document set Y where the score fell below Z. You get a list of affected records, the customers attached to them, and a reversal plan that can be worked through and checked off. Without it you have two options and both are bad. Notify everyone in the window, which is expensive and destroys trust in proportion to how wide you cast. Or notify nobody and hope the ones that mattered surface on their own.
Where the Hours Go in a Model Incident
Proportions of total incident time in the pattern we see. Reversal looks small here because it collapses when the ledger exists and expands without bound when it does not.
Two practical notes on the ledger. Keep it under the same retention and access controls as the data it references, because it is a copy of that data by another name. Where inputs cannot be stored, store a hash and a pointer, which is enough to enumerate and match without duplicating the sensitive content. If the inputs contain personal data, the ledger is in scope for deletion requests under GDPR, and an ISO 27001 or SOC 2 auditor will treat it as evidence rather than as a debugging convenience. Write the retention period down before somebody asks you what it is.
When the model is not yours
A large share of production systems now depend on a model somebody else operates, which means part of your incident surface sits outside your change control. That is workable, but it takes deliberate handling rather than trust.
Pin the version. Production calls a dated version identifier, never a floating alias. Development can track the newest one. The gap between them is where you find out about behavior changes on your schedule instead of theirs.
Shadow the next version. Run the candidate version against the canary set on the same inputs and diff the outputs before migrating. Paired comparison on identical inputs is far more sensitive than comparing two separate scores, and it surfaces format and refusal changes that an accuracy number hides entirely.
Put the deprecation calendar somewhere a person will see it. Provider version retirements are announced. They are announced in changelogs and emails that route to an inbox nobody reads. Migration deadlines belong on the same board as the rest of the roadmap.
Keep a second provider wired at the interface, even if you never route to it. The value is not a better price. It is having somewhere to go during an incident that is not "wait for the status page to update."
Negotiate the terms that matter. Version pinning with a stated support window, written notice ahead of deprecation, a behavior changelog rather than only a feature changelog, and a support path with a response commitment. These are more negotiable than most buyers assume, and they are worth more than a discount when something goes wrong.
The postmortem is a different document
Model incident postmortems fail in a specific way: the action items are unfalsifiable. "Improve the prompt." "Add better guardrails." "Monitor quality more closely." Six months later nobody can say whether any of it was done, and the same failure recurs with a different surface.
Three classes of action item are the ones worth writing. First, new examples in the canary set drawn from the actual failing inputs, with agreed correct answers. That is the regression test, and it is the only artifact that prevents a repeat of this exact defect. Second, a new monitor, almost always output-side, that would have caught it earlier than a human did. Third, a containment lever that did not exist and had to be improvised.
Record the version identifiers in the writeup, including the ones that turned out to be irrelevant. Six months on, a postmortem that says "the model started producing shorter answers" is unreadable, and one that names the version and the date is a citation. And keep it blameless in the specific way this domain requires: there is a strong pull toward blaming the model, which is the same mistake as blaming a new hire. The model produced what the surrounding system asked it to produce, under the inputs the surrounding system handed it.
Readiness — Contribution to Shorter Time-to-Mitigate
Ranked by how much each one shortens an incident, not by how hard it is to build. The top two are the cheapest of the six.
The pre-incident kit
Everything above reduces to a build list. None of it is exotic and most of it takes days rather than quarters, but all of it has to exist before the first bad output, because none of it can be created while people are arguing about whether to turn the feature off.
- An output ledger keyed to request id, carrying model version, prompt version, retrieval document ids, score, and the downstream action
- A canary evaluation set of several hundred examples, stratified by input class, versioned in your repository next to the code
- Scheduled evaluation runs against the production configuration, alerting on the delta rather than an absolute threshold
- Output-side monitors: human override rate, refusal rate, schema-validation failure, output length distribution, retrieval canary hit rate
- At least two containment levers per system, each tested once outside an incident
- Pinned model and embedding versions, with the previous version warm and switchable by configuration
- A severity rubric written down before it is needed, including what to do when the rate is unknown
- A reversal query that has been run once against real data, so you know it returns what you think it returns
- One rehearsal a quarter: replay a past failure against the current system and time the response
The rehearsal is the item most often cut and the one that finds the most. Replaying a known failure surfaces the gaps that documentation hides: the dashboard nobody has permission to see, the kill switch that requires a deploy, the ledger query that times out on a real window, the person who turns out to be the only one who knows how to pin a version. Every one of those is cheap to fix on a Tuesday afternoon and expensive to discover at 11pm.
Bottom line
Model incidents are not harder than ordinary outages. They are differently shaped, and the standard toolkit is aimed at the wrong property. Availability monitoring will not find them, the last deploy is usually innocent, severity has to be built out of a rate rather than a state, and recovery includes undoing work that already shipped. Build the output ledger, run the canary set as a monitor rather than a gate, pin the versions, and give the on-call engineer a lever they are allowed to pull without waking anyone. Those four things convert most model incidents from a multi-day scramble into an afternoon with a clean writeup at the end of it.
Frequently asked questions
Because every alert in a normal stack measures availability, and availability is unaffected. The service returns successful responses at normal latency while producing wrong answers. Detection requires output-side signals such as human override rate, refusal rate, output shape distribution, and retrieval canary hit rate, none of which exist unless somebody built them on purpose.
Combine four factors: how reversible the resulting action is, how autonomously the system acts, how much volume and visibility is exposed, and the defect rate relative to baseline. Estimate the rate by hand-labeling a random sample from the affected window rather than from the complaints. Fifty examples takes about fifteen minutes and is enough to choose a severity level.
Around 200 examples at a 90 percent baseline gives a standard error near 2.1 points, so runs bounce about four points on their own and only a five-point regression is reliably visible. Catching smaller movements takes thousands of examples, or a paired comparison that runs both configurations over identical inputs, which removes most of the shared noise.
Request id, timestamp, an input reference or hash, the model identifier and version, the prompt or template version, the embedding and index versions, retrieval document ids, the raw output, the score, and the downstream action taken. That set answers "did something change" in one query and makes the blast radius enumerable rather than guessed at.
Pin production to a dated version identifier rather than a floating alias, keep the previous version switchable by configuration, shadow-run candidate versions against your canary set on identical inputs before migrating, and put deprecation dates on the roadmap. In the contract, ask for version pinning with a stated support window, written deprecation notice, and a behavior changelog.