Degradation is silent by construction
When a database goes down, everything stops and everyone knows within a minute. When a model gets worse, throughput is unchanged, latency is unchanged, the error rate is zero, and the dashboard is green. The system produces the same volume of confident output it always did, and some larger fraction of it is wrong. There is no exception to catch, because from the software's point of view nothing failed. This is the single most important property of an AI system in production, and it is the reason the discovery usually arrives as a customer complaint rather than an alert.
The second most important property is that degradation is usually not the model's fault. Four distinct mechanisms produce the same symptom, they call for four different responses, and the most common one is a plain engineering bug that has nothing to do with machine learning. Teams that jump straight to retraining spend weeks and money on the wrong layer, and sometimes make things worse.
The world moved. The relationship the model learned changed. Fraud patterns adapt because there are people on the other side who are paid to adapt them. A competitor changes pricing and your demand model is now describing a market that no longer exists. A regulation changes what a category means. Nothing in your system is broken; the thing it was a model of is different.
The inputs moved. The relationship holds, but what arrives no longer looks like the training data. A supplier changed the format of a feed. A form got redesigned and a free-text field became a dropdown. The business expanded into a new region, or a new customer segment now supplies a third of the volume. A scanner was replaced and every document is 200 DPI instead of 300.
The pipeline broke quietly. This is the most common and the least discussed. A join that used to match now drops eight percent of rows. A null started arriving as a zero, and zero is a perfectly valid value for that field. A timestamp column changed timezone. An upstream team renamed a column and added a compatibility view that silently returns stale values. None of this is an ML problem, and all of it looks exactly like one.
The model itself changed. If you call a hosted model, its version can move underneath you. If someone “improved” a prompt on Friday, behavior changed on Friday. A library upgrade altered tokenization or default parameters. Your code is identical, your git history shows nothing, and the outputs are different.
You are probably here because
- The team that uses the system says it “used to be better” and cannot say when it changed
- Nothing in your monitoring has ever alerted, and you are no longer sure that is good news
- Someone is asking whether to retrain, and nobody can say what that would fix
- You are writing a support agreement and do not know what to require after acceptance
The table below matches symptoms to mechanisms. The section on proxies is what to watch when you have no labels, which is most of the time. The contract section is the part buyers control.
Matching the symptom to the mechanism
| What you notice | Most likely mechanism | How you would confirm it |
|---|---|---|
| A sudden step change on a specific date | Pipeline break, or the model version moved | Diff the input schema and the deployment log for that day. Almost always found in an hour |
| A slow slide over months | The world moved | Score an old holdout with the current model. If it still scores well, the model is fine and reality changed |
| Wrong on one segment, fine elsewhere | Inputs moved for that segment | Break every metric down by source, region, customer and document type |
| More blanks, nulls or abstentions | Upstream format change | Compare field-fill rates week over week. This is the cheapest check that exists |
| Outputs feel different in style, not accuracy | Prompt or hosted model version change | Replay a fixed set of inputs against the pinned prior version |
| Confidently wrong on cases it used to get right | Any of the four — start with the pipeline | Rerun the exact historical inputs end to end and compare to the stored outputs |
The last row is the reason to store inputs and outputs together with a version stamp from day one. Replay is the single most valuable diagnostic capability an AI system can have, it costs almost nothing to build in advance, and it is nearly impossible to reconstruct after the fact.
Why you find out from a customer
Here is the structural problem, and it is worth stating plainly to anyone who signs off on this kind of system. Accuracy is not observable in production. You have predictions. You do not have truth. Truth arrives later, if it arrives at all, and the gap is a property of your business rather than your engineering.
For a card fraud decision, the chargeback confirms the answer in weeks. For a collections propensity model, the outcome lands in one to three months. For a churn model with an annual contract, you learn how good last January's prediction was next January. For a document extraction system, you may never learn, because nobody re-reads the document once a number has been entered.
Write that number down for your system, because everything else follows from it. If truth arrives in three days, a modest monitoring setup is fine and you will catch problems fast. If it arrives in nine months, no amount of accuracy monitoring will help, and you need proxies plus deliberate human review of a sample — sampling is not a nice-to-have there, it is the only ground truth you will get in time to act.
What to watch when you cannot measure accuracy
All six of the following are computable from data you already have, without a single label. None of them measures accuracy. Together they catch most degradation weeks before an outcome-based metric would.
Human override rate. The best single signal in the list, and the most underused. If a person reviews, edits, corrects or overrules the model's output as part of the normal workflow, that rate is a free, continuous, human-generated quality measure. It rises before anyone can articulate why. It costs nothing to log and it is the first thing we ask for.
Input distribution. Field-fill rates, value ranges, category frequencies, document length, source mix. Cheap, sensitive, and it catches the whole class of quiet pipeline breaks.
Prediction distribution. If a classifier that flagged four percent of cases starts flagging eleven, something changed — the model, the inputs, or the world. You do not know which yet, and you do know to look.
Abstention and refusal rate. How often the system declines to answer, returns low confidence, or falls back. Rising abstention is often the earliest honest signal a system produces about itself.
Downstream behavior. Time spent on a review screen. How often a user rejects the first suggestion. How often a case is escalated. Users vote continuously; almost nobody counts the votes.
Cost and latency per unit of work. Not a quality metric, but a leading indicator: retries, longer inputs and fallback paths all show up here first and all mean something changed.
Signal value against cost to instrument — where we start
Our ordering, not a measurement. The bottom row is what vendors demonstrate first and what generates the most ignored alerts.
That last row deserves an explanation, because drift dashboards are the standard product demo. Run a distribution test daily on a few hundred rows and it will fire constantly on differences too small to matter — formal tests are exquisitely sensitive to sample size, and a business has weekday and weekend and month-end shapes that are perfectly normal. Alerts that fire weekly for no reason are worse than no alerts, because within two months everyone has learned to close them without reading. Alert on effect sizes large enough to matter, require the condition to persist several days, and route it to a named person rather than a channel.
The order of investigation, and why retraining comes last
When quality drops, there is a temptation to retrain, because retraining feels like doing something and it is the one action clearly labeled as ML work. It is the right move in maybe one case in four, and in some situations it makes the problem worse. Work in this order.
First, the pipeline. Compare the input data of last week to the same week a month ago, field by field. Look for schema changes, fill-rate shifts, new categorical values, duplicated or dropped rows. Find nothing and you have spent an afternoon. Find something and you have your answer, and it was never a model problem.
Second, the deployment history. Every model version, prompt revision, library upgrade and configuration change with a date. Line them up against the date the quality changed. A step change on a Tuesday and a deploy on that Tuesday is not a coincidence to be investigated further; it is the answer.
Third, the holdout. Score the current model against a labeled dataset from before the problem. If it still performs, the model is unchanged and the input or the world moved. If it does not, the model or its serving path changed. This single test splits the remaining possibilities cleanly and it takes an hour if the holdout exists — which is the argument for keeping one.
Fourth, segment everything. Overall metrics hide the truth. Degradation is usually concentrated: one document type, one region, one customer who changed their export settings.
Then, and only then, consider retraining. And check one thing before you do: whether your recent labels are contaminated by the model's own decisions. If the model declines a case and you therefore never observe the outcome, retraining on what you observed teaches the model that its past decisions were right. This feedback loop is subtle, it is common in credit, fraud, hiring and triage systems, and it produces a model that looks better on every metric you can compute while getting worse at the job.
Two weeks from “it feels worse” to a decision
Step four is where teams flinch, because it means paying someone to read a few hundred cases and mark them right or wrong. It is unglamorous and it is the step that ends the argument. Everything upstream of it is inference about a system nobody has actually measured.
The part the buyer controls: what the contract says about month nine
Most AI statements of work define quality at acceptance and go quiet afterward. The vendor demonstrates a number on a test set, everyone signs, and the document has nothing to say about the system's behavior a year later. Then quality drops, and the conversation about whose problem it is starts from nothing.
Define the metric and the measurement procedure, not just the number. “Ninety-two percent accuracy” is unenforceable. Ninety-two percent on which population, measured how, by whom, at what sample size, with what confidence interval? A metric without a procedure is a marketing claim.
Keep a holdout the vendor never sees. Hold back a labeled sample from the start, and add to it over time. It is your only independent instrument, and it is the thing that lets you evaluate a replacement vendor later on equal terms.
Say who investigates, and inside what window. Separate investigation from remediation. Investigation should be covered in the support agreement with a response time. Remediation may reasonably be scoped and priced when the cause is known, and the cause determines who pays — a vendor defect is theirs, and a supplier changing your data format is not.
Agree a retraining cadence and its price in advance. Quarterly, annually, or triggered by a defined threshold. Include who supplies the labels, because labeling is the real cost and it is usually the buyer's to carry.
Require version pinning and change notice. If the system calls a hosted model, require a pinned version and written notice before it moves. Require the same for prompt changes. A prompt edit is a behavioral release even though it touches no schema, and it should go through review like one.
Get the monitoring as a deliverable. Dashboards, thresholds, alert routing and the replay store, in your accounts, running on your infrastructure. Monitoring that lives inside the vendor's platform disappears with the vendor.
Monitoring is a small share of the build. Labels are the real expense.
Instrumenting a system properly — logging inputs and outputs with version stamps, distribution checks, a replay store, alerting on a handful of proxies — typically lands somewhere between five and fifteen percent of the initial build effort when it is done alongside the build. Bolted on afterward it costs more, because the logging was never designed in. Retraining compute is usually the cheapest line in the whole exercise, often hours of machine time. The expense that surprises people is human labeling, which is a per-item cost that recurs every time you need to know how the system is actually doing. Budget for it explicitly rather than discovering it during an incident.
The mistakes we are called in to fix
- No stored inputs and outputs, so nobody can tell when the change happened, only that it did
- Retraining as the first response, on labels contaminated by the model's own past decisions
- A drift dashboard nobody reads, because it alerted every week from the day it was installed
- A moving hosted model version, with no pin and no change notice
- Prompt edits shipped outside code review, so a behavior change has no record
- Accuracy defined at acceptance and never again, with no procedure attached to the number
- Aggregate metrics only, hiding a failure concentrated in one segment
- Monitoring inside the vendor's platform, which ends when the contract ends
Before you call an AI system done
- Inputs and outputs are stored together with model and prompt version stamps
- A replay set runs against every model, prompt and library change, with a published diff
- Human override rate is logged and charted from day one
- You know how long it takes for truth to arrive, and have written the number down
- A labeled holdout exists that the vendor has never seen
- A weekly human review quota is funded, however small
- Alerts require a meaningful effect size and multi-day persistence, and reach a named person
- Hosted model versions are pinned, with written notice required before a move
- Every metric can be broken down by segment, source and customer
- The support agreement names who investigates, how fast, and who pays for what
Bottom line
Every model degrades. That is not a defect and not a sign of a bad vendor; it is what happens when a fixed artifact meets a world that moves. The failure worth avoiding is not degradation, it is degrading invisibly for six months while confident output flows into decisions. The defenses are unglamorous and cheap if you ask for them at the start: store what went in and what came out, watch the humans who correct the machine, keep a holdout the vendor never sees, pin every version, and write down who investigates when the number drops. None of that requires a data science team. It requires the questions to be asked before the system is accepted, because after acceptance every one of them becomes a change order.
Frequently asked questions
There is no general answer, and any vendor who gives you one without knowing your domain is guessing. It depends entirely on how fast the underlying process changes. Adversarial settings such as fraud and abuse move in weeks because people are actively working against the model. Physical processes such as equipment failure can be stable for years. The useful question is not how fast it decays but how long it would take you to notice, which is a property of your instrumentation rather than your model.
You monitor proxies and you sample. The proxies that earn their keep are human override rate, input field-fill and schema checks, prediction and abstention distributions, and downstream user behavior. None measures accuracy; together they catch most real degradation early. Then fund a small, regular human review of randomly sampled cases — a few hundred a month is often enough, and it is the only true measurement in the whole setup.
A cadence is a reasonable default when fresh labels arrive naturally, and a poor one when they do not. Retraining is not free of risk: it can encode a feedback loop if your labels come only from cases the model let through, and it can quietly change behavior on segments nobody checked. Verify the pipeline, the deployment history and a holdout first. Retrain when you have evidence about which layer moved.
Whatever the contract says, which is often nothing. Write it explicitly: the metric and its measurement procedure, an investigation obligation with a response window, and a rule that separates causes. A defect in delivered work is the vendor's to fix. A supplier changing your input format, or the market changing, is a maintenance activity someone has to pay for, and agreeing the rate before it happens is much easier than after.
Log every input and output together with the model and prompt version that produced it, and keep them. It costs storage and an afternoon. It makes replay possible, which turns “it feels worse” into a dated, measurable comparison, and it is essentially impossible to reconstruct once the window has passed.