Every other production system tells you when it is wrong
A payments service throws an error. A queue backs up. A page fails to load and someone complains within the minute. Software mostly announces its own failures, which is why on-call rotations work. A model is different in one specific and inconvenient way: its output is a claim about the world, and nothing in the system knows whether the claim was true. It will keep producing well-formed, plausible, fast, confident answers indefinitely after it has stopped being right.
That single property is the whole subject. Everything in this article is an attempt to buy a signal that would otherwise not exist.
Four layers, and only the first is solved
Is it running? Latency, error rate, throughput, queue depth, cost. Your ordinary infrastructure tooling handles this and you probably already have it. It catches outages and catches nothing else.
Is the input what it was? Schema changes, null rates, ranges, category counts, volume by source. This is where most real incidents start, and it is the cheapest layer to build.
Is the output plausible? Score distributions, class balance, the rate at which the model abstains or refuses, output length, how often a human overrides it. These are proxies. They do not prove correctness, but they move before anyone notices a problem.
Is it right? Requires knowing the truth. Which is where it gets expensive.
Teams build the first layer, buy a tool that promises the third, and never seriously fund the second or fourth. The second is cheap and catches most incidents. The fourth is the only one that answers the question anyone actually asked.
Most model degradation is not the model
When a system that had been working starts producing bad answers, the cause is usually upstream and human. Somebody renamed a field. A partner started sending amounts in a different currency without saying so. A source system began populating a column that used to be blank, so a feature that meant “unknown” now means something else. A backfill rewrote three months of history overnight. A default value quietly changed from null to zero, and the model treats zero as a real measurement.
The model consumes all of this obediently, because that is what it does. Nothing errors. The numbers move.
The practical version: for each important input, track null rate, distinct-value count, minimum and maximum, and volume, daily, per source. Alert on step changes rather than gradual movement. Write down the expected shape of each field as an explicit agreement with whoever produces it, so a violation is a breach of something stated rather than a surprise. This is a few days of work and it will catch more real problems than any statistical drift package you buy.
The related failure is a mismatch between how a value is computed at training time and at serving time — a missing feature filled with the training median in one place and zero in the other. The only reliable check is to score the same records through both paths and compare, periodically, as a test.
The labels arrive late, or not at all
Whether you can measure accuracy at all depends on when the truth shows up, and that delay is the single most important design constraint in the whole system.
| Situation | When truth arrives | What to do about it |
|---|---|---|
| A user clicks or does not click | Minutes | Measure directly; this is the lucky case |
| A payment is disputed | Weeks to a few months | Report on cohorts by decision date; never compare an immature month to a mature one |
| A customer churns | The prediction horizon itself | Watch leading indicators; accept that scoring quality lags by that horizon |
| A risk decision plays out | A year or more | Sampled human review is the only timely signal |
| Extraction or classification with no outcome | Never | You must buy labels: sample and adjudicate weekly |
| Anything you declined to act on | Never, by construction | Approve a small random slice anyway, or you can only ever see half the picture |
That last row deserves its own paragraph, because it is the most commonly missed idea in production machine learning. If the model rejects an application and you therefore never learn what would have happened, your future training data contains only the cases you approved. The model then gets confidently worse in a region of the input space it can no longer see. The standard remedy is to let a small random fraction through against the model's advice, accept the cost of doing so, and treat that cost as the price of continuing to know anything. It has to be a deliberate decision by someone senior, because it means knowingly approving some bad ones.
Buy the labels you do not get for free
Where outcomes never arrive, the highest-value monitoring investment is almost always the least technical one: a person looking at a random sample of the model's decisions on a fixed schedule.
Concretely, one to three hundred sampled decisions a week, adjudicated by someone who knows the domain, at two or three minutes each. That is roughly five to fifteen hours a week of somebody's time. It is real money and it is far less than one bad quarter. It gives you a weekly accuracy estimate with an honest confidence interval, a growing labelled dataset for retraining, and — the part people undervalue — a human who actually knows what the system is doing.
Sample randomly, and oversample the uncertain cases separately, keeping the two sets apart so the random sample stays an unbiased estimate. Measure the reviewers against each other occasionally too: if two experienced people disagree on fifteen percent of items, the model was never going to beat that.
You are probably here because
- A model has been live for a year and nobody can say how it is doing
- Someone found a mistake the system had been making for months
- You are paying for a monitoring tool and it has never told you anything actionable
- The team that built it has moved on and you are not sure who owns it now
The label-delay section is the constraint. The section on drift alerts is why your current alerts are muted. The section on ownership is the one that actually decides whether any of this survives.
Drift is not decay, and treating it as such destroys your alerting
Input distributions move constantly. Marketing runs a campaign and the traffic mix changes. A large customer onboards. Seasonality arrives on time. A price changes. Every one of those shifts a distribution, and almost none of them makes the model worse.
Statistical drift measures compare a recent window against a reference and produce a number. The number is real and the interpretation is not automatic. Alert on it directly and you will send several notifications a week that mean nothing, and within about a month every one of them will be muted, filtered or ignored — including the one that mattered. This is the most reliable way to end up with monitoring that exists and protects nothing.
Three rules keep drift useful. Only monitor drift on inputs that the model actually depends on heavily, not all four hundred columns. Pair every drift signal with a performance question, so the alert reads “this feature moved and here is what happened to the outcome rate” rather than a bare statistic. And route drift to a weekly review, not to a pager — nobody is retraining a model at three in the morning, so it is not a page.
Aggregate numbers hide the failure
Overall accuracy is stable. One segment has collapsed. This is the shape of most real degradations, because the world changes in one place at a time: a new region, a new product line, a new acquisition channel, one large customer whose data looks nothing like everyone else's.
Pick five to ten slices that matter commercially and report every metric within each of them. Region, channel, product, customer size, new versus returning, and any single customer big enough to matter on their own. Then look at the smallest slices with real suspicion, because they move first and are the easiest to dismiss as noise.
Monitoring a language-model feature
When the output is text, there is no accuracy number and most of the standard machinery does not apply. What works is a mix of cheap automatic signals and a small amount of expensive human reading.
Automatically: refusal and empty-response rates, output length distribution, latency at the ninety-fifth percentile, retry rate, tool or function call failure rate, rate of responses failing a structural check such as valid formatting or required fields, and cost per request. Any of these moving sharply usually means a change worth investigating, including a change on the provider's side that you were not told about.
By hand: a fixed evaluation set of fifty to two hundred real cases with known good answers, run on every prompt change and on a schedule regardless, plus a weekly read of a random sample of live outputs by someone who knows what good looks like. Automatic scoring by another model is useful for triage and for catching large regressions; it is not a substitute for the human read, because it shares blind spots with the thing it is grading.
The thing that changes under you here is the provider: versions get deprecated, defaults shift, behaviour moves with no change on your side. Pin versions where you can, keep the evaluation set runnable on demand, and treat a provider release note as a reason to run it.
How often each signal catches a real problem before a customer does — our read
Our judgment from production work, not a study. The bottom row is what most monitoring products lead with; the top two are what most teams have not built.
An alert nobody can act on is not an alert
Before any alert exists, answer three questions in writing. Who receives it. What can they do within the hour. And what happens if they do nothing.
If the honest answer to the second is “nothing until morning,” it is a report, not a page. Most model monitoring belongs in a weekly review with a named owner and a short written record. Reserve paging for the small set of conditions that are both urgent and reversible: the model is returning errors, the input feed has stopped, the output distribution has moved so far that something is clearly broken, or a hard business limit has been breached.
And apply one discipline ruthlessly: an alert that has fired more than twice without anyone taking action gets retuned or deleted. Alerts nobody acts on are not free. They train the team to ignore the channel.
Rollback is the feature that makes the rest safe
Detection only matters if you can act. Being able to return to the previous model in minutes, without a deployment and without a meeting, is worth more than any additional metric.
That means every prediction is recorded with the model version that produced it, the previous version stays deployable, and switching is a configuration change rather than a release. It also means a new model runs in shadow for a period before it takes over — scoring the same live traffic, its outputs recorded and compared, affecting nothing. A week of shadow scoring costs a little inference money and catches most bad promotions before a customer sees one.
The same discipline applies to prompts and to feature logic. A prompt change is a model change. If it is not versioned and reversible, you have an undocumented deployment path into production.
What it costs, honestly
Building a workable monitoring setup for a single production model — input checks, output distributions by segment, a versioned prediction log, a weekly report, and a rollback path — is typically two to four weeks of engineering. The infrastructure to run it usually lands in the range of five to fifteen percent of what serving the model costs.
The human review loop is the larger line and the one that gets cut first. Five to fifteen hours a week of a knowledgeable person, indefinitely. When someone proposes removing it because the model has been fine for six months, that is precisely the argument that precedes finding out otherwise.
An unowned model decays in about a quarter
The most common failure is not technical. A model ships, the people who built it move to the next project, and the alerts route to a shared channel that belongs to nobody in particular. Six months later the numbers are worse and there is no record of anyone having looked.
Name a person. Not a team, a person, with the review on their calendar and the report going to somebody who will notice its absence. Ten minutes of a weekly meeting where a human says “here is what the model did last week and here is anything unusual” catches more than an elaborate dashboard with no reader. When we are asked to review a system that has quietly degraded, the finding is almost always this one.
When a spreadsheet is enough
Not every model needs this. If the volume is low, the decisions are reversible, a human sees every output anyway, and being wrong costs an apology rather than money, a monthly check by the person who owns the process is proportionate. Building a monitoring platform around such a model is a way of spending real money to protect against a small loss, and it is the kind of over-engineering that makes the next machine learning proposal harder to fund.
Scale the monitoring to what being wrong costs. Write that number down first, and let it choose the investment.
How monitoring goes wrong
- Watching the model and not the inputs, when the inputs are what changed
- Paging on drift, which trains everyone to mute the channel within a month
- Only aggregate metrics, so a collapsed segment hides behind a flat average
- No sampled human review in a system where outcomes never arrive on their own
- Training only on decisions you acted on, and losing sight of everything you declined
- Predictions logged without a model version, making any later investigation guesswork
- Prompt changes with no versioning, which is an unlogged deployment path
- No named owner, after which everything above stops happening within two quarters
Before you call a model production-ready
- The cost of a wrong answer is written down, and the monitoring is scaled to it
- Every prediction is logged with inputs, output, model version and timestamp
- Input checks cover schema, null rate, ranges, cardinality and volume by source
- Metrics are reported by five to ten business segments, not only in aggregate
- The label delay is known, stated, and reflected in how results are reported
- A sampled human review runs weekly, with a named reviewer
- Training and serving computation of features is compared periodically
- The previous model version stays deployable and rollback is a configuration change
- Every alert has an owner, an action, and a stated consequence of inaction
- One named person is responsible for this model after launch
Bottom line
Watch the inputs, because that is where most incidents begin. Buy the truth you do not get for free, through a small weekly human review, and treat that as infrastructure rather than an expense to trim. Report by segment, because averages hide exactly the failure you care about. Keep drift out of the pager. Make rollback a configuration change. And put one person's name on the model, because everything else in this article depends on somebody looking, and a system nobody owns will degrade quietly for a quarter before anybody notices.
Frequently asked questions
For one or two models, the useful parts — a prediction log, input checks, segmented metrics, a weekly report — are a couple of weeks of work on tooling you already run. Buy a platform when you have enough models that a shared way of doing this saves real time, or when you need the retention and access controls that come with it. A purchased platform does not remove the need for a sampled human review, which is the part that actually tells you whether the model is right.
On evidence, not on a calendar. Scheduled retraining without a performance signal is a ritual: it can quietly make things worse, because a model trained on recent data that includes a period of degraded behaviour learns the degradation. Retrain when measured performance drops, when the business changes in a way you can name, or when you have accumulated enough new labelled data to expect a real improvement — and validate every candidate against a held-out set before it goes anywhere near live traffic.
Input stability first, because that catches most incidents regardless of labels. Then proxies: how often a human overrides the model, the rate at which its outputs get edited before use, complaint volume, and the shape of the score distribution by segment. Then a weekly sampled review to get a timely accuracy estimate that does not wait three months. Report matured cohorts separately and never compare an immature month against a mature one.
With cheap automatic signals for change detection — refusal rate, empty responses, length, latency, structural validity, cost — plus a fixed evaluation set of real cases run on every change, plus a weekly human read of a random sample. Automatic grading by another model helps with triage and large regressions. It is not a substitute for the human read, because it shares failure modes with the system it is grading.
One named person, with the weekly review on their calendar and the authority to roll back without convening a meeting. It does not have to be the person who built it, and often should not be. What matters is that a specific human reads the report each week and that somebody notices if it stops arriving. Ownership assigned to a team rather than a person is the most common reason monitoring exists on paper and not in practice.