What model risk actually is
Model risk is the loss that comes from acting on a model output that was wrong, plus the loss that comes from using a correct model outside the conditions it was built for. The second half is the one that gets skipped, and it is where most real incidents live. A pricing model that works beautifully on the segment it was fitted to, quietly extended to a new product line by a team that did not know the boundary existed, has not malfunctioned. It has been used wrongly, and the framework exists to make that use visible before it costs something.
The vocabulary most teams use here — inventory, tiering, independent validation, conceptual soundness, ongoing monitoring — came out of banking supervision after the last financial crisis and has since spread into insurance, asset management, healthcare and anywhere a model output is hard to reverse. It is a good framework. It assumes a model is a fixed artifact, documented once, revalidated annually, changed rarely. Machine learning breaks that assumption in three specific ways, and the failure of most model risk programmes is not that they are too heavy. It is that they are heavy in the places where the risk is not.
You are probably here because
- Nobody can produce a list of the models running in production without asking around for a week
- Validation takes four months and the model retrains every three weeks
- A vendor changed something and your outputs moved, and you found out from a customer
- The documentation is thorough and no one has ever read it after sign-off
The first is an inventory and definition problem. The second is a change-control design problem. The third is a contract and pinning problem. The fourth is the one worth caring about most, and the last section is about it.
You cannot manage what nobody has listed
Every programme starts with an inventory and nearly every inventory is wrong in the same direction: it lists the things that were built by people who call themselves data scientists, and misses the things that make decisions. A threshold in a SQL view that routes cases to review is a model. A vendor score dropped into an underwriting rule is a model. A spreadsheet with a regression in it that someone runs monthly is a model, and it is usually the highest-risk item on the list because it has no version control, no owner and no test.
Use a functional definition and write it down: anything that takes input data, applies a method with assumptions, and produces an output used to make or support a decision. Technique is irrelevant. A hand-tuned rule set with fifteen conditions carries assumptions and produces decisions, and it deserves a row. Excluding it because it is not machine learning is how a firm ends up with a complete inventory of the low-risk half of its exposure.
Then hold the inventory to two standards that most fail. It must be discoverable — populated partly by scanning deployment configuration and job schedulers rather than entirely by self-declaration, because self-declaration always undercounts. And every row must name a person, not a team. A team does not answer email.
Tier by consequence, never by technique
The single most common design error is tiering by sophistication: neural networks get heavy scrutiny, linear models get light. Sophistication is not risk. A logistic regression that declines credit applications is far higher risk than a gradient-boosted model that ranks internal search results, and the framework should say so plainly.
Tier on the consequence of being wrong, using four dimensions that can be assessed in a fifteen-minute conversation: materiality — what value or how many decisions flow through it; reversibility — can a wrong output be undone the same day, or has money moved and a person been told; autonomy — does a human review each output, sample them, or none at all; and exposure — does the output reach a customer or a third party, or stay inside the building. The highest tier is where materiality is large, reversibility is low and autonomy is full. That combination, not the algorithm, is what should trigger the heaviest process.
| Tier | Typical profile | Validation depth | Revalidation trigger |
|---|---|---|---|
| 1 — Critical | Automated decisions reaching customers, hard to reverse, material value | Full independent validation: conceptual soundness, replication, outcome analysis, challenger | Annually, plus any material change, plus a monitoring breach |
| 2 — Significant | Material but human-reviewed, or automated and easily reversed | Independent review of design, data and testing; replication on a sample | Annually or on material change |
| 3 — Moderate | Internal decision support, one of several inputs | Peer review inside the line, documented; monitoring required | On material change |
| 4 — Low | Exploratory, internal, no decision depends on it alone | Registered in the inventory with an owner and a stated limitation | Registration refreshed annually |
One rule keeps the tiering honest: the tier is set by the risk function, not by the model owner. Self-tiering drifts downward every time, not from bad faith but because the person closest to a model reliably underestimates how far its output travels.
What independent validation can and cannot establish
Independent validation asks three questions, and knowing which is which prevents most of the wasted effort in the discipline.
Is it conceptually sound? Does the method fit the problem, are the assumptions stated and defensible, are the features available at the time the decision is made — the single most common fatal defect, and one that no amount of accuracy testing will surface, because a leaked feature makes the test scores look excellent. Is the training population the population it will run on?
Does it do what the documentation says? Replicate it. Independently rebuild the outputs from the documented method and data, and compare. This is the step most often reduced to reading a report, and it is the step that finds the discrepancies: preprocessing that differs between training and serving, a filter applied in one path and not the other, a version of a reference table that no longer exists.
Is it working now? Outcome analysis against realized results, performance by segment rather than in aggregate, stability over time, and a comparison to a simple challenger. The challenger is undervalued: fitting a small transparent model on the same data and comparing tells you what the complexity is buying. Sometimes it is buying a lot. Sometimes the answer is a few tenths of a point of a metric nobody uses, purchased with a permanent explainability problem, and that is worth knowing before deployment rather than after.
What validation cannot do is establish that the model will keep working, or that it will not be used somewhere it does not belong. Those are monitoring and control problems, and a programme that treats a validation sign-off as the end of the risk is exactly the programme that gets surprised.
Where validation effort should go on a machine learning model — our default weights
Weights sum to 100. Our starting allocation, not a measurement. The algorithm review is where reports are thickest and where defects are rarest.
The three places machine learning breaks the classical framework
The model is not the artifact. A trained file is inert. What makes a decision is a system: feature computation, joins to reference tables, a preprocessing path, the weights, a threshold, and post-processing. Validating the weights alone validates the smallest and most stable part. In practice more defects live in the feature pipeline than in the estimator, and the highest-value single test in this whole discipline is comparing features computed in training against features computed in serving, on the same entities, and looking at the distribution of the differences. It is a day of work and it finds things.
Retraining is a change, and it is invisible. A model that retrains monthly changes its behaviour monthly with no code change, no release note and no ticket. If revalidation is annual, the artifact validated last spring stopped existing shortly afterwards. The answer is not to validate every retrain, which is unaffordable. It is to validate the training process as a controlled system — pinned data window, frozen feature definitions, an automated acceptance gate on a held-out set, an automatic block on distribution shift beyond a threshold, and a diff report against the incumbent — and then to treat any retrain that passes the gate as an approved instance of a validated process. Any retrain that fails the gate goes to a person. That is the design that makes the cadence affordable and the control real.
The dependencies change under you. An upstream table gains a category, a source system changes a default, a data provider quietly alters a field's definition. Nothing in the model changed and the model now behaves differently. This is the change class with no owner in most organizations, and it is why data lineage and model risk cannot be run as separate programmes staffed by people who do not talk.
The limitations page is the most valuable document you will write
One page, plain language, no method detail: what this model is for, what population it was fitted on, the conditions under which its outputs should not be used, the known weak segments with numbers, and what a user should do when an input falls outside the range. It gets read; the fifty-page validation report does not. It is also the artifact that turns "used outside its boundary" from an argument into a check, because a boundary written down is a boundary a monitor can test.
Send us one model and we will tell you what a validator would find.
Email the documentation for a single model, the feature list, and how it is retrained, to contact@precisionfederal.com. You get back a written note naming the leakage risks, the replication gaps and the change-control holes we would flag first. Two business days. No charge, no meeting, and no report you have to read twice.
contact@precisionfederal.comChange control that fits the cadence
Every change-control policy has to answer one question: what is material? Answer it with a written list rather than case by case, because case by case means the answer is decided by whoever is in a hurry.
| Change | Usually material? | Minimum control |
|---|---|---|
| Retrain, same data source, same features | No, if the gate passes | Automated acceptance gate, diff report, logged |
| New or removed feature | Yes | Targeted revalidation of data and outcome sections |
| Decision threshold moved | Yes, always | Impact analysis by segment before it ships; this is a policy change, not a technical one |
| Upstream schema or definition change | Yes, and usually undetected | Contract test on the input; alert to the model owner by name |
| Library or runtime upgrade | Sometimes | Re-scored regression set; differences above tolerance investigated |
| New population or use case | Yes, always | Fresh validation; this is a new model wearing an old one's name |
The last row deserves emphasis because it is the most frequent real incident and the one least likely to be caught by any technical control. Extending a validated model to a population it was not fitted on is the model risk event that looks like a business decision and is documented nowhere.
Third-party and hosted models
You cannot validate what you cannot inspect, and the honest position is to say so and then do the four things that remain available.
Test on your own data. A held-out set from your population, scored through the vendor's interface, is a genuine outcome analysis and does not require access to anything internal. It is also the only evidence that transfers: a vendor's published benchmark was measured on a different population, and yours is the one that matters.
Pin the version and keep a golden set. A hosted model that can change without notice is an uncontrolled dependency. Pin a version where the vendor offers one, run a fixed set of inputs on a schedule, and alert on output drift. When a vendor changes something silently, this is how you learn about it from a monitor rather than from a customer.
Put notice in the contract. Advance notice of material behavioural changes, a deprecation window measured in months, and the right to test. These are negotiable more often than people assume, particularly before signature.
Wrap it with controls you do own. Input validation, output range checks, business rules that override implausible results, and a documented fallback for when the service is unavailable or clearly wrong. The vendor's model is a component; the system around it is yours, and it is where your actual control lives.
Generative models add one wrinkle that matters: the output space is open, so a held-out accuracy figure is not available in the usual form. What works instead is a fixed evaluation set with graded rubrics, run on every version change, plus measurement of the failure modes you can define crisply — unsupported claims, missing citations, refusals, format violations. It is less satisfying than a single number and it is the honest instrument.
The mistakes we are called in to fix
- Tiering by algorithm, so a spreadsheet regression driving pricing sits below a recommender
- An inventory built only from self-declaration, missing every rule-based decision system
- Validation of the weights and none of the feature pipeline, where the defects live
- Annual revalidation of a model that retrains monthly, validating an artifact that no longer exists
- No definition of material change, so materiality is decided by whoever is under deadline
- A hosted model with no pinned version and no golden set, changing silently
- Limitations recorded in a report nobody reads instead of a page users are given
- A validation function that has never blocked a deployment, which tells you what it is
Standing the programme up
First 90 days, in the order that works
Step four before step five is the ordering people find counterintuitive and it is the one that determines whether the programme survives its first year. Validate first and you validate artifacts that the next retrain replaces. Build the control on the process first and each validation keeps its value.
The test of whether any of this is real
Model risk management fails in a distinctive way. It does not collapse; it becomes documentation. Reports get written, sign-offs get collected, an annual cycle runs, and nothing that happens changes what ships. The vocabulary is all present and the function has no effect.
There are three questions that separate the real thing from the performance, and they can be answered in an afternoon. Has validation ever blocked or materially changed a deployment? If the answer is never, it is a review function, not a control. Has a monitoring threshold ever fired and caused a named person to do something specific? If alerts go to a dashboard and nowhere else, monitoring is decoration. Can the model owner state the model's limitations from memory? If it takes a document search, users do not know them either.
The cost of the whole programme is not the reports. It is the friction it puts in the path of shipping, and that friction is only justified where the consequence of being wrong is high. Which is why tiering — the least technical part of this — is the decision that determines whether the programme is worth its cost.
Before you call the programme working
- A written functional definition of a model that includes rules and spreadsheets
- An inventory populated partly by scanning, not only by self-declaration
- A named individual owner on every row, never a team
- Tiers set by consequence, assigned by the risk function
- A written material-change list, agreed before the first argument about one
- An automated retraining gate that can block, with a diff report against the incumbent
- Training-versus-serving feature comparison run and its results recorded
- A one-page limitations sheet per model, given to users rather than filed
- Every hosted model pinned, with a golden set and scheduled drift checks
- Every monitoring threshold mapped to a named person and a specific action
Bottom line
Most of the classical framework transfers intact, and the parts that do not are identifiable in advance. Validate the system rather than the artifact, because the feature pipeline carries more defects than the estimator. Control the training process rather than each trained instance, or the cadence makes the control theatre. Treat upstream data changes as model changes, because behaviour moves whether or not your code did. And tier on consequence, so the weight lands where being wrong is expensive. Do those four and the rest of the programme is paperwork in the good sense: a record of decisions someone actually made.
Frequently asked questions
Anything that takes input data, applies a method with assumptions, and produces an output used to make or support a decision. Technique is irrelevant — a fifteen-condition rule set and a monthly spreadsheet regression both qualify, and the spreadsheet is often the highest-risk item on the list because it has no version control, no test and no owner. Defining models as "things built by data scientists" produces a complete inventory of the low-risk half of your exposure.
Validate the training process as a controlled system rather than each trained instance. That means a pinned data window, frozen feature definitions, an automated acceptance gate on a held-out set, an automatic block on distribution shift beyond a threshold, and a diff report against the incumbent. A retrain that passes the gate is an approved instance of a validated process; one that fails goes to a person. Validating every retrain is unaffordable, and validating annually means validating an artifact that no longer exists.
Not in the full sense, and saying so plainly is better than pretending. What remains available is substantial: outcome testing on your own held-out population, a pinned version with a golden set and scheduled drift checks, contractual notice of behavioural changes, and control logic you own around the call — input validation, output range checks, override rules and a documented fallback. The vendor supplies a component; the system around it is yours.
By consequence, on four dimensions: materiality of what flows through it, reversibility of a wrong output, how autonomously it acts, and whether the output reaches anyone outside the firm. Never by algorithm — sophistication is not risk. And the tier should be assigned by the risk function rather than the model owner, because the person closest to a model consistently underestimates how far its output travels.
Three questions. Has validation ever blocked or materially changed a deployment? Has a monitoring threshold ever fired and caused a named person to take a specific action? Can a model owner state their model's limitations without looking them up? Three noes means the vocabulary is present and the control is not, which is the characteristic failure of this discipline — it does not collapse, it turns into documentation.