The question that starts this
Somebody outside your company asked what your model does and who checked it. A customer's procurement team sent a 90-question AI questionnaire. A carrier asked about algorithmic decisioning before renewing your errors-and-omissions policy. A federal prime said their flowdown now includes NIST AI RMF alignment and they need an answer in two weeks. An acquirer's diligence team asked for validation reports on the models inside the product they are paying a multiple for. In every version of this the models are running fine. The problem is that nobody can produce the record.
The reflex is to read what banks do, because banks have done this longest. That leads to SR 11-7, the 2011 supervisory guidance issued jointly by the Federal Reserve and the OCC as OCC 2011-12, and to the validation apparatus built around it over fifteen years. It is a good document. It is also written for institutions with a Chief Risk Officer, a standing model risk committee, and a validation group of forty people whose job is to disagree with the modeling group. Copying that org chart into a 900-person company consumes a quarter of your data science capacity and answers no questions faster.
The useful move is to separate the ideas in SR 11-7 from the machinery it assumes. The ideas are cheap and work anywhere. The machinery is expensive, and most of it exists to satisfy an examiner you do not have.
What SR 11-7 actually says, stripped of the bank
The guidance rests on three claims. First, a model is any quantitative method that turns input data into an output used to make a decision, which sweeps in spreadsheets, rules engines, and gradient-boosted trees on equal terms. Second, model risk has exactly two sources: the model can be wrong, and the model can be used wrong. Third, the control that manages both is effective challenge, meaning critical review by people with the competence to find the error, the incentive to look, and the standing to make somebody act on what they find.
That definition is the most valuable thing to steal, because it settles arguments you would otherwise have for a year. The pricing spreadsheet that four people maintain and the sales organization trusts is a model. The vendor scoring API you call 40,000 times a day is a model, and the fact that you did not build it changes your validation approach, not whether it needs one. Nobody argues their thing is out of scope because it lacks a loss function.
The second claim reorders your priorities. Most of the model failures we get called into are not modeling errors. The model was fine and somebody pointed it at a population it was never fit on, read a score as a probability when it was a rank, or kept using it eighteen months after the process underneath it changed.
SR 11-7 practices · what transfers to a non-bank, by value per hour spent
Our ranking from build work, not a survey. The ordering is the point: the top four cost weeks, the bottom two cost headcount.
Effective challenge without a second organization
Banks get independence from structure. Validation reports to risk, development reports to the business, and the two lines never merge below the CRO. A company with eleven people who understand the models cannot buy that, and pretending otherwise produces the worst outcome: a nominally independent reviewer who lacks the depth to find anything and signs off to avoid looking obstructive.
Independence is a means. What you need is a reviewer who does not own the model's success. Three arrangements get you there without a validation group.
Cross-assignment inside the team. The engineer who validates the churn model is the one who built the forecasting model, and the reverse. They have the depth and no stake in this one shipping. It weakens when both sit on the same roadmap and the same bonus, which is why the next arrangement matters.
A written charge instead of a job title. The reviewer's assignment says what they must attempt to break, not that they should look it over. Reproduce the headline metric from raw data. Fit the simplest defensible alternative and report the gap. Find one input distribution where the model is materially worse and quantify it. A reviewer told to find something reports something. A reviewer told to review approves.
An outside pass on the top tier only. One or two models carry enough money or consequence to justify a real outsider. A thorough external validation of one production model runs roughly $25,000 to $60,000 and four to eight weeks, depending on whether the training data can be reconstructed. Running that across nine models a year is a program nobody finishes.
What decides whether any of this works is the finding itself. A finding needs a severity, a named owner, a due date, and a place where open findings are visible to whoever runs engineering. Findings that live in a PDF attached to an email are decorative. Put them in the same tracker as production defects, because that is a queue somebody already reads every morning.
Tiering, and why three tiers beats five
Models do not all get the same treatment, and the sorting rule has to be simple enough to apply in a meeting. Two questions do almost all of the work: what happens to a person or to the business if the output is wrong, and how much room does the model have to act before a human sees it.
Consequence sets the base tier. A model that denies someone credit, screens a job applicant, prices health coverage, or triggers a safety action sits at the top no matter how accurate it is. A model that ranks internal search results sits at the bottom. Autonomy adjusts one step either way: acting without review moves it up, a human review with authority to reject moves it down. That resolves most edge cases without a committee, which is the point.
| Tier | What lands here | Validation before launch | Ongoing |
|---|---|---|---|
| Tier 1 | Decides an outcome for a person or moves more than a stated dollar threshold; acts without review in a regulated or safety path | Full independent validation: conceptual soundness, data lineage, replication from raw, benchmark against a simple challenger, subgroup performance, sensitivity and stress tests. Four to eight weeks | Monthly metrics, quarterly back-test, annual revalidation |
| Tier 2 | Informs a decision a person makes; customer-visible output; touches personal or contract-controlled data | Peer review under a written charge, held-out evaluation with a stated threshold, documented intended use and limits. One to two weeks | Quarterly metrics, drift alerts routed to the owner |
| Tier 3 | Internal productivity, drafting, ranking, search over non-sensitive material | Register it, name an owner, state what it is for. One to two days | Annual confirmation it still exists and is still Tier 3 |
Most companies land near 5 percent Tier 1, 25 percent Tier 2, and the rest Tier 3. If your tiering produces thirty Tier 1 models the threshold is wrong, and the program dies of its own weight by the second quarter.
Validating a model you did not build
SR 11-7 is explicit that a purchased model is still your model risk, and this is the gap that hurts commercial companies most, because a growing share of the decision logic in a product arrives through an API. The vendor will not give you weights, training data, or architecture. That constrains the method without removing the obligation.
Four things are available and they are enough. Read the vendor's validation and development documentation for what is missing rather than what is claimed. Build a held-out set from your population and measure the vendor on it, which is the most valuable hour in the process, because vendor accuracy figures come from the vendor's population. Fit a simple challenger on your own data: if logistic regression on eight features gets within two points, you have learned the price of the contract and the size of your dependency. Then run stability, sending fixed inputs on a schedule to catch silent version changes, because vendors update models without release notes.
Put the right to that testing in the contract while you still have bargaining room. A clause requiring notice of material model changes, performance data on your segment, and a testing window before forced upgrades costs nothing at signature and is unobtainable at renewal.
Monitoring, in the three flavors that cost differently
"Monitor for drift" hides three separate measurements with three separate price tags, and teams routinely buy the cheap one believing they bought the expensive one.
Input drift needs no labels, so it runs from day one. Compare the live feature distribution against the training window. Population stability index is the standard instrument: below 0.10 nothing has moved, 0.10 to 0.25 warrants a look, above 0.25 means the model is being asked about a different population than it was fit on.
Output drift is nearly as cheap. Watch the score distribution, the approval or flag rate, and the share of low-confidence results. For language systems, watch refusal rate and mean response length. An approval rate that moves eight points with no product change is a signal before you know which direction is wrong.
Outcome drift is what matters and is the one nearly everybody skips, because ground truth arrives late. Defaults resolve in months, a hiring decision in a year. Build the label pipeline anyway and use proxies meanwhile: downstream correction rates, complaint volume, override rate, rework hours. A model with no outcome measurement cannot be shown to be wrong, and systems that cannot be shown to be wrong fail quietly for a year and a half. Every alert needs a response written before it fires, routed to the named owner: investigate, retrain, restrict the population, or roll back.
Where model failures originate · incidents we have been called into
Relative frequency in our incident work, not a survey. Note where the bottom bar sits: validating only the math misses most of the risk.
Where language models break the older framework
SR 11-7 assumes a fixed input schema, a measurable output, and a stable population. A retrieval-augmented assistant satisfies none of that, and the mismatch is real rather than cosmetic. The output space is open, so accuracy is undefined and you need a graded evaluation set with human-scored rubrics instead. The input space is unbounded, which is why prompt injection is a live control problem. The OWASP Top 10 for LLM Applications is the practical taxonomy, and MITRE ATLAS is the reference for adversary technique coverage when the system faces anything hostile. The model changes underneath you when the provider ships an update, so version pinning has to be engineered rather than assumed. Retrieval configuration is part of the model: chunk size, embedding version, and the number of retrieved documents change behavior as much as retraining, and teams almost never record it as a version change.
The tiering logic and the documentation shape still hold. What changes is the evidence. Instead of a ROC curve you produce a rubric-scored evaluation set with per-category pass rates, a red-team log naming the attacks tried and the ones that worked, a record of the pinned model version and retrieval configuration, and a measured escalation path. NIST's Generative AI Profile, published July 2024 as NIST AI 600-1, maps this onto the AI RMF functions if a questionnaire wants the crosswalk.
How this maps to the frameworks people will ask you about
You will be asked to name a framework, usually by someone filling a cell on their own spreadsheet. Three matter, and they do not conflict.
NIST AI RMF 1.0 is the common language, and the one federal customers and their primes reference. Its four functions map onto the program directly: Govern is tiering and ownership, Map is inventory and intended use, Measure is validation and monitoring, Manage is findings and incident response. Mapping artifacts you already produce onto those four takes about a day and answers most questionnaires.
ISO/IEC 42001:2023 is the certifiable one, built as an auditable management system in the familiar ISO shape. It matters when a customer contract demands a certificate rather than an assertion. Certification typically runs 9 to 18 months from a standing start and carries real audit cost, so it should be a sales decision rather than a governance decision.
Sector rules override both where they apply. Under Regulation B at 12 CFR 1002.9 a creditor must state specific principal reasons for an adverse action, and the CFPB said plainly in Circular 2022-03 that model complexity is not a defense. Employment selection runs into the Uniform Guidelines at 29 CFR Part 1607 and its four-fifths convention, and NYC Local Law 144 has required annual independent bias audits of automated employment decision tools since July 2023. Selling into federal, the flowdown you meet references NIST AI RMF, 800-53 controls for the hosting environment, and 800-171 wherever controlled unclassified information is touched.
What the first six months costs
Numbers, so the budget conversation is short. A company with 20 to 60 models in scope and a working data platform can stand up a real program in about six months on roughly 0.5 to 1.0 full-time-equivalent plus a defined external assist.
Weeks 1 through 4, discovery and inventory: read the single sign-on app list, sweep expense data for model vendors, pull egress logs for calls to model API endpoints, and run a two-week no-consequence registration amnesty. Expect 30 to 50 percent more models than anyone predicted, mostly spreadsheets and vendor features. Weeks 3 through 6, tiering and policy, a six-page document rather than a binder. Weeks 5 through 12, validate the Tier 1 models one at a time, with the outside pass on the top one or two. Weeks 8 through 16, wire monitoring into the platform you already run instead of buying an observability product first. Weeks 12 through 24, the findings loop and the first quarterly review, which is where you learn whether any of it is real. Internal effort runs 0.5 to 1.0 FTE, external validation $25,000 to $60,000 per Tier 1 model, tooling often zero in year one.
The line that overruns is always inventory, because reconstructing training data for a model built three years ago by someone who has left takes weeks. Where reconstruction fails, say so in the record and treat the model as unvalidatable until it is rebuilt. That sentence is worth more to an auditor than a validation report resting on data nobody can locate.
The failure modes we see most
The binder. A policy written to survive a review it will never get, 40 pages long, approved once, opened never. The test is whether an engineer shipping a model this week can learn what they owe in under five minutes.
Validation as a gate rather than a loop. A model validated at launch and never revisited is a model validated on a population that no longer exists. Revalidation cadence belongs in the tier definition, and the annual reassessment should be a calendar entry with an owner rather than an intention.
Monitoring the model instead of the decision. Accuracy holding steady while the business outcome degrades is common, and it means the model is fine while the process around it moved. Instrument the decision, not only the prediction.
Inventory in a spreadsheet. It decays within two quarters. Attach model records to assets that some team already reconciles: the configuration management database, the service catalog, the GRC platform, the model registry that your deployment pipeline already writes to.
Bottom line
The bank framework is worth reading. The bank organization is not worth copying. Take the definition of a model, the two sources of risk, and effective challenge. Build an inventory you trust, three tiers set by consequence and autonomy, validation sized to the tier, monitoring split across input, output, and outcome, and a findings queue somebody reads. That program answers the customer questionnaire, survives the diligence request, satisfies the federal flowdown, and catches the model that has been quietly wrong since the population shifted last spring. The version that costs a department is the version nobody finishes.
Frequently asked questions
No. It is supervisory guidance for institutions examined by the Federal Reserve and the OCC. It is widely used as a reference standard outside banking because its definitions are clear and auditors, insurers, and acquirers recognize it. You borrow the concepts, not the obligation.
Not as a separate reporting line. What you need is that the reviewer does not own the model's success, works from a written charge that says what to try to break, and can file a finding that lands in a tracked queue. Reserve outside validators for the one or two highest-consequence models.
Under the SR 11-7 definition, yes, if it turns input data into an output used to make a decision. Pricing sheets and capacity planners routinely carry more business consequence than the machine learning models in the same company, and they almost never have an owner or a version.
Test it on a held-out set drawn from your own population, fit a simple challenger on your data to measure what the contract is buying, read the vendor's documentation for what it omits, and monitor for silent version changes. Negotiate change notice and a testing window into the contract at signature.
Only if a customer contract requires a certificate. It typically runs 9 to 18 months and carries recurring audit cost. NIST AI RMF alignment answers most questionnaires and most federal flowdown language at a small fraction of the effort.