The problem is not accuracy, it is testimony
A ratings firm, a lender, a health-tech company and a defense-adjacent manufacturer all eventually hit the same wall. The model works. The offline metrics are strong. The business case is written. Then someone from a supervisory function, an examiner, a state insurance department, a notified body, an auditing agency, asks a question that the team has never had to answer out loud: why did this score come out the way it did, and how do you know it will keep doing that. The team answers with a chart. The chart is not an answer. Six months of remediation follows, and it costs more than building the model did.
We call this the ratings-agency problem because ratings firms lived it first and at the largest scale. A rating is an opinion that carries regulatory weight, so the method behind it became a supervised object rather than a trade secret. Everyone building consequential models is arriving at the same place by a different road. Credit and capital models got there through SR 11-7. Insurance rating models got there through state filing regimes and the NAIC model bulletin. Clinical decision support got there through FDA software guidance. Employment screening got there through New York City Local Law 144 and its bias audit. The domains differ. The demand is identical, and it is not a demand for accuracy.
The demand is for testimony a third party can check. Not a claim that the model is good. A record that lets someone who does not trust you, and who has authority over you, reconstruct what you did, satisfy themselves you did it, and write their name next to that conclusion. Accuracy is one input to that record. It is rarely the input that stalls a launch.
What Actually Delays a Model Launch — Our Ranking From Remediation Work
Editorial weighting from remediation engagements and practitioner reading. Illustrative of where launches stall, not a measured statistic.
SR 11-7 is the text everyone is quietly using
The Federal Reserve and OCC supervisory guidance on model risk management, issued in April 2011 as SR 11-7 and OCC Bulletin 2011-12, is twenty-one pages long and remains the most useful document in this field. It applies to banks. Its vocabulary spread well past them, because insurance regulators, procurement offices and internal audit functions all borrowed it, and because it describes what any serious reviewer wants.
Four ideas from it do most of the work. First, a model is a quantitative method that turns input data into an estimate, and model risk is the risk of loss from decisions based on incorrect or misused output. Misuse is named as a separate failure. A correct model used outside its intended purpose is a model risk event, which is why intended use has to be written down before anyone can say the model is fine.
Second, validation is a set of activities carried out by parties who are independent of development and who have both the competence and the standing to challenge what they find. That last clause is the one firms skip. A reviewer who reports to the person shipping the model is not independent, and an examiner can read an org chart.
Third, validation has three components: evaluation of conceptual soundness, ongoing monitoring including process verification and benchmarking, and outcomes analysis including back-testing. Conceptual soundness means the theory and the design choices, documented with evidence, including the developer's own testing of assumptions and limitations. A validation that only reports metrics has done one of three parts.
Fourth, and most quoted, the guidance asks for effective challenge: critical analysis by objective, informed parties who can identify limitations and assumptions and produce appropriate change. It also states the standard for documentation directly, that model development should be documented so that parties unfamiliar with a model can understand how it operates, its limitations, and its key assumptions. That sentence is the entire specification. Most teams fail it, and it is not because their model is bad.
What a reviewer opens first
Watch an examiner or an independent validator work and the sequence is consistent. They read the stated purpose and the population it applies to. They look for the boundary, the conditions under which the model is not to be used. They pick a handful of real decisions and ask you to reproduce them. They ask who reviewed the model and what that person had the authority to do about what they found. They ask what changed since the last review and how you knew.
Nothing in that sequence is about your architecture. A gradient-boosted ensemble with clean lineage and honest limits clears review faster than a logistic regression whose feature table was rebuilt last spring and whose old scores can no longer be recomputed. Interpretable model families help less than people assume. Reproducibility decides the meeting.
Reproducibility here has a precise meaning: given a decision made on a specific date, recover the exact inputs as they stood at that moment, the exact model version, the exact threshold and policy in force, and produce the same output. That requires point-in-time feature storage rather than a live query against current data, content-addressed model artifacts rather than a path that gets overwritten, and thresholds stored inside the version record rather than in a config map someone can edit at 2am. We wrote the schema for this in a model registry that earns its keep. The regulatory case is the same engineering, with a longer retention clock.
The reason code is a product surface, not a report
Where a decision affects a person, an explanation is often a legal obligation with a specific shape. Under the Equal Credit Opportunity Act and Regulation B at 12 CFR 1002.9, an adverse action notice must state the specific principal reasons for the action. The CFPB has said plainly that this holds for complex algorithms, and that a creditor may not rely on the checklist of sample reasons in the regulation's appendix if those do not reflect the actual reasons. The regulator's own framing is that the technology is not an excuse.
This has an architectural consequence that teams miss. Reason codes must come from the scoring path, at scoring time, stored with the decision. A post-hoc explainer run later against a rebuilt feature vector will drift from what actually happened, and the drift is invisible until someone compares a notice against a stored decision and finds they disagree. SHAP and its relatives are useful diagnostic tools. They are a poor system of record, because the explanation they produce depends on a background dataset and a model state that both move.
The design that survives is boring. Compute contributions inside the inference call. Map them to a fixed, reviewed vocabulary of reasons written in language a customer can act on. Store the vocabulary version alongside the decision so a notice sent in March can be defended in November. Then test the mapping itself: sample decisions, have a human read the reason next to the input, and count how often the reason is true and actionable. That number belongs in your validation report, and almost nobody produces it.
Fairness testing is a filing artifact now
Disparate impact analysis moved from an internal exercise to a filed document in several sectors at once. New York City Local Law 144, enforced since July 5, 2023, requires an annual independent bias audit of automated employment decision tools, calculation of selection or scoring rates and impact ratios by sex and by race and ethnicity category, and public posting of a summary of results. Colorado's SB 21-169, effective for insurers, bars unfair discrimination through external consumer data and algorithms and carries a governance and testing regime through Division of Insurance regulation. The NAIC model bulletin on the use of artificial intelligence by insurers, adopted in December 2023, has been issued by a majority of states and asks for a written AIS program covering governance, risk management and testing.
The engineering point is that these regimes want a repeatable measurement, not a one-time study. Impact ratios across protected classes, computed on production decisions, on a schedule, using a documented method, with the results stored. Build it as a pipeline that runs monthly against real outcomes and writes to a table. Build it as a consulting deliverable and you will pay for it again every year, and each year's numbers will be slightly incomparable to the last.
Two mechanics matter more than the choice of metric. Class attribution: in many contexts you do not collect race or ethnicity, so proxy methods such as Bayesian Improved Surname Geocoding get used, and the proxy's own error rate has to be reported alongside any ratio it produced. And thresholds: the four-fifths rule from the Uniform Guidelines on Employee Selection Procedures is a rule of thumb inside a specific enforcement context, not a general safe harbor, and presenting it as one in front of a regulator invites a question you will not enjoy.
Where the Documentation Budget Should Go — Weight by Review Impact
Editorial weighting from validation and remediation work. The last row is the section teams write first and reviewers skim.
Frameworks: which one, and how much it actually costs
Three names come up in every buying conversation, and they do different jobs. The NIST AI Risk Management Framework, released January 2023, is free, voluntary, and organized around four functions: govern, map, measure, manage. It gives you a vocabulary regulators recognize and a structure for the artifacts. It certifies nothing, which is exactly why it is easy to adopt.
ISO/IEC 42001:2023 is the certifiable one, a management system standard in the mold of ISO 27001. If a customer or a supervisor wants a third-party attestation that your governance exists, this is the route. Budget accordingly: an accredited certification body's audit for a mid-size scope commonly runs in the tens of thousands of dollars for the initial two-stage cycle, plus annual surveillance, and the internal preparation is usually the larger number. Six to twelve months from a standing start is realistic. Where the model is generative or agentic, the OWASP Top 10 for LLM Applications and the MITRE ATLAS adversarial threat matrix supply the attack vocabulary that a general framework does not.
If the buyer is a federal agency or a contractor selling into one, a further layer applies. NIST SP 800-53 controls carry the authorization, SP 800-171 governs controlled unclassified information on your systems, and FedRAMP governs cloud services sold to agencies. These are separate from model risk and stack on top of it. A model that is beautifully validated and hosted somewhere the customer cannot authorize does not ship.
The honest cost picture for a first consequential model, built to survive a real review, from our own project planning: three to six months of engineering effort dedicated to reproducibility, lineage, reason codes and monitoring, on top of the modeling itself. In a typical mid-market build that is a meaningful fraction of the program budget. It is also the fraction that decides whether the model reaches production, and it is roughly a third of what the same work costs when done as remediation with a supervisory deadline on it.
Independence you can actually staff
Effective challenge needs someone who can say no and survive it. Most companies below a few thousand employees cannot staff a separate validation group, and the usual response is to have the lead engineer write a validation report about their own model. A reviewer spots that in under a minute.
Three arrangements work. A validator inside the company who reports through risk, audit or legal rather than through the product line, even if the role is part-time. An external validation engagement, scoped as an independent review with its own findings and a management response, which is what banks do for models they cannot staff internally. Or a rotating internal review where a team that did not build the model performs the validation under a written charter that gives it authority to hold a release. All three produce the artifact a reviewer needs: a finding written by someone with something to lose, and a documented response.
What does not work is a checklist signed by a manager. The signature is not the point. The point is evidence that someone competent looked for the failure and reported what they found.
The artifact set, and where it lives
What a serious review consumes is smaller than teams fear and more structured than they expect. A model document covering purpose, population, design rationale, data, assumptions, limitations and the developer's own testing. A validation report from an independent party, with findings and management responses. A monitoring specification naming metrics, thresholds, cadence and owner. A change log tying every version to an approval and a reason. And an evidence store: point-in-time features, versioned artifacts, decision records with reason codes.
| Artifact | Question it answers | Where it fails |
|---|---|---|
| Model document | What is this for, on whom, and where does it stop | Written as an architecture paper; intended use appears on page 14 in a subordinate clause |
| Validation report | Who challenged it, with what authority, and what did they find | Authored by the development team; findings section is empty |
| Data lineage | Where did each feature come from and under what right | A pipeline diagram instead of a per-feature record; third-party terms unread |
| Decision record | Can you reproduce this specific score from that specific day | Features recomputed from current data; the old value no longer exists |
| Monitoring spec | How will you know it stopped working, and who acts | Dashboards with no thresholds and no named owner |
| Change log | What changed, when, why, and who approved it | Git history offered as governance; threshold edits absent entirely |
Each is either a byproduct of a system built correctly, or a document written under deadline pressure about a system that was not. The first costs engineering time. The second costs a launch date.
Retention is longer than your infrastructure assumes
The clock on this evidence is set by regulation, not by your storage policy. Regulation B requires retention of records of a credit application for 25 months, and 12 months for business applicants under the relevant provision. Fair lending examinations reach back across cycles. Insurance filings persist for the life of the rate. Where a decision is later contested, the question arrives years after the model that made it was retired.
The design implication arrives late: retired models and their feature stores have to remain reconstructible, not merely archived. A pickled artifact whose runtime environment no longer builds is not evidence. Pin the base image by digest, store the resolved dependency lock, and test a cold reconstruction of a retired version once a year.
How to build so the explanation already exists
The consistent pattern across the companies that clear review without drama is that they treated the reviewer as a user with requirements, at design time, alongside the customer. Concretely, four decisions made early carry most of the weight.
Write the intended-use statement before the first model is trained, including the population and the conditions under which the model is not to be used. It will be wrong and you will revise it. Writing it forces the scope conversation while scope is still cheap to change.
Make point-in-time correctness a property of the feature layer rather than a convention people follow. If the training path and the serving path can read different values for the same entity at the same timestamp, you have a reproducibility failure that will surface during a review rather than during testing.
Put thresholds, calibration and fallback behavior inside the version record, so changing a cutoff produces a new version with an approval attached. More production surprises come from an unrecorded threshold edit than from a bad retrain, and a threshold change is exactly the kind of thing a reviewer asks about.
Generate reason codes in the scoring path and store them with the decision. Then measure whether they are true, and put that measurement in the validation report. A firm that can show a reason-code accuracy number has already answered a question most competitors cannot.
Bottom line
The ratings-agency problem is not a documentation problem that arrives after the model works. It is a property the system either has or does not, and retrofitting it costs several times what building it in costs, on someone else's schedule. The regulator is not asking you to make the model simpler. They are asking you to make it accountable: reproducible from stored inputs, bounded by a stated purpose, challenged by someone independent, explained by the same code path that made the decision, and watched by a monitor with a threshold and an owner. Build that and the review is a conversation. Skip it and the review is the project.
Frequently asked questions
Not as law. As the reference text your reviewers were trained on, almost certainly. Insurance regulators, internal audit functions, enterprise risk committees and procurement offices have all adopted its vocabulary, so writing to it means writing in the language your reviewer already reads.
Rarely. Reviewers ask far more often about reproducibility, lineage and independent challenge than about model class. A well-instrumented ensemble clears review faster than a simple model whose past scores cannot be recomputed.
You can compute them that way, but store them with the decision at scoring time. A post-hoc run against a rebuilt feature vector drifts from what actually happened, and the divergence only appears when a notice is compared against a stored record.
NIST AI RMF is free and gives you structure and vocabulary. ISO/IEC 42001 gives you a third-party certificate when a customer or supervisor demands attestation. Most firms start with the framework and certify later, if the market asks.
Give the reviewer a reporting line outside the product organization, hire an external validator under a scoped engagement, or run a chartered internal review by a team that did not build the model. What matters is authority to hold a release and a written finding with a management response.