The question that starts this
A customer's risk officer sends four questions before renewal. Who owns this model. What changed in it since we signed. What is it not supposed to be used for. Show us the validation that somebody other than the person who built it performed. Whoever fields that email inside your company knows within about ninety seconds whether the answers exist as documents or whether they exist as things people remember. That gap is the whole subject. Governance is not the policy you wrote. It is whether the record was being kept while the work was happening.
The reason this arrives now, and keeps arriving, is that your output stopped being an input to a human judgment and became the judgment. A score that once informed an underwriter now approves the case. A ranking that once suggested an order now sets it. When your number is the decision, your customer inherits your model risk, and their auditors, their regulators, and their own board start asking about a system they cannot see.
The good news is that this is a solved engineering problem with an unusually mature literature behind it. Banks have run formal model risk management under the Federal Reserve and OCC guidance SR 11-7 since 2011. That document is fifteen years old, twenty-one pages, and still the clearest description of what governing a model means. NIST AI RMF 1.0 gave the same ideas a vocabulary that non-banks accept. ISO/IEC 42001, published December 2023, made it certifiable. None of this requires inventing anything. It requires building four artifacts and keeping them current.
What your customer is actually asking
Read the four questions again and notice that none of them is about accuracy. Nobody asks a vendor to prove the model is good, because everyone knows that claim is unfalsifiable across the range of conditions the model will actually meet. They ask about control. Can you tell me what it is, who is accountable, what changed, and who checked the person who built it.
This shift confuses technical teams badly. An engineering group answers with a benchmark table and gets a follow-up asking the same questions again. The benchmark answers "is it good," which was never in question. The risk officer is testing whether a failure would be noticed, bounded, and traceable. Those are properties of a process, not of a model, and no evaluation result demonstrates them.
The practical translation: your governance package needs to make a specific claim, which is that when this model does something wrong, we will find out from our own instrumentation rather than from your complaint, we will know which version did it, and we will be able to put back the one that did not.
Vendor Governance Questions — Our Ranking by How Often They Stall a Renewal
Relative frequency across vendor diligence work, not a survey. The ordering is the point: the top two consume most of the delay.
SR 11-7 is still the best twenty-one pages on this
The Federal Reserve and OCC published SR 11-7 in April 2011 for banks, and the framing has outlived its original audience because it got the definitions right. It defines model risk as the risk of adverse consequences from decisions based on incorrect or misused model output, and it splits the cause into two: the model has a fundamental error, or the model is used incorrectly. That second half is the one vendors forget. A model that is technically sound and applied to a population it was never fitted on is a failure of governance, not of data science, and the vendor who shipped it without a written scope statement owns part of that.
Three mechanisms in the document do the work. Effective challenge, meaning critical review by people with the competence, the incentive, and the standing to say no. An inventory covering every model in use with attributes recorded per model. And validation broken into three components: evaluation of conceptual soundness, ongoing monitoring including benchmarking, and outcomes analysis comparing predictions against what happened.
Standing is the load-bearing word in effective challenge. A junior reviewer who reports to the model's author cannot perform it, regardless of skill. The reporting line is the control. When a customer asks who validated the model, they are checking the org chart, not the résumé.
The four artifacts, and nothing else at first
A governance program that starts by writing a policy produces a policy. Start with the artifacts instead, because they are what a customer, an auditor, and your own on-call engineer read. The policy is a description of how these four get maintained, and it should be written last and be short.
| Artifact | What it has to answer | How it usually fails |
|---|---|---|
| Model inventory | Every model in production, its version, its owner, its risk tier, its customers, its last validation date | A spreadsheet maintained by one person, correct on the day of the audit and stale six weeks later |
| Model card per model | Intended use, out-of-scope use, training data provenance, performance by slice, known failure modes | Written once at launch, never revised through eleven retrains |
| Change record | What changed, why, who approved it, what evidence was produced, what the rollback is | Threshold edits and prompt changes that never produced a version, so the history has holes nobody can see |
| Validation report | Conceptual soundness, an independent test on data the builder did not choose, outcomes analysis | Performed by the team that built the model, which makes it a test report rather than a validation |
Four documents. A company with three models in production can build all four in six to eight weeks of one senior engineer's time plus a few days each from the model owners. The reason it usually takes a year is that firms start with a framework selection exercise, which produces meetings.
Tier the models or drown
Applying the full package to every model is how governance programs die. Nobody sustains quarterly independent validation on forty models, half of which rank internal search results. Tier by consequence and be honest about the tiers.
The test that works: what happens to a person or an account when this model is wrong, and how quickly would anyone notice. A model whose error denies someone credit, flags an employee, alters a clinical recommendation, or changes what a customer is charged is a high tier no matter how simple the math is. A gradient-boosted model that reorders a product carousel is not, no matter how sophisticated it is. Consequence sets the tier; the technique does not.
Three tiers is enough. High tier gets independent validation before release and annually thereafter, full model card, monitoring with paging thresholds, and a named accountable owner. Middle tier gets a documented internal review by someone outside the build team, a model card, and monitoring without paging. Low tier gets an inventory row and a scope statement. Write the tier criteria down before you tier anything, because tiering after the fact invites everyone to argue their model into the cheap bucket.
Change control is where the money is
Of the four artifacts, the change record is the one that most often does not exist in any form, and the one that answers the question customers ask first. It also catches the failure mode we see more than any other in this work: a threshold or a prompt gets edited under time pressure, the behavior of the system changes materially for a meaningful share of inputs, and no version was created, so the change history shows a quiet period during exactly the week everything went strange.
The rule that fixes it is short. Anything that can change the output changes the version. Weights, features, preprocessing, thresholds, calibration, fallback behavior, system prompts, retrieval corpora, the model endpoint you call at a third party. All of it. If it is edited in a config map or a vendor console rather than in a versioned artifact, you have a governance hole with a nice dashboard over it.
For anything built on a foundation model you do not control, add one more line to the record: the provider's model identifier and the date you pinned it. Providers deprecate and retire model versions on their own schedule, and a silent server-side update means your governed system changed without a change record. Pin the version, subscribe to the deprecation notices, and treat a forced migration as a release with its own validation, because that is what it is.
Where the Governance Record Has Holes — Frequency We Find Each Gap on First Assessment
Editorial ranking from assessment work, illustrative rather than a measured statistic.
Outcomes analysis is the part everyone skips
Monitoring input distributions is easy and popular. Comparing what the model predicted to what actually happened is harder, less popular, and the only thing that catches a model that has quietly stopped working while all its input statistics look normal.
The obstacle is real: labels arrive late. A default shows up in ninety days, a claim resolves in six months, a maintenance prediction is confirmed on the next inspection cycle. Teams conclude that outcomes analysis is impossible and stop there. The workaround is to build the reconciliation job anyway and let it run on a lag. A monthly job that joins ninety-day-old predictions against outcomes that have since landed, reports realized accuracy by slice, and compares it against the acceptance run, is worth more than every drift dashboard combined. It is also, in our experience, roughly two weeks of work when the prediction log has stable identifiers, and roughly two months when it does not, which is the actual argument for logging a stable prediction id from the first release.
Where outcomes never become observable, say so in the model card and name the proxy you use instead. A stated proxy with an honest description of what it fails to capture reads as competence. Silence about outcomes reads as evasion, and a reviewer who has read a hundred of these will call it.
Which framework to point at
Customers ask you to name a framework, so name one and align to it rather than assembling a private taxonomy that forces every reviewer to translate.
NIST AI RMF 1.0 is voluntary, free, and the common vocabulary in the United States. Its four functions of Govern, Map, Measure, and Manage map cleanly onto the four artifacts. It is the right choice when your customers are American and mixed across sectors, and the accompanying Playbook gives concrete suggested actions per subcategory that reviewers recognize on sight.
ISO/IEC 42001 is certifiable, which matters when procurement wants a certificate rather than a description. Budget realistically. A first certification for a small organization typically runs a year or more of preparation and a five-figure audit fee, and the standard's requirement for internal audit and management review means somebody's job changes permanently.
SR 11-7 is where the language comes from when your customer is a bank, an insurer, or anyone whose examiner uses those words. If a customer says "effective challenge" or "model inventory" in a diligence call, answer in that vocabulary.
OWASP LLM Top 10 and MITRE ATLAS cover the adversarial half for language-model systems: prompt injection, training-data poisoning, model extraction, insecure output handling. These are threat catalogs, not governance frameworks, and they belong in your security testing plan and referenced from the model card rather than substituting for the governance package.
If federal customers are in your future, add NIST SP 800-53 and, for controlled unclassified information on your systems, SP 800-171. FedRAMP authorization applies to the cloud service offering rather than to the model, and it is a separate and considerably longer program. Do not conflate the two in a customer conversation; the distinction is a fast credibility test.
What this costs and how long it takes
Real numbers, for a company with three to eight production models and no formal program today.
Assessment and inventory. Two to four weeks. Interview the model owners, walk the deployment path, build the inventory, assign tiers. The deliverable is a list of models with owners and gaps, and it usually finds one or two models nobody had counted because they shipped inside a product feature rather than as a model.
The first model card and change record. Three to five weeks for the highest-tier model, then one to two weeks per model afterward as the template stabilizes. The first one is slow because it forces decisions about what a version is.
Independent validation of the top-tier model. Four to eight weeks depending on whether a clean held-out set exists. Half the work is often assembling an evaluation set the build team did not choose, and that is the half that produces the finding.
Outcomes reconciliation. Two weeks to two months, decided almost entirely by whether prediction logs carry a stable joinable identifier.
Whole first cycle: three to five months of largely one senior engineer's time, plus a few days each from model owners and one person outside the build team with the standing to disagree. Ongoing maintenance is far smaller once the gates are in the release pipeline rather than in a checklist someone remembers.
How we do this work
We build the artifacts and the machinery under them, in that order, and we start with the change record because it is what stalls renewals. The inventory comes out of a walk through the deployment path rather than a questionnaire, because questionnaires miss the models that shipped as features. Validation is performed by an engineer who did not build the model and who reports outside the build team for the duration, which is the only arrangement that makes the report worth anything to your customer.
The gates go into the release pipeline. A promotion that lacks an evaluation record, an approver, and a rollback target fails the build rather than fails a review, because a control that depends on someone remembering it is not a control. Then the policy gets written, in a few pages, describing what already runs.
What you have at the end is a package you can hand a customer's risk officer without a meeting first: an inventory, a card per model, a change history with no holes, and a validation report signed by someone who did not build the thing. That package is also the answer to their next four questions, which is the point of building it before they arrive.
Bottom line
When your customers make decisions on your model's output, your governance becomes part of their control environment, and they will eventually ask you to prove it exists. The proof is four artifacts kept current by machinery rather than by discipline: an inventory, a card per model, a change record with no gaps, and a validation performed by somebody with the standing to say no. SR 11-7 named the mechanisms fifteen years ago, NIST AI RMF gave them a shared vocabulary, and ISO 42001 made them certifiable. The work is knowable, it is bounded, and it takes months rather than years when it starts with the artifacts instead of the policy.
Frequently asked questions
Not as regulation. It applies as the clearest available definition of what governing a model means, and its vocabulary is what your customers in regulated sectors will use when they ask. Borrow the mechanisms: inventory, effective challenge, and validation in three parts.
Independence is about reporting line and incentive, not headcount. An engineer from a different team who reports outside the model owner for the duration of the review satisfies it. So does an outside firm. The person who trained the model does not, at any level of seniority.
Usually not at first. Most customers accept alignment to NIST AI RMF with the artifacts to back it. Certification matters when a procurement process requires a certificate rather than a description, and it is worth planning a year and a five-figure audit fee for.
Yours, for the system your customer bought. Pin the provider's model version, record it in the change history, subscribe to deprecation notices, and treat a forced migration as a release requiring its own validation. Your customer's risk officer will not accept a provider's documentation as a substitute for yours.
Run the reconciliation on a lag rather than skipping it. A monthly job joining old predictions against outcomes that have since resolved gives realized accuracy by slice. Where outcomes are genuinely unobservable, name the proxy in the model card and state what it fails to capture.