What the letter is, and why it landed on your team
SR 11-7 is a supervisory letter issued in April 2011 by the Federal Reserve, published the same day by the OCC as Bulletin 2011-12. It is short. It defines a model, defines model risk, and then describes three things a firm has to do: develop models with documented rigor, validate them independently, and govern the whole set with policies, an inventory, and named accountability. That is the entire structure. Everything else you have been handed, the 40-page validation template, the tiering matrix, the annual attestation, is somebody's interpretation of those three headings.
It landed on engineering for a reason that has nothing to do with banking. The definition of a model in the letter is functional and deliberately wide: a quantitative method that applies statistical, economic, financial, or mathematical theories and assumptions to process input data into output used for a business decision. Nothing in that sentence says the model was fitted by a data scientist. A ranking service, a fraud threshold, a routing rule with fifteen conditions, an embedding-based match, a language model in a review pipeline. If the output changes what the business does, it is inside the definition, and somebody in compliance has now read that sentence carefully.
The second reason is spread. SR 11-7 governs banks, but its vocabulary migrated. Insurance regulators borrowed it. Ratings and data firms found it in the diligence questionnaires their customers send. Health-tech companies selling into payers get asked for validation reports written to its structure. The Fed's own SR 15-18 extended it to large firms' capital planning, and the CFTC, FHFA, and state insurance departments all echo pieces of it. If a Fortune 500 buyer's risk team is asking you for a validation report, they are asking for SR 11-7 shaped output whether or not they use the number.
Where validators find gaps · machine learning systems under SR 11-7 review
Editorial weighting from public sources and practitioner reading. Illustrative, not a measured statistic.
The two sources of model risk, and why the second one bites
The letter names two. A model can have fundamental errors and produce inaccurate output. Or a model can be used incorrectly, applied outside the conditions it was built for. Engineering teams instinctively work the first and ignore the second, because the first is a bug and the second is somebody else's decision.
The second one is where the expensive incidents live. A propensity model fitted on one customer segment gets called by a new service for a segment it never saw. The model did not break. Nothing alerted. The scores looked plausible because scores always look plausible, and eight months of decisions went out before anyone reconciled outcomes. The mechanism that would have caught it is not better modeling. It is a documented boundary of use, enforced at the call site, so that an input outside the fitted range returns a refusal instead of a number.
That is an engineering control and nobody in compliance can build it for you. It is worth saying plainly because it reframes the whole exercise: SR 11-7 conformance is mostly software design, not paperwork.
Component one: development evidence, produced as you build
The letter asks that model development be documented well enough that a competent third party who was not there can understand what was built, why the design was chosen, what data went in, and how it was tested. The failure mode is universal and it is a timing failure. The work happens, then six or nine months later somebody is asked to write it up, and the record of why the team rejected the simpler baseline is in a Slack channel that has since been archived.
Four artifacts satisfy this, and each can be produced at the moment the decision is made rather than reconstructed later. A statement of purpose and boundary, written before the first model is fitted, naming what decision the output feeds and what populations and ranges it is fitted for. A record of alternatives, three or four lines per rejected approach, saying what was tried and what the measured result was. A data lineage record naming sources, extraction window, exclusion rules, and known quality problems. And a test record that is the output of a run, not prose about a run.
The mechanism that makes this cheap is putting all four in the repository next to the code, versioned with it, and making the training pipeline emit the last two automatically. A training job that writes its dataset hash, row counts, exclusion counts, feature statistics, and held-out metrics into a signed artifact on every run has produced most of a validation package as a side effect. A team that does this spends hours on documentation. A team that does not spends four to six weeks per model, at a fully loaded cost somewhere between forty and eighty thousand dollars, and produces a worse record.
Component two: effective challenge, and what makes it effective
This is the phrase in the letter that matters most and gets diluted fastest. Effective challenge means critical analysis by objective, informed parties who can identify limitations and assumptions and produce appropriate change. The letter unpacks it into three conditions, and each is a real test.
Incentive. The challenger has to be free to say the model is wrong without it costing them. A validator who reports to the head of the business that owns the model does not meet this, no matter how good they are. In a mid-size company the practical answer is usually a reporting line into risk, legal, or the CTO, plus a written rule that the challenger's findings go into the record unedited by the model owner.
Competence. The challenger has to be able to read the code and rerun the experiment. Model risk teams staffed entirely with people who cannot run a training script produce reviews that check whether documents exist. That is a completeness check, not a challenge, and validators from the regulator can tell the difference in one conversation.
Influence. Findings have to be able to stop a release. If every finding to date has been accepted as a risk and the model shipped anyway, you do not have effective challenge, you have a file. One documented case where a validation finding delayed a launch is worth more in an examination than a hundred clean reports.
The scale question comes up immediately. A firm with nine models does not need a validation department. Independence is a property of the reporting line and the record, not of headcount. A qualified engineer from another team, a contracted reviewer, or an internal audit function can all satisfy it, provided the three conditions above hold and the review is written down with its findings, its disposition, and who decided.
Component three: governance, the inventory, and the policy nobody reads
The third component asks for policies, roles, an inventory, and documented accountability. The part with teeth is the inventory, because it is the only artifact where absence is visible. A model missing from the inventory has never been tiered, validated, or monitored, and the first question in every examination and most customer diligence reviews is how the list was built.
Self-declaration always undercounts. The inventories that hold up are built partly by discovery: scanning deployment manifests and job schedulers for anything that loads a serialized model or calls a scoring endpoint, reading egress logs for traffic to model API providers, and searching the data warehouse for scheduled jobs whose output feeds a decision table. Then a short amnesty window catches the rest. Two weeks, no consequences for registering something, and anything discovered afterward gets switched off.
Every row names a person. Not a team, not a department. A team does not answer email at eleven at night when the scores go sideways.
| SR 11-7 asks for | What engineering actually builds | Where it lives |
|---|---|---|
| Sound development, documented | Purpose and boundary statement, rejected-alternatives log, dataset hash and lineage emitted by the training job, held-out metrics written as a run artifact | Repository, versioned with the code; artifacts in the registry |
| Conceptual soundness review | A rerunnable notebook or script the challenger executes, not a slide deck describing it | Repository, with pinned dependencies and a fixed seed |
| Outcomes analysis | A join between predictions and realized outcomes on a schedule, with the lag documented | Warehouse table, one row per prediction, retained for the model's life |
| Ongoing monitoring | Input distribution checks, output distribution checks, subgroup performance, alert thresholds set before launch | The same monitoring stack as the rest of production |
| Limits of use | Range and population checks at the call site that refuse rather than extrapolate | Serving code, enforced, with the refusal counted |
| Change control | Retraining as a promoted release with a gate, never a cron job that quietly swaps weights | The deployment pipeline |
| Inventory and ownership | A registry row created by the pipeline itself, so registration is not a human step | Model registry, reconciled against deployment manifests |
The three places machine learning strains the 2011 assumptions
The letter assumes a model is a fixed artifact, documented once, validated before use, revalidated annually, changed rarely. Three properties of modern systems break that, and knowing which one you are hitting tells you what to build.
Cadence. A model that retrains every two weeks cannot wait four months for validation. The answer is to validate the pipeline rather than the artifact: the validator reviews the training code, the data contract, the acceptance thresholds, and the promotion gate once, thoroughly, and then each retrain is checked automatically against those thresholds. A run that fails the gate does not deploy. A run that passes deploys with a record. Full revalidation triggers on a material change, meaning new features, a new population, a new objective, or a threshold drift beyond a documented band.
Opacity. The letter asks for conceptual soundness, and a gradient-boosted ensemble on 400 features does not have an economic story per coefficient. Substitute evidence works: a simple benchmark the complex model must beat by a stated margin to justify itself, stability of feature importance across refits, sensitivity analysis on the top features, and a documented review of any feature that is a proxy for something you may not use. That package answers the question the letter is asking, which is whether the design is defensible, not whether it is linear.
Dependency. A hosted model you call by API is a third-party model under the letter's own vendor paragraph, and the paragraph is explicit that using a vendor product does not transfer the risk. The provider can change the weights under a stable model name. Pin versions where the contract allows it, keep a fixed evaluation set you rerun on a schedule so a silent change shows up as a metric move, and negotiate notice of material changes into the contract before you are dependent rather than after.
Monitoring that a validator will accept
Most teams monitoring a model in production are monitoring the service, not the model. Latency, error rate, throughput. All green while the thing quietly stops working.
Four signals cover the ground. Input drift, measured on the features that carry the most weight, using a stated statistic with a threshold set before launch rather than tuned after the first alarm. Output drift, meaning the score distribution and the approve or decline rate, which usually moves before anyone can measure accuracy. Realized performance, which requires the outcome join and arrives late in every domain worth modeling, sometimes ninety days late, and the documented lag is part of the record. And subgroup performance, computed on the same schedule as the aggregate, because a model can hold its overall number while degrading badly on one population.
Every threshold needs a named action attached. An alert with no owner and no defined response gets muted in a quarter, and a muted alert is worse than no alert, because the record now says you were watching.
Effort to satisfy each requirement · share of it that is engineering work
Editorial weighting from practitioner reading. Illustrative, not a measured statistic.
What this costs and how long it takes
For a company with somewhere between ten and forty models and no formal program, standing up the inventory, the tiering rule, the templates, and the pipeline instrumentation is a three to five month effort. The discovery and inventory phase runs two to four weeks. Instrumenting the training and serving pipelines to emit evidence automatically is where the engineering time goes, typically six to ten weeks depending on how many distinct stacks are in play. Writing the policy is a week and it is the least important week.
The recurring cost after that is what surprises people, in a good direction. Validation of a new model runs one to three weeks when the evidence is emitted by the pipeline, against four to eight weeks when it is assembled by hand. That gap is the return on the instrumentation work, and it repeats on every model and every material change for the life of the program.
The number worth watching is not documents produced. It is how long it takes to answer one question: which models are running right now, who owns each, when was each last validated, and what is each one's current monitoring status. A program that can answer that in an afternoon is working. A program that needs a week of asking around has produced paperwork, not control, and both an examiner and a serious customer will find that out in the first hour.
The mistakes we get called in to fix
The most common is a validation function that reviews documents rather than models. It is staffed by people who cannot run the training code, so the reviews check completeness against a template. Everything passes, nothing is challenged, and the program fails its first real examination on the effective challenge test specifically.
The second is tiering by technique. Neural networks get heavy scrutiny, linear models get light. A logistic regression that declines applications is higher risk than a deep model ranking internal search results, and a framework that says otherwise is spending its scarce review capacity in the wrong place. Tier on consequence: what happens to a person if the output is wrong, how reversible it is, whether a human reviews it, and whether the output leaves the building.
The third is retraining that lives outside change control. The model in production is not the model that was validated, nobody made a decision to change it, and the promotion happened on a schedule. Everything else in the program is undermined by this one gap, and closing it is usually two weeks of pipeline work.
The fourth is documentation with no reader. Thorough, template-complete, never opened after sign-off. The fix is to write the development record for the engineer who will inherit the system in two years, and let the validator read the same document. One artifact with a real audience beats two written for no one.
Where to start on Monday
Build the list. Not the policy, not the templates. Scan the deployment manifests and the schedulers, read the egress logs, run the amnesty, and put a name on every row. Almost every program that stalls, stalls because the list was never trustworthy and everything downstream inherited that.
Then take the single highest-consequence model on the list and drive one complete cycle end to end: purpose and boundary statement, evidence emitted by the pipeline, an independent review by someone who reruns the code, monitoring with thresholds and named owners, and a promotion gate on retraining. One model taken all the way through teaches the organization more than nine models taken a third of the way, and it produces a working template the next model inherits for a fraction of the cost.
Bottom line
SR 11-7 asks for three things: development you can defend, challenge that can actually stop a release, and governance that knows what is running. The letter's assumptions about fixed, slow-changing models are the part that strains against machine learning, and the resolution is to validate pipelines and gates rather than artifacts. Almost all of the real work is engineering, which is the good news, because engineering evidence emitted automatically by a build is cheaper, more accurate, and more convincing than a document assembled from memory after the fact.
Frequently asked questions
Not as a matter of law. It applies to institutions supervised by the Federal Reserve and the OCC. It reaches other companies through diligence: regulated customers, insurers, and acquirers ask for validation evidence written to its structure, and its vocabulary has been adopted by insurance and housing-finance regulators.
Anything that takes input data, applies a method with assumptions, and produces output used to make or support a business decision. Technique is irrelevant. A threshold in a SQL view that routes cases to review qualifies, and it is often the highest-risk row on an inventory because it has no owner and no test.
Not for anything consequential. Independence is about the reporting line and the record, not headcount. A qualified reviewer from another team or an outside party works, provided their findings enter the record unedited by the model owner and can delay a release.
Validate the pipeline. The reviewer approves the training code, data contract, acceptance thresholds, and promotion gate once. Each retrain is then checked automatically against those thresholds, and full revalidation triggers on a material change such as new features, a new population, or drift beyond a documented band.
Yes. The guidance is explicit that using a vendor product does not transfer the risk. Pin the version where the contract allows, keep a fixed evaluation set you rerun on a schedule to detect silent changes, and negotiate notice of material changes into the agreement.