Building an AI feature a federal auditor will accept

The question an assessor actually asks

An assessor sitting down with your AI feature does not ask whether the model is good. That question has no answer they can write into a finding. They ask a narrower and much harder one: show me the record of what this system decided, on what input, using which model version, under whose authority, and show me that the record was produced by the system rather than assembled by a person after the fact. Almost every team that fails an AI assessment fails on that last clause. They have policies and a governance committee with minutes. What they cannot produce is a machine-generated trail linking a specific output to a specific model version, a specific input, and the human accountable for the decision at that moment.

This gap has a build shape, not a compliance shape. Logging that captures a request and a response but not the model artifact identifier is useless six months later, when three fine-tunes have shipped and nobody can say which one produced the output under review. An evaluation report stored in a slide deck is not evidence, because nobody can rerun it. A prompt kept in application source and edited by whoever was on call is a configuration change to a decision system, made without review, and an assessor treats it exactly that way.

The work is bounded, mostly mechanical, and far cheaper before the feature ships than after. What follows is what the evidence looks like, which frameworks drive which requirements, and what the two paths cost.

Evidence is a build artifact, not a document

The test that separates a system that passes from one that does not is whether the artifact was produced by the running system or written by a person describing the running system. A person can describe an access control policy. Only the system can produce the log line proving the policy was enforced at 04:17 on a Tuesday against a service account that should not have had the token.

Six artifacts carry most of the weight, and every one of them is a code change rather than a policy change. First, an immutable inference record: input hash, output, model artifact identifier, prompt or configuration version, retrieval index build date, latency, and the identity of the calling principal. Second, a model registry entry per artifact, with training data lineage, evaluation results, and an approval record tied to a named person. Third, evaluation runs stored as data rather than as reports, so any of them can be rerun against a newer artifact and compared. Fourth, an override and escalation log capturing every case where a human changed or rejected the machine output, which is the record regulators care most about and the one almost nobody builds. Fifth, a change history for prompts and thresholds under the same review discipline as application code. Sixth, drift monitoring that emits a time series, not an alert that fires and disappears.

None of that is exotic engineering. All of it is roughly free at design time and expensive to add later, because retrofitting an inference record into a system already in production means backfilling a period you cannot backfill, and the assessor will note the gap.

Which framework is actually driving your review

Directors often ask which standard to build to, as though one governs. Usually two or three apply at once, and they ask for different things. Sorting out which one your reviewer works from is the cheapest hour you will spend on the whole program.

NIST AI RMF 1.0 is the common vocabulary in federal work. It is voluntary, and it is organized as four functions with subcategories, which matters because agencies write those subcategory numbers into requirements. Its Govern function asks who owns the risk and who approves use. Map asks what the system is, who uses it, what data feeds it. Measure asks for tested performance including on subpopulations. Manage asks what happens when the measurement goes bad. The accompanying Playbook gives suggested actions per subcategory, and mapping your artifacts to those numbers is the difference between a reviewer recognizing your evidence and a reviewer doing translation work they will not do.

ISO/IEC 42001:2023 is the certifiable one. When a large integrator or a government customer asks for a management system rather than a set of artifacts, this is what they mean. It is auditable by an accredited body and takes most organizations nine to eighteen months from a standing start. Worth it when a contract or a procurement questionnaire names it, and rarely otherwise.

SR 11-7 governs model risk management at supervised financial institutions and is the oldest and most demanding of the set. Written in 2011 for credit and capital models, it applies cleanly to any model driving a decision, and its central demand is independent validation: somebody other than the developers tests the model, reviews conceptual soundness, and signs a document. If your buyer is a bank, an insurer, or a ratings firm, this is the governing text whatever the AI-specific standards say.

NIST SP 800-53 Rev. 5 and SP 800-171 Rev. 3 govern the surrounding system, not the model. FedRAMP applies if the thing is a cloud service sold to agencies. OWASP LLM Top 10 and MITRE ATLAS are threat catalogs rather than control frameworks, and their value is that they give a security reviewer a shared name for the attacks your architecture has to answer: prompt injection, training data poisoning, model extraction, insecure output handling.

The failure that costs the most: no output-to-artifact binding

Here is the single most expensive gap, and it appears in maybe three quarters of first reviews. An output exists. A log of it exists. And there is no field anywhere linking that output to the exact model artifact that produced it.

The mechanism is ordinary. A team deploys behind an endpoint name. The endpoint is updated. The log records the endpoint name, because that is what the SDK returns. Four months later somebody disputes a decision, and the question is which weights were live at 09:42 on March 14. The deployment history sits in a CI system with a 90-day retention policy. The answer is gone.

The fix is one field and a discipline. Log the content hash of the model artifact on every inference, alongside the prompt version, the retrieval index build identifier, and the configuration hash. For a hosted third-party model where you cannot hash weights, log the provider's full versioned model string plus the date, and record separately that the provider controls that binding, which an assessor would rather see disclosed than discover. That is about a day of engineering against a remediation project reconstructing four months of history from logs designed for something else.

Human oversight has to be a feature, not an assurance

Every AI governance framework requires meaningful human oversight, and nearly every implementation of it is a sentence in a document saying a person reviews the output. That sentence produces no evidence.

Oversight becomes auditable when it becomes a state machine. The output enters a review state. A named identity acts on it: accept, modify, reject, escalate. That action, that identity, and the timestamp are recorded next to the original output. The record is queryable. Now three questions have answers: how often do humans overrule the model, on which kinds of case, and is the override rate trending in a direction that means the model has drifted or the reviewers have stopped reading.

That last measurement is the one that turns oversight from a compliance line item into something operationally useful. An override rate falling from 12% to 2% over six months might mean the model improved. It usually means reviewers learned that clicking accept is faster. A system emitting that time series catches the problem. A system with a policy paragraph does not, and the first anyone hears of it is when a bad decision reaches a customer.

Evaluation the buyer can rerun

An evaluation stored as a PDF is an assertion. An evaluation stored as a dataset, a scoring script, and a result record is evidence, because a third party can run it again and get the same number.

The engineering is modest. Keep the evaluation set versioned and separate from anything the model trained on, with a manifest of item identifiers and checksums so both sides can prove they scored the same items. Keep the scoring code in the repository and delivered, since metric definitions differ across libraries and an averaging choice alone can move a reported number by several points. Store per-item outputs, not summary statistics, because without them no dispute is adjudicable and no subgroup can be recomputed when somebody asks a question you did not anticipate. Run it in CI against every candidate artifact so results accumulate as a series rather than a one-time exercise.

Subgroup results deserve their own line. An aggregate number is a weighted average over whatever the evaluation set contained. A system at 93% overall can sit at 64% on the case type that makes up 5% of volume and carries most of the risk. Name the slices in advance, give each its own floor, and size the set so each slice carries enough items to support a real measurement. If your application touches individuals in a way that implicates nondiscrimination law, those slices are not optional and a reviewer will ask for them by name.

Threat evidence: what your security reviewer wants

The security half of an AI review runs on different questions than the governance half, and a team that has done careful model documentation is often caught flat here.

Prompt injection is the one that generates findings, and the answer is architectural rather than textual. Instructions telling the model to ignore injected instructions are not a control. Controls are: the model has no credentials of its own and every tool call runs under the end user's authorization; tool calls are allowlisted with typed parameters rather than assembled from free text; output going into a downstream system is validated against a schema before it moves; retrieved content is marked as data at the boundary and never concatenated into a system instruction. Each of those is testable, and a test that runs in CI is evidence.

Beyond injection, expect questions on training data poisoning if you fine-tune on anything user-supplied, on model extraction if your endpoint is externally reachable, on insecure output handling wherever generated content reaches a shell, a query, or a browser, and on the supply chain for the weights themselves. MITRE ATLAS gives that last one a vocabulary. Answering all of it with an architecture description and a set of CI tests takes a competent team two to four weeks.

The two cost paths, in real numbers

The reason to do this at build time is not virtue. It is arithmetic, and the arithmetic is lopsided enough that it usually ends the discussion.

Work	Built in during development	Retrofitted after a finding
Inference record with artifact binding	1 to 3 engineer-days	2 to 6 weeks, plus an unfillable historical gap
Model registry and approval records	1 to 2 weeks	4 to 10 weeks reconstructing lineage for shipped models
Rerunnable evaluation harness	2 to 3 weeks	6 to 12 weeks, often including relabeling a test set
Human override state machine	2 to 4 weeks	A product change with UX, training, and a rollout
Injection and output-handling controls	2 to 4 weeks	Architecture change under deadline, plus retest
Typical all-in	Roughly 8 to 14 percent of feature cost	$250K to $1.5M plus two to three quarters of delay

The retrofit ranges are wide because they depend almost entirely on one thing: whether the system was logging enough to reconstruct history. A team that logged inputs, outputs, and artifact identifiers from day one but never organized them into evidence is doing a reporting project. A team that logged only requests and responses is doing an archaeology project with a hole in the middle that no amount of budget closes.

Sequencing it into a build already underway

Most directors reading this are not starting a feature. They are eight months into one, and the question is what to do on Monday.

The order that recovers the most ground per week starts with the logging, because it is the only item where every day of delay creates permanently unrecoverable evidence. Ship the inference record with artifact binding first, even if nothing consumes it yet. Second, put prompts, thresholds, and configuration into version control behind the same review that guards application code, which takes a day and closes a finding class outright. Third, stand up the model registry and backfill what you can honestly reconstruct, marking what you cannot rather than guessing. Fourth, turn the newest evaluation into a rerunnable harness in CI. Fifth, build the override state machine, which is last because it is the only one that touches the product surface and needs design work.

Two things not to do. Do not commission a governance policy before the logging exists, because it will describe controls the system does not enforce, and an assessor comparing the two finds that worse than silence. And do not chase a certification no customer has asked for.

What this changes about how you scope the work

The practical shift is that AI assurance stops being a phase and becomes a set of acceptance criteria on the feature itself. The feature is not done when the model hits its metric. It is done when the model hits its metric and the system emits the record proving it, on data somebody other than the builder selected, with a human decision path that leaves a trail.

Written that way, the requirement is enforceable by an engineering manager on a Thursday rather than by a governance committee at the end of a quarter. That is the whole trick. Teams that get through assessment cheaply converted the compliance language into build tickets early. Teams that do not spend the same effort later, under deadline, with less design freedom and a reviewer already holding a finding.

Bottom line

A federal auditor accepts an AI feature when the system itself produces the evidence: outputs bound to model artifacts, evaluations anyone can rerun, human decisions recorded as data, configuration under change control, and threat controls that exist in the architecture rather than in a paragraph. That work costs a low double-digit percentage of the feature when it is designed in and a quarter-million dollars and two quarters when it is not. The decision in front of a director is not whether to do it. It is whether to do it while the design is still open.

Frequently asked questions