A program office does not sign off on a model. It signs off on an argument. Somewhere in the review chain a person with their name on the decision has to be persuaded that this component, in this system, doing this task, behaves acceptably across the conditions it will meet, and that when it does not, something catches it. On a conventional component that argument is usually implicit, carried by the verification matrix and a long institutional memory of how such parts behave. On an AI component there is no institutional memory, the verification matrix does not close on its own, and the argument has to be written down.
This is written for the chief engineer or safety lead at a prime who is fielding a system with a learned component in it, and who has worked out that the usual package will not carry the decision. The good news is that the discipline for this already exists. Safety-critical engineering has spent decades building structured arguments with evidence, and the structure transfers cleanly. What changes is the evidence.
What an assurance case is, in one section
An assurance case is a claim, an argument, and evidence. The top claim is a sentence about the system in its context: this component performs its function acceptably within a defined operating envelope, with defined residual risk. That claim decomposes into sub-claims. Each sub-claim is supported either by further sub-claims or by evidence: a test result, an analysis, a design feature, or a process record.
Three properties make it different from a document that merely describes the system. The argument is explicit, so a reviewer can see why the evidence supports the claim rather than inferring it. Every claim is bounded by a stated context, so nobody is claiming more than was tested. And the case names its own weaknesses: the assumptions it rests on and the residual risk that remains, which is what makes the sign-off meaningful rather than ceremonial.
The case is also the thing that survives the program. When the model is retrained in year three, the question is not "does it still work" but "which branches of the argument does this change affect, and what evidence has to be regenerated". A program with a written case answers that in a week. A program without one re-argues everything from scratch.
The five branches that carry the weight
Across the AI systems our engineers have delivered into production, the same five sub-claims do most of the work. They are not the only decomposition, but they are the one that reviewers follow most easily.
One: the envelope is defined and detected. The claim is that the conditions under which the component is trusted are written down, and that the system can tell when it is outside them. This is the branch programs most often skip, and it is the one a reviewer probes first, because a performance claim without a stated envelope is a claim about nothing. The evidence is a specification of the operating conditions, an analysis showing the test data covered them, and a designed mechanism, such as an input distribution check, a confidence measure or an explicit out-of-scope classifier, that detects when the system is outside and does something defined about it.
Two: performance inside the envelope is measured, not asserted. The claim is that within the stated conditions, the component meets the thresholds the mission requires. The evidence is the test campaign: metrics tied to the operator's decision, a decision threshold justified by the cost of a miss against the cost of a false alarm, results broken out by condition with adequate sample counts per bin, and a stated label quality that bounds the top of the claim. A single pooled figure supports nothing.
Three: degradation and manipulation are bounded. The claim is that behavior under realistic degradation and under the manipulations an adversary can actually perform stays inside acceptable limits or is detected. The evidence is a stated threat model matched to the system's real exposure, testing against natural degradations, testing against physically realizable manipulations, and provenance controls over training data, weights and dependencies with a demonstrated ability to rebuild from source.
Four: human oversight is real, and designed. The claim is that a person is positioned to catch the failures the component will have, with the information and the time to do it. This is the branch that most often fails a review honestly, because oversight is usually asserted rather than designed. Evidence is interface design showing what the operator sees and when, measured operator performance rather than assumed vigilance, workload analysis at the actual alert rate, and a defined authority for override.
Five: the fallback is engineered and exercised. The claim is that when the component is unavailable, out of envelope, or detected as degraded, the system does something safe and useful. Evidence is the designed degraded mode, its own acceptance criteria, and test results showing the transition works — including the transition back, which is where most fallback designs are weakest.
Where reviewers push hardest on an AI assurance case
Editorial weighting, illustrative rather than measured. The last row is deliberately low: novelty is not an argument a reviewer can sign.
Where the argument usually breaks
Reviewing draft cases, the same three defects appear.
The claim is unbounded. "The component classifies contacts with high accuracy" is not a claim; it is a sentiment. A claim has a subject, a bound, a condition set and a threshold: this component, at this decision threshold, on this task, across these enumerated conditions, meets these per-condition floors. Reviewers who read a lot of cases spot the unbounded version immediately, and the rest of the document loses credibility with it.
Evidence is attached without an argument. A folder of test reports is not a case. The reviewer's question is always "why does this evidence support that claim", and if the document does not answer it, the reviewer constructs their own answer, which is usually less favorable than the one the team would have written. Every piece of evidence needs a sentence saying what claim it supports and what it does not.
Assumptions are hidden. Every case rests on things not tested: that the sensor stays calibrated, that operators are trained to the assumed standard, that the input format does not change, that the deployment environment matches the test environment. A case that lists these and attaches a monitoring or process control to each one is stronger than a case that pretends they do not exist, because the reviewer knows they exist and is watching to see whether the team does.
Two shapes of argument, compared
Programs generally choose between arguing from the component's properties and arguing from the system's containment of it. The choice drives what evidence is expensive.
| Dimension | Argue from component performance | Argue from system containment |
|---|---|---|
| Top claim | The model is accurate enough that its errors are tolerable | The system limits the consequence of any model error to an acceptable level |
| Where the evidence goes | Test campaign volume, condition coverage, statistical confidence | Architecture, interlocks, monitors, human authority, degraded modes |
| Cost driver | Data collection and labeling for the rare and severe cases | Systems engineering and integration test of the containment |
| What breaks it | A condition nobody tested, or drift after fielding | A failure path the containment did not anticipate |
| Best fit | Advisory outputs, high alert rates, reversible consequences | Consequential or fast-acting decisions where errors are expensive |
| What to do | Use both. Containment carries the severe cases; performance carries the ordinary ones | Use both. State which branch carries which hazard, explicitly |
The practical answer for almost every fielded system is a hybrid, and the value of writing it down is that the hybrid forces a decision the program would otherwise leave implicit: for each identified hazard, which branch is carrying it. When that mapping is written, gaps become visible. A hazard carried by neither branch is the finding a review is meant to produce, and it is far cheaper to find in a document than in an operational assessment.
Connecting to the safety case and the security case
A program that already has a safety case or a system security package does not need a separate parallel document, and producing one is a mistake. The AI assurance argument is a set of branches that plug into the arguments already there.
On the safety side, the hazard analysis already exists. The AI component adds new causal paths to existing hazards and occasionally a new hazard. The correct move is to run the analysis with the component in it, treating the model as a source of specific failure types (a confident wrong output, an output outside the expected range, a silently degraded output, an unavailable output, an output with unusual latency) and to trace each to the hazards it can contribute to. The mitigations that result become branches of the assurance case, and the case references the hazard analysis instead of restating it.
On the security side, the model and its data are assets in the system's authorization boundary and are covered by the control set the program already applies. What the AI component adds are a handful of control conversations that the standard package handles thinly: provenance of training data and model weights, integrity of the retraining path, protection of the model artifact and any embedded data, logging sufficient to reconstruct a decision after the fact, and treatment of any external content the model consumes as untrusted input rather than instruction. Write those as specific control implementations with evidence, and the assurance case references them.
The result is one argument with three views, not three arguments. That matters practically: when a change occurs, the program traces it once.
What a program office gets back from writing the case early
Editorial weighting, illustrative rather than measured. The last row is deliberately low: a reviewer signs evidence about this system, not literature.
Evidence a specialist partner produces
The technical work behind an assurance case is largely evidence generation, and it is specific enough to scope and price. What our engineers deliver into the case falls into five packages.
Envelope specification and detection. A written statement of the operating conditions with quantitative bounds, an analysis of how well the test data covers them, and an implemented mechanism that flags inputs outside the envelope at run time, with its own measured false-positive and false-negative rates. That detector is itself a component and gets its own small evidence set.
The evaluation package. The metric chain from mission task to threshold, the split rule and its rationale, the per-condition results with confidence intervals and sample counts, the measured label agreement rate, and a reproducible pipeline that regenerates every number in the report from recorded inputs. Reproducibility is what converts a report into evidence.
The stress and manipulation package. Results under natural degradation, results under realizable manipulation with the threat model stated, and the provenance record for data, weights and dependencies with a demonstrated rebuild.
The oversight package. The interface design decisions with their rationale, the alert rate the operator actually experiences, measured override behavior where a trial is possible, and the authority model showing who can do what.
The monitoring and change package. The fielded monitors and what they measure, the trigger table mapping indicator values to actions and decision authorities, and the change process showing which parts of the case a given kind of change invalidates. This last artifact is what keeps the case alive after fielding, and it is the one most programs discover they need only after the first retraining.
How we work inside a prime's program
Precision Federal builds AI, data platforms and software and delivers them into production inside federal agencies. On a program fielding an AI component we come in as the specialist subcontractor on the model, the evaluation and the assurance evidence, while the prime holds the system, the safety and security cases, the customer relationship and the integration.
The first four weeks. We read the requirements, the hazard analysis and the security package, and produce a draft case structure: the top claim with its context, the branch decomposition, the hazard-to-branch mapping, and an evidence register listing what exists, what is planned and what is missing with an estimate for each. Programs consistently find the third column the most useful thing they get, because it converts an open-ended worry into a costed list.
Weeks five through sixteen. We generate the evidence packages, in the prime's environment and repositories, and write them into the case in the program's document standard. Where the evidence disagrees with the desired claim, we say so early and propose the design change or the narrowed claim, because a case that overstates is discovered at the review with no time left to fix it.
What the prime keeps. All of it. The case document, the evidence, the models, the evaluation pipeline, the detectors, the monitoring code, and the customer relationship. Intellectual property in the delivered work is assigned by a present written assignment; any pre-existing tooling we bring is named, carved out and licensed back perpetually so the sustainment team is never blocked. The exit package (repository, pipeline, infrastructure as code, configuration and runbook) is a condition of final payment, with a handover rehearsal where the receiving team regenerates the evidence while our engineers watch. On a scored proposal we are content to be named with a defined technical scope and committed key personnel, which generally reads better to an evaluator than an unnamed pool.
How it is priced. The case structure and evidence register works as a firm-fixed-price first milestone, since its output is defined. The evidence generation works as fixed-price milestones against written acceptance criteria, which here are measured thresholds on named datasets and named artifacts delivered into the case, or as a committed team at a fixed monthly rate for programs where the review cycle sets the pace. Starting with a small fixed-price increment is how we prefer a new teaming relationship to begin.
Five things that make a case sign-able
Write the top claim first and make it bounded. Everything else is decomposition. A vague top claim produces a vague case however good the testing was.
Map every hazard to a branch, in a table. The unmapped hazard is the finding. Better the team finds it.
State the residual risk in the team's own words. A case that names what it does not cover is trusted on what it does. A case that claims complete coverage invites a search for the gap.
Make the evidence reproducible. A number nobody can regenerate is an assertion. A pipeline that regenerates the whole report from recorded inputs is evidence, and it also makes the next review cheap.
Design the change process into the case. Say which branches a model update, a data refresh, a threshold change or an environment change invalidates. Without it, the case is a snapshot, and the first improvement after fielding will be blocked by nobody knowing what it costs to re-argue.
Bottom line
Fielding AI in a defense system is an argument problem before it is a modeling problem. The argument has five branches that carry most of the weight: a stated operating envelope with run-time detection, measured performance per condition inside it, bounded behavior under degradation and realizable manipulation, human oversight that is designed and measured rather than assumed, and an engineered fallback whose transitions have been tested. Plug those branches into the hazard analysis and the security package the program already has, name the assumptions and the residual risk, and make every number reproducible from a recorded pipeline. That is a document a chief engineer can sign and a program can maintain. The first step is one email with a one-page brief describing the system, the component and the review it faces, and we return a scoped, priced statement of work.
Frequently asked questions
A structured argument that a component behaves acceptably in a defined context, supported by evidence. It has a bounded top claim, a decomposition into sub-claims, and evidence attached to each leaf with a stated reason why it supports the claim. It also names the assumptions it rests on and the residual risk that remains. That last part is what makes a sign-off meaningful, and it is what distinguishes a case from a folder of test reports.
A written operating envelope with a run-time mechanism that detects when the system is outside it. Performance measured per operating condition with adequate sample counts, a justified decision threshold and a stated label quality. Results under natural degradation and under manipulations an adversary could realistically perform, with the threat model written down. A designed and measured human oversight arrangement. An engineered fallback with tested transitions. And monitoring that continues producing evidence after fielding.
No, and producing a parallel document is a mistake. Run the existing hazard analysis with the component in it, treating the model as a source of specific failure types: confident wrong output, out-of-range output, silently degraded output, unavailable output, unusual latency. Trace each to the hazards it can contribute to, and let the resulting mitigations become branches of one argument that references the existing hazard analysis and security package rather than restating them.
With two branches, and by saying which hazard each one carries. One argues from component performance: errors are rare enough and mild enough at the chosen operating point to be tolerable. The other argues from system containment: whatever the model does, the architecture, interlocks, human authority and degraded modes limit the consequence. Severe and fast-acting hazards should be carried by containment, ordinary ones by performance, and the mapping should be an explicit table.
Only the affected branches are re-argued, provided the case says which those are. Build a change process into the case that maps kinds of change (a model update, a data refresh, a threshold adjustment, an environment change) to the branches each invalidates and the evidence that must be regenerated. With a reproducible evaluation pipeline, regenerating that evidence is largely automated. Without this, every improvement after fielding reopens the whole argument.