A test plan written for software assumes that the same input produces the same output, that a passing test stays passed, and that the thing under test does not change unless someone changes it. An AI component violates all three. The output distribution shifts with the input distribution. A model that passed last quarter's regression suite can fail this quarter's because the world moved, not because the code did. And the failure modes that matter to a program office are not crashes; they are confident wrong answers inside the operating envelope. The test director who treats a model as one more configuration item will produce a test report the independent evaluator does not accept, and will find that out late.
This is written for the person on a prime's side who owns test and evaluation on a program that now has a model in it: a test director, a chief engineer, or the systems engineering lead who has to sign the verification cross-reference matrix. The question is not whether AI can be tested. It is what the test approach has to look like so that the program office, the operational test authority, and an independent evaluator all agree the evidence means something.
Why the standard verification approach does not close
Verification on a defense program works by decomposition. A requirement is written, allocated to a component, and verified by inspection, analysis, demonstration or test. The verification event is repeatable, and the evidence is a pass or fail against a threshold. That machinery is sound and should not be thrown away. It breaks in four specific places when the component is a learned function.
The requirement is written against a task, not a metric. An operator requirement says the system shall detect and classify contacts of interest within a stated time. A model reports precision and recall on a held-out set. Nobody has connected the two, so the test report shows a number the operational community cannot interpret and the requirement stays unverified. The connection has to be built deliberately: a mission task, a scenario set that exercises it, and a metric whose value maps to whether the task succeeded.
The test set is a piece of the design, not a fixture. On conventional software the test data is an input chosen by testers. On an AI component, the data that trained the model and the data that tests it come from the same collection process, and if nobody governed the split, the test result is a measurement of memorization. Independent evaluators know this and will ask how the split was made. A test approach that cannot answer precisely has already lost the argument.
Performance is a distribution, not a point. A single accuracy number over a mixed test set hides the cases that decide the program. Performance at night, at range, against a rare class, in the sensor mode the operator actually uses. The program office cares about the worst relevant slice, not the average. A test plan that reports one number invites the question it cannot answer.
The article under test keeps changing. Retraining, a data refresh, a threshold adjustment, a swap in the preprocessing chain — each produces a different function with the same part number. Configuration control for models is not the same problem as configuration control for code, and a program that does not solve it will re-run the whole campaign every time somebody improves something.
What evidence carries weight with an independent evaluator
Editorial weighting, illustrative rather than measured. The last row is deliberately low: a public benchmark says nothing about this program's data.
Building the metric chain from the mission task down
The single most useful thing a test approach can contain is an explicit chain from an operational task to a number a model produces. Our engineers build it in four links, and we build it before any model work starts, because the chain determines what data has to be collected.
Link one: the task and its decision. Write the operator's decision as a sentence. The watch stander decides whether to cue a sensor. The maintainer decides whether to pull a part. The analyst decides whether a document is responsive. Every AI component supports a decision, and the decision has a cost structure: what a miss costs, what a false alarm costs, and how long the operator has.
Link two: the operating point. That cost structure sets the threshold. A system where a miss is catastrophic and a false alarm is a few seconds of operator attention runs at a very different point on the curve than one where each alert triggers an expensive dispatch. Fix the operating point in the requirement, not in the model. Then the model's job is to move the whole curve, and the test measures performance at the point the program actually uses.
Link three: the slices. Enumerate the conditions under which the task must succeed and make each one a reporting bin with its own minimum sample count. Sensor mode, time of day, weather, range band, target class, platform variant, operator workload. This is where the test plan does its real work, because a slice with eleven examples in it cannot support a claim, and finding that out during the campaign is much worse than finding it out during planning.
Link four: the aggregate the program office signs. The verification cross-reference matrix needs a row that can be marked complete. Write it as a per-slice floor plus an aggregate, so that a strong average cannot conceal a failed condition. That sentence is the whole reason the chain exists.
Test data governance is the part that gets programs in trouble
On every AI program we have engineered, the hardest technical argument was not about the model. It was about the data. Three questions decide whether the evidence survives review.
How was the split made? Random splitting is almost always wrong on operational data. If frames from the same collection event land on both sides, the test measures how well the model remembers that afternoon. Split by the unit that matters: mission, sortie, platform, site, subject, document family. Then state the rule in the test plan, and hold a portion out under change control so no one can tune against it. We keep a sealed set that the development team cannot read, and we open it once, at the milestone, with the government present if the program allows.
Where did the labels come from and how good are they? Ground truth is manufactured, and it has an error rate. A model measured at ninety-four percent against labels that are themselves ninety-one percent consistent is telling you almost nothing about the top of the range. Measure label agreement on a sample, report it, and set the ceiling of the claim accordingly. This one number changes how an evaluator reads the entire report, and volunteering it buys credibility that no amount of test volume does.
Is the test set representative of the deployed condition? Almost never, at first. The collection was done where collection was convenient. Write down the difference between the test population and the fielded population, and either collect against the gap or state it as a limitation with a monitoring requirement attached. An evaluator who finds that gap themselves will discount the whole campaign. An evaluator handed it in the plan will treat the rest of the numbers as honest.
Stress and adversarial testing, scoped to what the program can act on
Adversarial testing on a defense program is often either skipped or done as theater. Both are avoidable. The useful version starts from the threat the program's security posture already assumes and asks what an adversary who has that access can do to the model.
We structure it in three tiers. The first is natural degradation: degradations that occur without an adversary. Sensor noise, compression artifacts, weather, occlusion, calibration drift, a firmware update that changes the input format slightly. These are cheap to generate and they find real problems. A model whose accuracy collapses under mild blur has learned something about the collection rig rather than the target.
The second is input manipulation: perturbations designed to cause a wrong answer. What matters here is realizability. A perturbation that requires per-pixel control of the sensor feed is not a threat if the adversary cannot reach the sensor feed. A pattern that can be painted on a surface, or a document crafted to be ingested, is. Test the realizable ones, state the threat model in one paragraph, and stop. The test report is stronger for having a bounded threat model than for having a long list of academic attacks.
The third is supply and data path integrity: what happens if training data, a pretrained weight file, or a preprocessing dependency is tampered with. This is largely a process and provenance question rather than a testing question, and it belongs in the test plan as an inspection item: hash and record every artifact, know where the weights came from, and be able to rebuild the model from source data.
For systems built on large language models the same three tiers hold, with prompt injection and retrieval poisoning as the input-manipulation cases. The engineering answer is the same as everywhere else: treat retrieved and user-supplied text as data rather than instruction, constrain the tool surface the model can reach, and test the constraint rather than trusting the prompt.
Two ways to run the campaign
The choice below is usually made implicitly by the schedule. Making it explicitly is worth a meeting.
| Dimension | Model treated as a fixed configuration item | Model treated as a continuously evaluated component |
|---|---|---|
| What is baselined | One weight file, one preprocessing chain, one threshold | The pipeline, the data governance, and the evaluation rig |
| Cost of an improvement | A regression campaign, often the full one | An automated evaluation run plus a delta review |
| How drift is caught | By a field report, after operators lose trust | By monitoring against the same metrics used at test |
| What the program office reviews | A test report at a milestone | A standing report, plus milestone reports drawn from it |
| Where it fits | Stable task, rare updates, tight accreditation boundary | Evolving data, planned retraining, option years for sustainment |
| Typical failure | The baseline goes stale and nobody may update it | Evaluation infrastructure is treated as overhead and underfunded |
Most programs need the second shape with the first shape's discipline layered on top: a frozen, signed baseline for each fielded release, produced automatically by a pipeline that runs on every candidate. That combination is what lets a program improve a model between milestones without reopening the entire verification argument.
The evaluation rig is a deliverable, not a tool
The most valuable artifact we hand a prime on an AI program is usually not the model. It is the rig that produces the numbers. Treated properly it is a configuration-managed deliverable with its own requirements, and the program can keep using it for the life of the system.
What it contains: a versioned dataset registry that resolves a named test set to exact content, with hashes; the slice definitions as code so that a reported bin cannot silently change meaning; the metric implementations, including the confidence intervals, so nobody argues about whether a two-point difference is real; scenario runners that drive the component the way the system does, through the same interfaces, rather than in a notebook; and a report generator that emits the same tables into the test report, the program review deck, and the monitoring dashboard, from one computation.
That last point matters more than it sounds. When the milestone report, the sustainment dashboard and the developer's local run are three different computations, they disagree, and the program spends its credibility explaining the disagreement instead of discussing the system. One computation, three renderings.
The rig also settles the argument about repeatability. Nondeterminism in model training and in some inference paths is real, and an evaluator is right to ask about it. The answer is to make the evaluation deterministic even when training is not: fixed seeds where they apply, pinned dependency versions, recorded hardware and runtime, and a stated tolerance band derived from repeated runs rather than assumed. Then a difference outside the band means something, and a difference inside it does not.
Continuous evaluation after fielding
The campaign does not end at the milestone, and the program office increasingly knows it. Monitoring for an AI component has three layers, and the mistake is to build only the first.
Input monitoring. Track the distribution of what the model actually sees against the distribution it was tested on. This catches the sensor that was recalibrated, the new platform variant, the site whose lighting differs, the document format that changed. It requires no labels, which is why it is the layer that works from day one.
Output and behavior monitoring. Score distributions, rate of operations near the decision threshold, abstention rate, latency, and the rate at which the operator overrides. Operator override is the cheapest label a fielded system produces and most programs throw it away. Capture it deliberately, with a reason code, and it becomes both a monitoring signal and a source of the next training set.
Periodic truthed evaluation. On a stated cadence, pull a sample of fielded data, have it labeled under the same rules used at test, and run the rig. This is the only layer that produces a defensible number after fielding, and it needs to be in the contract with hours attached or it will not happen.
Tie all three to a written trigger table: what value of what indicator causes what action, and who decides. A drift alarm with no pre-agreed response produces a meeting. A drift alarm tied to "retrain from the governed pool, re-run the rig, present the delta at the monthly review, re-baseline if it passes" produces a system that stays trustworthy.
Where a program's test effort pays back most
Editorial weighting, illustrative rather than measured. The last row is deliberately low: volume of attacks is not the same as a stated threat model.
How we work inside a prime's program
Precision Federal builds AI, data platforms and software and delivers them into production inside federal agencies. On a defense program we come in as the specialist subcontractor on the model and evaluation scope while the prime holds the system, the customer relationship and the integration. That division works because our scope is defined by artifacts rather than by attendance.
The first four weeks. We read the requirements and the draft test plan, write the metric chain from mission task to threshold, inventory the data with its provenance and label quality, and stand up the evaluation rig against whatever model exists today, even a placeholder. The output is a written test approach with the slice table, the split rule, the sample-count math per bin, and a list of the data gaps that need collection. That document is the thing the program office and the independent evaluator argue with, and having it early is worth more than any model result.
Weeks five through twelve. The rig runs in the prime's environment on the prime's infrastructure, wired into the program's pipeline. We produce the first honest baseline, including the slices that fail, and the tier-one degradation results. We write the CDRL-ready test reports in the program's format, not ours.
What the prime keeps. All of it. Source code, the rig, the data pipelines, the trained artifacts, the documentation, and the customer relationship. We assign the intellectual property in the work under a present written assignment, carve out and license back any pre-existing tooling we bring so the prime's sustainment team is never blocked, and deliver an exit package as a condition of final payment: repository, build pipeline, infrastructure as code, environment configuration and a runbook. We are visible or not as the prime chooses, though on a scored proposal a named specialist with a real technical scope and committed key personnel generally reads better to an evaluator than an unnamed pool.
How it is priced. Two shapes. Fixed-price milestones against written acceptance criteria, where the criteria are measured thresholds on named datasets rather than adjectives, which suits a defined test-approach and rig scope. Or a committed team at a fixed monthly rate for a stated period, which suits sustainment and continuous evaluation where the work is real but the exact sequence is not knowable in advance. We are comfortable with either, and with a small firm-fixed-price first increment as a way for a new teaming relationship to prove itself cheaply.
Five failure modes worth naming in the plan
Reporting one number. A pooled accuracy figure is the fastest way to lose an evaluator's confidence, because the first question is always about a condition, and the pooled number cannot answer it.
Letting the developer choose the test set. Not because anyone is dishonest, but because tuning against a set is what development is. The sealed set exists so the last measurement is the honest one.
Skipping label quality. Every claim above the label agreement rate is unsupported, and the evaluator will compute it eventually.
Testing the model instead of the system. The fielded thing includes preprocessing, thresholds, the human interface and the operator's workflow. A model that scores well and an interface that presents its output badly produce a failed operational assessment.
Funding no monitoring. A program that fields a model with no continuous evaluation has committed to discovering degradation from a complaint. That is the most expensive way to find out and the hardest to explain.
Bottom line
Test and evaluation for an AI component is not a new discipline bolted onto a program. It is the existing discipline applied to a component whose behavior depends on data, which forces three additions: an explicit chain from mission task to metric and threshold, governed test data with a stated split rule and a measured label quality, and an evaluation rig that produces the milestone report and the fielded monitoring from one computation. Build those three early and the program can improve its model without reopening its verification argument. Build them late and every model change costs a campaign. The first step is one email with a one-page brief describing the system, the mission task and the data, and we return a scoped, priced statement of work.
Frequently asked questions
Write them against the operational decision the component supports, not against a generic accuracy figure. State the mission task, the cost of a miss versus a false alarm, the resulting decision threshold, and the operating conditions that must each be met. Then express the requirement as a per-condition floor plus an aggregate, so a strong average cannot conceal a failed condition. That form gives the verification matrix a row that can be marked complete and gives the evaluator something interpretable.
Split by the unit that creates correlation, not at random. Mission, sortie, platform, site, subject or document family, depending on how the data was collected. Random splitting of operational data usually puts near-duplicate records on both sides and measures memorization. State the split rule in the test plan, and keep a sealed portion under change control that the development team cannot read, opened once at the milestone. Also measure and report label agreement, because no claim can exceed the quality of the ground truth.
Start from the access an adversary realistically has and test what that access permits. Natural degradations first, since sensor noise, weather, occlusion and format changes find real problems cheaply. Then physically realizable input manipulations, with the threat model stated in a paragraph. Then provenance and integrity of training data, weights and dependencies, handled as an inspection item with hashes and a rebuild capability. A bounded threat model tested thoroughly is stronger evidence than a long list of attacks that could not occur.
Baseline the pipeline and the evaluation rig rather than only the weight file, then freeze and sign a specific artifact for each fielded release. Every candidate model runs through the same automated evaluation, producing the same slice table with confidence intervals, and the delta against the current baseline is what the review board sees. That combination lets a program improve a model between milestones without repeating the full verification campaign, provided the pipeline and rig themselves are under change control.
Three layers. Input monitoring compares what the model sees against what it was tested on and needs no labels, so it works from day one. Output monitoring tracks score distributions, abstention rate, latency and operator overrides, which are the cheapest labels a fielded system produces. Periodic truthed evaluation samples fielded data, labels it under the test rules and re-runs the rig, which is the only layer that yields a defensible number after fielding. Pair all three with a written trigger table naming the response and the decision authority.