The five words that end a review
Where did that number come from. It gets asked in a closeout meeting, in a technical interchange, in a prime's diligence packet, in an audit entrance conference. It sounds like a clarifying question and it is not. It is a test of whether the reporting team has a system or a habit. A team with a system answers in about fifteen seconds by opening a record. A team with a habit answers with a name, a rough recollection of a spreadsheet, and a promise to go look. The second answer costs more than the first even when the number turns out to be right, because the reviewer now has to price the risk that the rest of the numbers are the same way.
The failure mode is rarely fabrication. It is far more often a figure that was correct on the day it was produced, then got copied into a slide, then into a summary, then into a final report, losing one layer of context at each hop until nothing remains but the digits. By the time it reaches the reader who has to act on it, nobody can say which dataset it came from, which version of the code produced it, whether it was measured on real inputs or produced by a model standing in for inputs nobody had, or whether it was simply lifted from someone else's paper.
Several fields solved this, and each solved it the same way: attach a label saying what kind of claim the number is, and keep a record that lets a stranger get back to the thing that produced it. Metrology did it with calibration certificates, software supply chain with build attestation, federal cost analysis and simulation credibility each with their own vocabulary. The engineering is not hard. The discipline is in doing it while the work is happening, because retrofitting provenance is where the cost goes.
Four labels, four different claims
Every figure in a report is one of four things, and mixing them is the single most common defect we see in delivered technical documents. A number is measured when it came out of an instrument or an execution on real data. It is modeled when it came out of a fitted or analytical relationship applied to inputs, some of which may be assumed. It is simulated when it came out of executing a model that stands in for a real-world system the team did not have access to. It is cited when it came from somebody else's document.
Those are not degrees of confidence on one scale. They are four different claims that fail in four different ways, and a reader who cannot tell them apart cannot judge any of them. A measured throughput of 41 requests per second and a simulated throughput of 41 requests per second are the same digits carrying opposite amounts of information about what will happen in production. Printing them in the same table with the same formatting is a reporting error even when both are correct.
The labelling convention has to survive the trip out of the report. Numbers migrate into decks, one-pagers, dashboards and emails, and they migrate without their footnotes. A convention that only exists in the appendix of the final report is a convention that stops working the moment anybody quotes it. The label belongs next to the digits, in the table cell or the caption, wherever the number goes.
| Label | What the number asserts | What has to exist behind it | How it fails |
|---|---|---|---|
| Measured | This system, on this input, produced this output | Input manifest with hashes, code version, environment, config, seed, raw output, and the transform from raw output to the printed figure | Silent drift in inputs or environment; the run that produced it can no longer be re-executed |
| Modeled | Under these assumptions, the relationship implies this value | The functional form, the fitted parameters and their provenance, the assumption set, the input ranges the fit is valid over | Use outside the range the model was fitted on, with no warning attached |
| Simulated | A stand-in for the real system behaved this way when executed | Model version, scenario definition, input pedigree, verification and validation evidence, and the accepted use the model was approved for | Reported as though it were measured; validated for one purpose and used for another |
| Cited | Someone else reported this, and we are repeating it | The primary source, the exact page or table, the population and conditions it applies to, and whether it was checked | The chain leads to a source that never measured it either |
Cited is the label that gets abused
Citation feels like the safe option. It transfers the claim to a named authority and moves the argument along. What it does not transfer is evidence, and the government's own audit guidance is explicit about it. GAO's framework for assessing data reliability states the point in one line: attributing the data to its source does not alleviate the need to assess the reliability of the data. Naming where a figure came from is the beginning of the work, not a substitute for it.
The most careful demonstration of how badly this can go is Steven Greenberg's 2009 analysis in the BMJ of a citation network in the biomedical literature. He mapped every PubMed-indexed English-language paper addressing one specific belief about a protein's role in a muscle disease: 242 papers, 675 citations between them, and 220,553 citation paths supporting the claim. He identified three mechanisms by which unfounded authority accumulated. Citation bias, where papers weakening the belief were cited less. Amplification, where papers presenting no data at all on the question expanded the apparent support for it. And invention, including the conversion of a stated hypothesis into a cited fact. None of that requires a dishonest author. It only requires many honest authors each citing one layer up.
The practical defense is cheap. When a figure is cited, the record stores the primary source rather than the secondary one, the specific table or page, and one sentence on the conditions the figure applies to. If those three things cannot be filled in, the figure is not usable as evidence and belongs in the prose as context, not in a results table. We have written separately on carrying that discipline down to the sentence level in generated text, in span-level provenance.
The run record behind a measured number
For measured figures, the unit of provenance is the run, not the number. A run record is the set of facts that lets a person who was not there re-execute the thing that produced the figure and see the same value come out. At minimum that means a content hash of every input file, the exact code revision, the resolved dependency versions, the configuration and any random seeds, the command that was issued, the raw output before any aggregation, the transform from raw output to the printed figure, the wall-clock time, and the identity of whoever or whatever launched it. Give the whole bundle an identifier, print that identifier next to the number, and the five-word question has a five-second answer.
The conceptual model for this was standardized years ago. The W3C's PROV data model, a Recommendation since 30 April 2013, describes provenance as relations among three things: entities, which are the data items; activities, which are the processes that act on them; and agents, which are the parties responsible. The useful relations are wasGeneratedBy, used, wasAssociatedWith and wasDerivedFrom. Most teams do not need to adopt PROV as a serialization format. They do need its discipline, which is that every reported artifact points back to the activity that produced it and the agent accountable for it, with no gaps in the chain.
The software supply chain community arrived at the same structure under commercial pressure and added one idea worth stealing. SLSA build provenance, whose specification reached version 1.2 with an approved Build track, splits an attestation into two sections: a build definition describing what was built, and run details describing how that particular execution proceeded and who ran it. The idea worth stealing is that the provenance is emitted by the build platform, not written by the author of the artifact. Self-reported provenance is better than none. Platform-emitted provenance is what survives an adversarial reading, because the party with the incentive to shade the record is not the party writing it.
That principle transfers directly to evaluation. If the harness writes the run record automatically, at the moment of execution, with no human in the loop, then the record is evidence. If an engineer fills in a results spreadsheet by hand at the end of the week, the record is testimony. Both have their place; only one of them holds up when the reviewer is paid to doubt it. We go deeper on building harnesses that behave this way in evaluation harnesses that survive a review.
How graded the input question actually is
Traceability is not binary, and the best public rubric for grading it comes from NASA. NASA-STD-7009B, the agency's standard for models and simulations, approved 5 March 2024 and superseding 7009A, defines input pedigree as a record of traceability from the input data's source through every aspect of its transmission, storage and processing to its final form. It then grades that record on a five-level ladder. The levels below are the standard's own definitions, compressed.
Input pedigree — the five defined levels
Bar length shows the level, not a percentage. Source: NASA-STD-7009B, Table 16, Level Definitions for the M&S Input Pedigree Factor. The standard sets no required level; it requires only that the level be determined and reported.
That last clause is the part most teams miss, and it is the most useful sentence in the whole standard. NASA imposes no minimum. It requires that the level be determined and reported, and leaves the decision-maker to decide whether that level is good enough for the decision in front of them. This is a far more honest posture than a pass or fail gate, and it is one any delivery team can adopt unilaterally. Grade the pedigree, print the grade, let the reader price it.
Modeled and simulated are not the same word
Teams use these interchangeably and reviewers do not. In the defense vocabulary the distinction is carried by three separate activities. Verification asks whether the model was implemented correctly against its specification. Validation asks whether the model's outputs compare favorably against a referent drawn from the real-world system. Accreditation is an official determination that the model is acceptable for a specific intended use. DoD Instruction 5000.61, reissued effective 17 September 2024, governs verification, validation and accreditation of models, simulations, distributed simulations and associated data, with DoD Manual 5000.102 carrying the procedures.
The word that does the work there is specific. A model accredited for one purpose is not thereby accredited for another, and a simulated figure reported without the accompanying statement of what the model was accepted for is a number floating free of the only thing that made it meaningful. This is the same failure as reporting a benchmark result without saying which split it was measured on, and it is worth reading alongside choosing a baseline you can defend.
NASA's standard adds two reporting requirements that translate cleanly into commercial work. When results go to decision-makers, explicit warnings are required for a defined list of conditions, each accompanied by at least a qualitative estimate of the impact. The list includes unmet acceptance criteria, violated model assumptions, execution warnings and errors, and outstanding defects. And the estimate of uncertainty must be reported in one of three ways: quantitatively, qualitatively, or as a clear statement that neither is available. That third option is the one worth adopting everywhere. A stated absence is information. A silent absence reads to a reviewer as a claim of precision the team never made and cannot defend.
The uncertainty half of the number
A figure reported to four significant digits with no interval attached is asserting something about repeatability that the team has usually not checked. Metrology fixed this a long time ago. NIST Technical Note 1297 groups the components of measurement uncertainty into two categories by how they were evaluated: Type A, by statistical analysis of a series of observations, and Type B, by other means, which typically means scientific judgment drawing on prior data, manufacturer specifications, and experience with the instrument. Both are legitimate. Reporting them without saying which is which is not.
The reporting conventions in accredited testing carry the same idea into the document. Under ISO/IEC 17025:2017, the clause covering reporting of results requires calibration certificates to state measurement uncertainty, the conditions under which the calibration was performed, and how the measurements are metrologically traceable. A calibration lab cannot hand a customer a bare number. There is no reason a delivery team should hand a program office one.
In practice this is a small habit: run the thing more than once, report the spread, and say what varied between runs. If repeat runs come out identical, say that too, because it tells the reader the variability lives entirely in the inputs. Our piece on uncertainty a decision-maker can use covers presenting it without burying the finding.
What an evaluator or auditor actually asks
Federal reviewers are not improvising. Under generally accepted government auditing standards — the Yellow Book, whose 2024 revision took effect for engagements beginning on or after 15 December 2025 — auditors assess the sufficiency and appropriateness of computer-processed information whether they extracted it themselves or received it. Appropriateness there is a quality measure covering the relevance, validity and reliability of the evidence. GAO's guide on assessing data reliability, GAO-20-283G, gives the framework: reliability means the data are applicable to the purpose and sufficiently complete and accurate, where completeness is whether relevant records and fields are present and populated, and accuracy is whether recorded data reflect the underlying reality.
The determination has exactly three outcomes: sufficiently reliable for the purpose, not sufficiently reliable, or of undetermined reliability. That middle category matters more than it looks. Data of undetermined reliability can still be used, but the report must explain the steps taken, the reason the determination could not be made, and the rationale for using the data anyway. Notice what is being graded there. Not the number. The record behind it.
The guide's appendix lists the documentation an auditor will request, and it reads like a checklist of things a delivery team should already have: a data dictionary defining every field, written policies for data entry and validation, record counts and value ranges, evidence that users were trained, and any prior reviews of data quality. If a team can produce that packet on request, an assessment that could have consumed weeks becomes a short conversation.
Cost figures get the same treatment from a different guide. GAO's cost estimating guide, GAO-20-195G, organizes 18 best practices across 12 steps and grades an estimate against four characteristics: comprehensive, well-documented, accurate and credible. The well-documented characteristic is pure provenance. It requires that the documentation show the source data used, the reliability of that data, and the estimating methodology behind each element, and that it describe how the estimate was developed so that a cost analyst unfamiliar with the program could understand what was done and replicate it. That last phrase is the standard worth holding every technical number to as well.
| What gets asked | What answers it | What does not |
|---|---|---|
| Where did this number come from? | A run identifier that resolves to inputs, code revision, config and raw output | The name of the person who ran it |
| Would I get the same number? | Pinned environment, recorded seeds, and one command that regenerates it | An assurance that the pipeline is deterministic |
| How complete is the underlying data? | Record counts, coverage by field, and a written statement of known gaps | That the data came from the customer's own system |
| How accurate is it? | An interval, a spread across repeat runs, or an explicit statement that neither was computed | More decimal places |
| What happened between the source and the slide? | The recorded chain of transforms, each pointing at the step before it | A spreadsheet nobody can reconstruct |
| What if the analyst is gone? | A record written to be read by a stranger, stored where the contract requires | An offer to reach out and ask them |
Closeout is when the record stops being optional
The obligation outlives the project. Under FAR Subpart 4.7, contractors make records available for three years after final payment, or the specific periods at FAR 4.705 for certain record classes, whichever expires first, with longer periods where a clause, statute, or unresolved litigation requires it. The audit and records clause at FAR 52.215-2 carries the same three-year window into the contract itself. For recipients of federal awards rather than contracts, 2 CFR 200.334 sets retention at three years from submission of the final financial report, extended where litigation, claims or audit findings remain open.
Closeout does not happen quickly enough to help. FAR 4.804-1 sets file closeout time standards of six months for firm-fixed-price contracts, twenty months for most others, and thirty-six months where indirect cost rates have to be settled. So the realistic planning assumption is that a question about a figure produced in month three of performance can arrive four or five years later, aimed at a team that has turned over. That is the reason the run record has to be legible to a stranger rather than sufficient for its author.
One caution on paragraph numbers. FAR Part 4 is being rewritten. A proposed rule under FAR Case 2026-001 covering Parts 1, 2, 4, 33, 39, 40 and 53 published on 23 June 2026 with comments closing 23 July 2026, and many agencies have been operating under class deviations directing their workforce to follow the overhaul text rather than the codified part. The retention obligation itself is stable and statutory in character, but cite the clause your contract actually incorporates, and check whether the awarding agency is running a deviation. Our piece on records retention for software vendors works through the practical filing side.
A convention a team can adopt this week
None of this requires a platform purchase. It requires that five facts travel with every figure, and that the harness rather than a person be responsible for writing them down.
- A label. Measured, modeled, simulated or cited, printed in the cell or the caption, not only in the appendix.
- A run identifier that resolves to inputs, code revision, environment, config and raw output.
- An uncertainty statement — an interval, a spread across repeats, or an explicit note that neither was computed.
- A date for when the run happened, distinct from when the document was written.
- An agent — the person or system that produced it, recorded automatically at execution.
- A stated scope for anything modeled or simulated: what it was validated for and what it was not.
- A storage location that satisfies the contract's retention period, decided before the first run rather than at closeout.
The cost is front-loaded and small. Writing the manifest and the run-record emitter into the harness is a first-sprint task, and after that the marginal cost per number is zero because nobody is doing it by hand. The expensive version is the retrofit, where raw outputs have to be regenerated from environments that no longer exist and the team ends up re-running experiments to justify figures that were correct the first time. Teams that have lived through one such retrofit never skip the emitter again.
Bottom line
Provenance is not paperwork sitting beside the technical work. It is the property that converts a number into evidence. Four labels tell a reader what kind of claim they are looking at. A run record lets them get back to what produced it. An uncertainty statement, even one that says none was computed, tells them how hard to lean on it. Federal reviewers already grade all three, under vocabularies that have been public and stable for years. The teams that write these facts down while the work is happening keep their numbers. The teams that plan to reconstruct them later find out at closeout what reconstruction costs.
Frequently asked questions
It means every figure points back to the activity that produced it and the party responsible, with no gaps. The W3C PROV data model formalizes this as entities, activities and agents linked by relations such as wasGeneratedBy and wasDerivedFrom. In practice it means a run identifier printed next to the number that resolves to inputs, code version, configuration and raw output.
Put the label in the table cell or caption, not only in the appendix, because numbers get quoted without their footnotes. Measured means real inputs through the real system. Modeled means a relationship applied to inputs, some possibly assumed. Simulated means executing a stand-in for a system the team did not have. Each carries a different obligation of supporting evidence.
Auditors assess the sufficiency and appropriateness of computer-processed information regardless of who extracted it. GAO's data reliability framework grades data as sufficiently reliable, not sufficiently reliable, or of undetermined reliability, based on completeness and accuracy for the stated purpose. Expect requests for a data dictionary, entry and validation procedures, record counts and value ranges, and any prior data-quality reviews.
Under FAR Subpart 4.7 and the audit and records clause, contractors make records available for three years after final payment, subject to the specific periods at FAR 4.705 and longer where a clause, statute or open litigation requires. Federal award recipients fall under 2 CFR 200.334, three years from the final financial report. FAR Part 4 is under active rewrite, so cite the clause your contract incorporates.
Yes, if the absence is stated. NASA's standard for models and simulations requires uncertainty to be reported quantitatively, qualitatively, or as a clear statement that neither is available. A stated absence is information a reader can price. A silent one reads as a precision claim the team never made and cannot defend.