The failure mode is not invention. It is smoothing.
When an analytic product built on a language model goes wrong, it rarely goes wrong by inventing a whole event. It goes wrong by smoothing. Two records merge into one fluent sentence. A hedge in the source becomes a flat assertion in the summary. A threshold is restated a little rounder than it was written. A figure that belonged to one entity ends up attached to another. None of that reads as fabrication, and by the time it reaches a customer it is indistinguishable from analysis.
Citing sources does not fix it, and there is a measurement rather than an opinion behind that claim. Nelson Liu, Tianyi Zhang, and Percy Liang audited four generative search engines and found that on average only 51.5% of generated sentences were fully supported by their citations, and only 74.5% of citations supported the sentence they were attached to (Evaluating Verifiability in Generative Search Engines, Findings of EMNLP 2023, arXiv:2304.09848). Every system in that study was already showing its sources. The footnote was there. The support was not.
That result is the design problem in one line. A citation is a testable assertion — this specific text is entailed by that specific span in that specific record — and if nothing in the pipeline tests it, the citation is decoration. So the useful question is not how to discourage a model from making things up. It is what in the system would fail loudly if it did.
The specification already exists, and it predates the models
Anyone building a derived-insight product for a defense or intelligence customer should start with two directives written long before this generation of models, which describe the problem more precisely than most engineering documents do. Intelligence Community Directive 203, Analytic Standards, signed by the Director of National Intelligence on 2 January 2015, sets out nine Analytic Tradecraft Standards. Read as engineering requirements rather than as writing guidance, three of them specify a traceable product.
- Properly describes quality and credibility of underlying sources, data, and methodologies — products should identify the sources and methods judgments rest on, and use source descriptors per ICD 206. The directive names the factors: accuracy and completeness, possible denial and deception, age and continued currency, technical elements of collection, source access, validation, motivation, possible bias, expertise.
- Properly expresses and explains uncertainties associated with major analytic judgments — likelihood must be expressed using one of two fixed seven-term sets (almost no chance through almost certain(ly), or remote through nearly certain), analysts are strongly encouraged not to mix terms across the two rows, and a confidence level must not be combined with a degree of likelihood in the same sentence.
- Properly distinguishes between underlying intelligence information and analysts' assumptions and judgments — statements conveying underlying information must be clearly distinguishable from statements conveying assumptions or judgments, with assumptions stated explicitly when they bridge key information gaps.
The third is the hard one for a generative system, because model output arrives in a single undifferentiated register. Retrieved fact and inferred conclusion come out in the same voice, at the same apparent confidence, often in the same sentence. ICD 203 requires them to be visibly different things in the product. You do not get there by asking nicely in a system prompt; the distinction has to exist in the data structure before it can exist in the prose.
ICD 206 is a record definition, not a style guide
ICD 206, Sourcing Requirements for Disseminated Analytic Products, effective the same day and superseding the 17 October 2007 edition of the same directive, supplies the machine-readable half. It requires source reference citations presented as sequentially numbered endnotes, generated each time a source is directly cited in the text or a specific judgment, assessment, estimate, alternative hypothesis, or confidence level depends on that source. Then it enumerates what the endnote has to contain.
Read this as a schema your pipeline populates
Portion marking presenting the classification of the citation; identification of the information originator, author, producer, or owner; an unambiguous source identifier sufficient for efficient location and retrieval; a title for the source document or information; the date of issuance, publication, or posting — and, in the directive's own words, "if date of online posting is unknown, then date of online access"; page number where applicable; classification of the information extracted from the source; overall classification of the source document; and a source descriptor.
Every one of those fields is something a pipeline can fill from a captured document, and any field it cannot fill is a defect the build should surface rather than a blank the prose glosses over. ICD 206's glossary defines the source descriptor as "a brief, narrative exposition of factors that affect or indicate the quality or credibility of a single source." A model can draft that field from what the record contains; it should never supply it from general knowledge.
One further provision bears directly on open sources. Where a cited source is dynamic — the directive's own example is an Internet posting — or ephemeral, or not subject to systematic storage, ICD 206 requires that a record of the source be preserved for at least one year by the producing element. Read as a build requirement rather than as records management, that means capture is part of the pipeline, not a courtesy extended when someone remembers.
Build the records first, render the prose second
The common pipeline shape is retrieve, prompt, generate, then attach citations. Attaching citations last means the citation gets chosen to fit a sentence that already exists — which is precisely the failure the EMNLP audit measured. Inverting the order changes what the system is capable of producing.
Retrieve and freeze. Assemble the candidate documents, store the bytes, and stop. Nothing downstream touches the live network again.
Extract spans, not paraphrases. The unit is a document identifier plus character offsets plus the literal text at those offsets. A paraphrase at this stage has already discarded the thing you need later. We have written separately on what span-level provenance costs and what it buys.
Build the claim set as records. Subject, assertion, source identifier, offsets, capture timestamp, and a field marking the item as reported information or as our own inference — ICD 203's third standard, expressed as a column.
Render prose from the records. The written product becomes a view over a table. A sentence with no backing record cannot be rendered, because there is nothing to render it from.
The evaluation contract exists in the literature as well. Rashkin et al., Measuring Attribution in Natural Language Generation Models (Computational Linguistics 49:4, December 2023), formalizes Attributable to Identified Sources — the requirement that output pertaining to the external world be verifiable against an independent, provided source. AIS is a per-sentence property and it is measurable. Measure it on the shipped rendering rather than on the retrieval step, because the rendering is what the customer reads.
Two consequences are where most implementations quietly leak. First, numbers are copied, never generated: every numeric literal, date, quantity, and threshold is lifted byte-for-byte from a span and diffed against it, and rounding or unit conversion is a transform that gets its own record and label. Second, the model does not introduce entities. Selecting, ordering, grouping, and classifying are jobs a small model does well and that you can build a check around. Naming an organization, person, or place that no retrieved span contains is not one of them.
Discouraged versus structural
"Only use the provided context" in a system prompt is discouragement. It is a request, and requests degrade under long context, adversarial input, and questions the corpus does not answer. Structural means the render step cannot emit a sentence with no record behind it, and a separate process fails the build when it tries.
| Failure mode | What it looks like in the output | The mechanical check |
|---|---|---|
| Unsupported sentence | A fluent claim carrying a citation whose span does not contain it | Per-sentence entailment check against the cited span; unsupported sentences are dropped, not flagged for someone to review later |
| Merged records | Facts from two documents fused into one sentence | One claim, one record identifier; a sentence carrying two identifiers is split before rendering |
| Softened hedge | The source says "may," the product says "will" | Likelihood drawn only from ICD 203's fixed term sets; modal language selected from a list rather than generated |
| Number drift | A threshold restated slightly rounder than the record states it | Numeric literals copied from the span and byte-diffed against it before render |
| Stale source | The page changed or disappeared after it was read | Content hash plus capture timestamp; the product rebuilds against captured bytes, never against the live URL |
| Injected instruction | Retrieved text steering the model's behavior | Retrieved content handled strictly as data and never as instruction — the pattern NIST's adversarial machine learning taxonomy (NIST AI 100-2e2025, March 2025) catalogs as indirect prompt injection |
None of those checks is clever, and that is the point. Cleverness is what you reach for when the architecture permits a failure and you are trying to talk the system out of it. The checks that survive an assessment are the boring ones that run on every build.
Provenance is a record about the corpus, not a URL
Open sources move. Pages get edited, aggregators reindex, licensing terms change, posts are deleted. A URL captured on Tuesday is not a citation to what was read on Tuesday. For each document we store the retrieval timestamp and method, the retrieved bytes, a content hash over those bytes, the responding host, and the chain of transforms that produced the text the extractor saw.
There is no need to invent a vocabulary for that. W3C's PROV-O (W3C Recommendation, 30 April 2013) already defines Entity, Activity, and Agent along with relations including wasDerivedFrom and wasGeneratedBy — enough to express "this claim was derived from this extraction activity over this captured document by this component version." Using a published ontology also means the provenance stays legible after it leaves your system, which matters when the product has to be defensible somewhere you are not.
The same record does double duty on the accreditation side, because most of what it holds is what the Audit and Accountability family in NIST SP 800-53 Rev. 5 asks for anyway — AU-3 on the content of audit records, AU-8 on time stamps. Build provenance as a first-class artifact rather than a logging afterthought and the assessment evidence becomes a byproduct of normal operation.
What changes when it has to run at IL5 or IL6
DISA's DoD Cloud Computing Security Requirements Guide defines the Impact Levels: IL2 for public or non-critical mission information, IL4 for controlled unclassified information, IL5 for higher-sensitivity CUI, mission-critical information and national security systems, and IL6 for information classified up to SECRET. Those levels drive the architecture harder than any modeling decision does; we go through the component-by-component consequences in a separate piece on what the levels actually change.
At IL5 and above there is no hosted frontier API on the mission path. You carry weights or you do not run. Treated correctly that is not a limitation to work around — it pushes the design toward small models doing narrow, checkable jobs (span selection, entity typing, classification, ordering) rather than one large model doing everything and being trusted about all of it. Those are also the jobs you can wrap a mechanical check around.
A component that has to drop into a boundary somebody else accredited needs a specific shape: no outbound network calls required for function, configuration by file rather than by console, no telemetry as a condition of operation, logs emitted as structured records the host can ingest, and deterministic output for a fixed corpus and a fixed seed. Determinism deserves the emphasis. A component that answers the same question differently over the same corpus does not have an accuracy problem; it has no defined output, and an undefined output cannot be validated, reproduced, or defended in a review.
One constraint gets missed regularly. The corpus may be open — the IC OSINT Strategy 2024–2026, released by ODNI and CIA on 8 March 2024, defines OSINT as intelligence derived exclusively from publicly or commercially available information that addresses specific intelligence priorities, requirements, or gaps — but the questions asked of it usually are not. In many deployments the query log is the most sensitive artifact in the whole system. Build so the component never needs to phone home and that problem does not arise.
Where the line is for us
Stated scope is more useful to a buyer than claimed breadth, so here is ours in the places it actually binds.
Will you let the model fill a gap from general knowledge?
No. If the corpus does not contain it, the product does not say it — including in the cases where the model happens to be right. A system permitted to be right from memory once is a system you can no longer audit, because you no longer know which sentences came from the data. Gaps get named as gaps, which ICD 203 asks for anyway when it directs analytic elements to identify and address critical information gaps and products to note the causes of uncertainty.
Can you give us a confidence score on every conclusion?
We can give a likelihood expressed in ICD 203's fixed vocabulary, alongside the evidence that drove it and the assumptions it depends on. We will not emit a calibrated-looking percentage derived from a model's estimate of its own correctness. That number is not measured, it will be read as though it were, and the first time someone audits it the rest of the product loses credibility with it.
Do you hold an IL5 or IL6 authorization?
No, and we will not imply otherwise. We build components designed to run inside a boundary a customer or their cloud provider has already accredited, and we will write the boundary description, data-flow documentation, and control-inheritance narrative that has to go into the System Security Plan. Representing inherited authorization as our own is the kind of claim that survives a sales conversation and fails an assessment.
Will you build the collection side as well?
Within a stated limit. We build collection against publicly or commercially available sources where access is documented and lawful, including licensed commercial feeds. We do not build collection that depends on bypassing authentication, defeating access controls, or misrepresenting identity to obtain data, and that answer does not change based on who is asking or what the mission is.
Can we see the evaluation, including the parts that went against you?
Yes — that is the only version worth showing. We report recall and false-extraction rate together and per category, because recall on its own can be raised by simply producing more items. And we hold out a blind split labeled before the system ever runs against it. An evaluation whose labels were written after we saw our own output measures how well we tuned, not how well the system generalizes.
Frequently asked questions
That every statement in the output resolves to a specific record, a specific span within it, and a capture timestamp — and that the resolution is checked by the system rather than asserted by it. The standard from the research literature is Attributable to Identified Sources (Rashkin et al., Computational Linguistics 49:4, 2023): output about the external world must be verifiable against an independent, provided source.
No. The Liu, Zhang, and Liang audit of generative search engines (Findings of EMNLP 2023) found citations frequently failed to support the sentences attached to them, in systems that were already citing everything. A citation means something only when a separate process tests whether the cited span entails the sentence, and drops the sentence when it does not.
By carrying the distinction as a field on the claim record before any prose exists, then rendering the two categories differently in the product. ICD 203 requires analytic products to clearly distinguish statements conveying underlying intelligence information from statements conveying assumptions or judgments, and to state assumptions explicitly when they serve as the linchpin of an argument or bridge key information gaps.
ICD 206 enumerates the elements of a source reference citation, including an unambiguous source identifier sufficient for efficient retrieval, the date of issuance, publication or posting (or date of online access when the posting date is unknown), and a source descriptor covering quality and credibility factors. It also requires that a record of a dynamic or ephemeral source be preserved for at least one year.
Because a component that returns different answers to the same question over the same corpus has no defined output, and nothing without a defined output can be validated, reproduced, or defended in an accreditation review. Accuracy can be improved incrementally after fielding; reproducibility is a property you either designed in or did not.