LLM Documentation Pipelines for Defense Combat-System Problem Reports

Why combat-system problem reports are a hard documentation problem

Defense combat-system software has a heavy paperwork tax. Every defect or anomaly an engineer finds becomes a structured problem report — a multi-section document that follows a strict template and gets read by configuration boards, sustainment leads, and safety reviewers. Writing those reports well takes a long time. Writing them badly takes even longer, because they come back for rework.

The published software-engineering literature on defect-report quality (Bettenburg et al., FSE 2008 onward) is consistent on what makes reports useful. Clear reproduction steps. Faithful environment description. Specific symptom characterization. Correct severity classification. The same literature is consistent on where reports fail — missing reproduction context, vague symptom prose, mis-classified severity, and sections that drift out of the template.

This is the kind of problem LLMs (large language models) are well suited to assist with, but only when assembled carefully. A single prompt to a generic chatbot will not produce a defensible structured document. A multi-engine pipeline, where each engine has one narrow job, is what the open literature reports as working.

The multi-engine architecture

"Multi-engine" just means breaking the work into several specialist modules instead of one large model doing everything.

The published pattern, recurring across academic and industry write-ups (Microsoft Research on documentation generation, Salesforce's CodeT5 work, the open-source LangChain and LlamaIndex tutorials on document pipelines), names roughly the same five jobs:

1. Intake module. Reads the raw artifact — engineer notes, log excerpts, screenshots, test outputs — and normalizes it into structured fields. The output is a JSON object the rest of the pipeline can consume reliably.

2. Standards engine. Holds the document template (which sections exist, what each section must contain, what the formatting rules are). Every downstream module checks its output against this template. Think of it as the rulebook the rest of the pipeline must satisfy.

3. Section-generator module. A fine-tuned LLM that writes one section at a time, using the structured intake plus the standards engine's section spec. Fine-tuning on a domain-specific corpus matters more than model size at this stage.

4. Anomaly-classification module. Often an AutoML classifier (AutoML = automated machine learning, where the system selects and tunes the model itself) that assigns the report's anomaly to a taxonomy. This is what feeds severity and routing.

5. Validator module. Checks the assembled draft against the standards engine and a set of explicit constraints. Flags anything that fails for human review.

Each module is small enough to test, evaluate, and replace independently. That is the entire reason for the decomposition. A monolithic prompt cannot be debugged piece by piece; a five-module pipeline can.

The intake module: messy in, structured out

Intake is the part that turns the engineer's reality — chat logs, half-written notes, screenshots, log excerpts — into a clean JSON object the rest of the pipeline can trust.

Concretely, an intake module produces fields like component, build_version, environment, reproduction_steps, observed_symptom, expected_behavior, and relevant_log_excerpts. The published evaluation discipline treats this as a structured-extraction task with a strict schema, not as a free-form summarization task.

The dominant published failure mode is silent dropping. The engineer mentioned a build version in a Slack message; the intake module did not extract it; the report goes downstream missing context the engineer assumed everyone had. The fix is also published: explicit schema validation on every intake output, with a "missing-field" prompt back to the engineer when a required field cannot be filled with confidence.

The methodological tip from the open literature is to keep the intake schema small. Five to ten required fields, validated strictly. Extending the schema is easy; debugging a thirty-field schema with mostly-empty fields is not.

The standards engine: the rulebook everything checks against

The standards engine is the part that knows what a "good" report looks like. It is not an LLM. It is a structured representation of the document template, the section specifications, the formatting rules, and the rejection criteria.

The published practice is to encode the standards engine declaratively — YAML, JSON Schema, or a domain-specific language — rather than to bake it into prompt strings. This separation matters for two reasons. The standard changes more often than the model. And every downstream module needs the same standard to check against.

One concrete pattern from the open literature: each section spec includes a list of must-have facts ("identifies the failing component"), a list of must-not patterns ("does not contain unverified causal claims"), and a length budget. The validator checks each. When a section fails, the failure points to the specific spec bullet that was violated, not to a vague "section quality" score.

Fine-tuned section generators

The section-generator modules are the part most people picture when they hear "LLM documentation pipeline." They are also the part where size matters least.

Public benchmarks on structured documentation tasks (Microsoft's DocFM work, Hugging Face's report-generation evaluations, several IEEE Software Engineering papers from 2022 onward) all show the same pattern. A medium-sized model (7B to 13B parameters) fine-tuned on roughly 1,000 high-quality domain examples beats a large general model used zero-shot. The fine-tuning teaches the section-specific tone, the standards-engine vocabulary, and the rejection patterns of the reviewer audience.

The published evaluation methodology is also instructive. Don't measure the section in isolation; measure whether the assembled report is acceptable to the reviewer audience. A section that scores well on isolated metrics but produces a report the reviewer rejects is a worse outcome than a section that scores marginally lower but yields a report that closes review.

Domain corpora are the rate-limiting input. The open practice is to bootstrap from existing approved reports, redact identifying detail, and train on the redacted versions. The corpus stays small but high-signal.

AutoML for anomaly classification

Anomaly classification is the part that decides "what kind of problem is this." The classification feeds severity, routing, and reviewer assignment, so it is consequential.

AutoML (automated machine learning) tools — AutoGluon, FLAML, H2O AutoML — have matured to the point where they reliably outperform single hand-tuned classifiers on structured-text classification tasks with modest training data. The open literature on industrial defect classification (recurring in IEEE Software and ICSE) reports F1 scores in the 0.78 to 0.90 range on multi-class taxonomies of 20 to 60 anomaly classes, depending on training-set size and class balance.

The methodological discipline here is to hold the taxonomy out of training data when measuring generalization. A classifier that learned to repeat the most common labels on a balanced training set will look fine on an internal test set and fail on a quarter where a new class appeared. The published evaluation protocols use temporal hold-out (train on quarters 1-3, test on quarter 4) to surface this failure mode.

Anomaly classification is also where calibration matters most. Reviewers do not need a confident-sounding wrong label; they need a confidence-aware label. Published systems that surface a calibrated probability ("this looks 62% Class A, 28% Class B, 10% other") are accepted at higher rates than systems that surface only the top label.

Module	Job	Common technique	Failure mode
Intake	Messy artifacts → structured JSON	Schema-constrained LLM extraction	Silent dropping of unmentioned fields
Standards engine	Rulebook the rest checks against	YAML / JSON Schema / DSL	Standard baked into prompts; drifts
Section generator	Writes one section at a time	Fine-tuned 7B-13B LLM on domain corpus	Hallucinated reproduction steps
Anomaly classifier	Assigns the anomaly taxonomy class	AutoML over structured-text features	Confident wrong label without calibration
Validator	Checks draft against standards engine	Rule pass + LLM-as-judge	Vague "quality score" not tied to spec

Output validators

The validator is the last gate before a draft goes to a human reviewer. Its job is to catch the obvious failures so the human reviewer's time is spent on the subtle ones.

Effective validators in the open literature do two passes. The first pass is rule-based: every must-have fact present, every must-not pattern absent, length budgets respected, structured fields well-formed. This is fast and deterministic. The second pass is an LLM-as-judge with strict grounding — "given the standards engine spec, does this section satisfy it" — flagging anything the rule pass missed.

The published warning is to never let the LLM-as-judge be the sole validator. It introduces its own hallucination surface, and that surface compounds with the section-generator's hallucination surface. The two-pass architecture — rule first, judge second, human last — is the consistent finding across the published evaluations.

Where the published research lands today

LLM-assisted documentation pipelines are a maturing field. The intake, standards-engine, and validator layers are well-understood and routinely deployable. The section-generator layer is mature with disciplined fine-tuning. The anomaly-classification layer is mature for taxonomies that change slowly and harder for taxonomies under active revision.

The dominant remaining challenge in the open literature is end-to-end evaluation. Module-level scores look good; end-to-end "did the report pass review on the first try" rates are harder to predict from the module scores. The published practice is to maintain an end-to-end evaluation harness with a held-out review-outcome label, and to optimize the pipeline against that harness rather than against module-level proxies.

Reviewer adoption is the other recurring theme. A pipeline whose output looks like a draft an engineer would have written gets accepted; a pipeline whose output reads like generic LLM prose gets rejected on style alone, even when the technical content is correct. Domain fine-tuning matters not just for accuracy but for tone fit.

Reviewer-in-the-loop integration

None of the published deployments cut the human reviewer out. The successful pattern is a pipeline that produces a defensible draft and a reviewer who edits and approves it.

The interface details that show up across the open literature are consistent. Show the reviewer the assembled draft and a side-by-side of the structured intake. Surface the standards-engine spec next to the section being reviewed. Highlight any validator warnings inline. Track every reviewer edit as a labeled correction the pipeline can learn from.

Edits are the most valuable feedback signal in the system. A pipeline that captures and incorporates reviewer edits over time improves more steadily than one that only learns from periodic corpus refreshes. The published practice is to treat the edit stream as a continuous fine-tuning signal, retraining the section generator at a regular cadence with the new examples weighted appropriately.

Frequently asked questions

How we use this site

We write articles like this to make our reading visible — what we think the open literature says, what we think the open gaps are, and where careful work might land. We do not use these pages to preview proposed approaches in active program spaces. Precision Federal is a software-only SBIR firm. If your office is funding work in this area and would value a software-first partner with a documented public-reading habit, we welcome the introduction.