A program that intends to add AI in three years is making decisions today that will determine whether that is a six-month effort or a two-year one. Most of those decisions are not about models. They are about what gets collected, how it is stored, whether anyone can tell where a record came from, and whether the people who need it can get to it. Programs that treat data engineering as something the AI phase will handle arrive at that phase with terabytes they cannot train on. The work to prevent that is ordinary, it is fundable now, and it produces value in the current phase whether or not the AI phase ever happens.
This is written for the chief engineer or program manager who can see the AI requirement coming and wants the program positioned for it. The question is what to build in the current phase, how to justify it in the plan and the budget, and where a specialist partner does the work.
What "we have the data" usually means
Nearly every program believes it has data. What it usually has is a set of operational systems that retain enough to do their own jobs, plus archives, plus logs with retention windows set by storage cost rather than by any analytic purpose. That is not a training corpus and the distance between the two is where the AI phase disappears.
Six specific problems recur, and each is cheap to prevent and expensive to fix retroactively.
The outcome was never recorded. A system records what was decided but not what happened afterward. Without the outcome there is no label, and without labels supervised learning has nothing to learn from. Programs discover this after they have already committed to a schedule.
Records are overwritten rather than versioned. An operational system that keeps only current state is answering its own question correctly. A model needs to know what the state was at the moment a decision was made, which means either event history or a snapshot cadence. Retrofitting history that was never kept is impossible, not merely difficult.
Nobody can say where a record came from. Without lineage, a model that behaves oddly cannot be investigated. Worse, an evaluation cannot be trusted, because nobody can rule out that the training set and the test set share a common source.
Free text carries the meaning and nothing structures it. Operator notes and remarks fields often contain the information a model would need. Preserved as raw text with a timestamp and an author, they are usable later. Truncated, normalized or dropped, they are gone.
Retention is shorter than the training window. A ninety-day log retention makes seasonal patterns invisible. Retention windows set for operational troubleshooting are almost never long enough for training, and extending them is a policy conversation with a lead time.
Access takes months. The data exists, is well kept, and no engineer can get to it without a chain of approvals that nobody started. Access design is part of data strategy, not an administrative afterthought.
What stops a program from training a model on data it already owns
Editorial weighting, illustrative rather than measured. The last row is deliberately low: raw volume is rarely the binding constraint.
The six things to build in the current phase
None of this requires knowing what model will eventually be trained. All of it is useful to the program on its own terms, which is what makes it fundable now.
An event log with outcomes. Record what was decided, by whom, on what basis, and what happened next. Append-only, timestamped, with a stable identifier that lets a decision be joined to its consequence. This one artifact is the difference between a program that can train a model and one that cannot, and it is also the best operational audit trail a program can have.
Schema contracts on every source. Each feed declares its structure. Arrivals are validated. A change that violates the contract is caught at ingestion, with a defined behavior: reject, quarantine, or accept with a flag and an alert. Silent schema drift is the quietest way for a data asset to rot, because nothing appears to break for months.
Lineage as a first-class field. Every record carries where it came from, when it arrived, which pipeline version processed it and what transformations were applied. This is inexpensive at ingestion and impossible to reconstruct later.
Storage in an open format with a catalog. Columnar files in object storage with a table format that supports schema evolution and time travel, plus a catalog that says what exists, who owns it, what it means and who may see it. A catalog nobody maintains is worse than none, so assign the ownership when the catalog is created.
A labeling capability, even if it is small. A workflow, written guidance, a way to measure agreement between labelers, and adjudication of disagreements. Starting this in the current phase on a modest scale is what makes it possible to scale it later, because the hard part is the guidance and the quality method, not the tooling.
Access designed rather than requested. Roles, an approval path with named owners, a de-identified or synthetic derivative for development use, and an environment where analysis is permitted. The goal is that a new engineer can be productive in days.
The labeling problem, in detail
Label quality sets the ceiling on model quality, and it is the part of a data strategy most often written as a single line in a plan. It deserves more, because it is where the effort and the risk actually sit.
Written guidance is the product. Two competent people looking at the same record will disagree about the right label more often than anyone expects, and every disagreement they resolve differently is noise the model has to fit. The guidance document, with worked examples of hard cases, is what reduces that. It gets revised continuously as new edge cases appear, and each revision needs a decision about whether earlier labels are re-done.
Agreement has to be measured, not assumed. Assign a sample of records to more than one labeler and measure how often they agree. That number tells you the ceiling on achievable model performance, and it tells you when the guidance needs work rather than the model.
Domain expertise cannot always be bought cheaply. Some labels require an operator's judgment. When that is true, the labeling budget is a claim on scarce people and the schedule has to reflect it. Designing the task so an expert adjudicates rather than labels everything is usually the way through.
Weak supervision buys a starting point. Rules, existing system outputs, and heuristics can produce noisy labels at volume that get a first model working while careful labeling proceeds on a smaller set. This is a practical way to start when expert time is the constraint.
Every label carries provenance. Who labeled it, when, under which version of the guidance, and whether it was adjudicated. Without that, a guidance change cannot be applied retroactively and label quality cannot be audited.
Splits, and why they are a data decision
The most consequential technical decision in a data strategy is how evaluation data is separated from training data, and it has to be made when the storage is designed rather than when the model is trained.
A random split measures whether a model can interpolate among records it has effectively already seen. That is almost never the operational question. The real question is whether the system will work on a new location, a new platform, a new unit, a later time period, or a different sensor configuration. The held-out set has to be split on whatever dimension the deployment will vary along, which means the storage layer has to carry that dimension as a field on every record.
Two related traps are worth naming. Leakage happens when a feature encodes the answer, often through a field populated after the outcome was known; the defense is to record when each field was populated relative to the decision. And near-duplication across the split boundary inflates results, which is why deduplication and a grouping key belong in the pipeline rather than in the modeling notebook.
None of that is model work. It is data engineering done with the eventual evaluation in mind, and it is the clearest example of why the data phase and the AI phase cannot be fully separated in planning even when they are separated in the budget.
Two ways to fund it
The practical obstacle is rarely disagreement about the value. It is that the AI benefit is in a future phase and the cost is in this one. Two framings work, and they can be combined.
| Framing | How it is justified | What the program gets now | What it enables later |
|---|---|---|---|
| Operational instrumentation | Better program insight, audit trail and troubleshooting | Dashboards, incident forensics, defensible decision records | The labeled event history a model trains on |
| Data quality and integration | Reduces manual reconciliation and errors between systems | Validated feeds, fewer downstream defects | Schema contracts and lineage the pipeline needs |
| Modernization of storage | Cost and sustainment of aging data stores | Open formats, a catalog, lower vendor dependence | Storage a training pipeline can read directly |
| Access and governance | Compliance, records obligations, controlled sharing | Faster approvals, a de-identified development set | Engineers productive in days rather than months |
| An explicit AI readiness line | Named future capability with a stated dependency | Less on its own; depends on the future phase surviving | Everything, if the phase is funded |
The first four are the durable route. They are justified by value the program receives in the current phase, which means they survive a budget review that the fifth may not. Writing the data work as AI readiness alone ties it to a phase that has not been funded yet, and reviewers notice.
What to write into the program plan
Specific language beats intent. Six things belong in the plan and the technical baseline.
- A data architecture section that names sources, formats, volumes and retention. With retention justified against analytic need rather than against storage cost alone, because extending it later is a policy conversation with a long lead time.
- Outcome capture as a requirement on the operational system. Not a nice-to-have in an appendix. If the system that makes decisions does not record what happened next, no later phase can recover it.
- Schema contracts and validation as an acceptance criterion. Written so that a feed is not accepted until its contract exists and validation runs.
- A catalog with named owners per dataset. Ownership assigned in the plan, because an unowned catalog stops being true within a quarter.
- An access model with approval paths and lead times stated. Including the de-identified or synthetic derivative that development uses, so engineering is never blocked on a queue.
- A labeling work package with a real basis of estimate. Even a small one. Showing labeling effort in the cost baseline is what tells a reviewer the program understands what the AI phase will cost.
Effort in the current phase, weighted by what it saves in the AI phase
Editorial weighting, illustrative rather than measured. The last row is deliberately low: choosing a model years early is the one decision worth deferring.
What not to do early
Restraint matters as much as investment, because a program that over-builds in the data phase spends credibility it will need later.
Do not choose a model architecture. It will be obsolete and the choice constrains nothing useful today. Do not buy a large platform on the theory that it will be needed; open formats in object storage with a catalog keep the options open at a fraction of the commitment. Do not build a feature store before there is a model to serve. Do not label at scale before the guidance has stabilized, because early labels made under unstable guidance often have to be redone. And do not promise an accuracy number in a plan, because a number asserted before the data has been examined is a commitment nobody can honor and everyone will remember.
How a specialist partner does this work
Precision Federal builds AI, data platforms, software and cloud systems and delivers them into production inside federal agencies. On a program positioning for a later AI phase we come in as the data engineering under the prime or directly on the program, and the shape is deliberately plain.
The first three weeks. We inventory the sources: what exists, what it means, who owns it, how it is stored, what its retention is, and how a record could be joined to its outcome. We produce three documents. A source register with owner, format, volume, retention and access lead time per feed. A readiness assessment saying, for each candidate future use, what data would be needed and what is missing today. And a proposed architecture with the storage layout, the schema contract mechanism, the lineage fields and the split dimension named. Those three make the data work estimable, and they are useful to the program whether or not we build it.
What we build after that. Ingestion with validation and quarantine. The event log with outcome capture. Lineage on every record. Storage in open formats with a catalog and assigned owners. The labeling workflow with guidance, agreement measurement and adjudication. The de-identified or synthetic derivative for development. Monitoring on freshness, volume and schema conformance, with alerts that reach a person. And the documentation a later phase needs to use any of it.
What the program keeps. The data, which was never ours. The code, delivered under a written assignment rather than a recital, with our pre-existing tooling named, carved out and licensed for use in the delivered system so no future maintainer is blocked. The catalog and the documentation. And the ability to have somebody else run the pipeline, which is what the automation and the runbooks exist to make true.
How it is priced. Fixed-price milestones where the scope is written, with acceptance stated as measurements rather than adjectives: a feed validated at a stated rate, a catalog covering named datasets, a labeling workflow with a measured agreement figure. A committed team at a named allocation where the backlog is negotiated. We do not price a defined outcome as hours and then discover the accountability question at acceptance.
How to start. One email with a one-page brief: what the program does, what systems produce data and who owns them, what the eventual AI use is expected to be, where the work would have to run, the phase boundary that matters, and who can approve a scope change. We come back with a scoped, priced statement of work.
Bottom line
The AI phase is decided in the current phase. Record outcomes joined to decisions, because that is the one thing no later effort can recover. Put schema contracts, validation and lineage at ingestion, where they are cheap. Set retention against the pattern length a model would need to see rather than against storage cost. Store in open formats with an owned catalog. Start a small labeling capability so the guidance and the quality method exist before scale is needed. Design access so a new engineer is productive in days. Justify all of it by the value the program gets now, not by a phase that has not been funded. A program that does that arrives at its AI phase with a corpus and a pipeline. A program that does not arrives with terabytes and a two-year problem.
Frequently asked questions
Six things, none of which require knowing what model will be trained. Record outcomes joined to decisions in an append-only event log. Put a declared schema contract and validation on every source, with a defined behavior on violation. Carry lineage as a field on every record. Store in open columnar formats with a catalog that has named owners. Stand up a small labeling capability so the written guidance and the agreement measurement exist before scale is needed. And design an access model with a de-identified development derivative so engineers are productive in days.
Usually because the operational systems answer their own questions correctly and a model needs different things. Outcomes were never recorded, so there is nothing to label. Current state overwrites history, so past context at the moment of a decision is gone. Lineage is missing, so evaluation cannot rule out that training and test data share a source. Retention is set for troubleshooting rather than for the pattern length a model would need. And access takes months. Raw volume is rarely the binding constraint.
Justify it by value in the current phase. Operational instrumentation gives the program dashboards, incident forensics and a defensible decision record, and incidentally produces the labeled event history a model needs. Data quality and integration reduces manual reconciliation. Storage modernization lowers sustainment cost and vendor dependence. Access and governance work speeds approvals. Each of those survives a budget review on its own merits, while a line item justified only as AI readiness is tied to a phase that has not been funded yet.
Enough to be visible in the cost baseline, because hiding it signals that the program has not done this before. The effort is dominated by the written guidance and the quality method rather than by the tooling. Plan for guidance with worked examples of hard cases, a qualification step for labelers, overlapping assignments on a sample to measure agreement, adjudication of disagreements, and provenance on every label recording who applied it under which guidance version. Where labels require operator judgment, design the task so experts adjudicate rather than label everything.
Because the split dimension has to exist as a field on every record, which is a storage decision rather than a modeling one. A random split measures interpolation among records the model has effectively already seen, which is almost never the operational question. If the system will face a new location, platform, unit, time period or sensor configuration, the held-out set has to be split on that dimension. Recording when each field was populated relative to the decision guards against leakage, and a grouping key in the pipeline guards against near-duplicates crossing the boundary.