The workstream went yellow four months ago and nobody moved, because yellow on a program of this size is normal and the team said it was close. Now the model does not perform on the customer's data the way it performed in the demonstration, the pipeline breaks on inputs nobody anticipated, the delivery is late twice over, and the last technical exchange with the customer went badly. The program manager is not looking for an explanation. They are looking for someone who can say, with dates, what happens next, and then be right about it.
Recovering an AI or data workstream is a different job from building one. The constraints are harsher, the political situation is real, and the most valuable thing anyone can produce in the first month is not code. It is a measurement that everyone believes. This is written for the program manager or chief engineer who owns a red workstream and has to decide, this week, what to do about it.
Why AI workstreams go red in a particular way
These failures have a recognizable shape, and it is worth naming because the shape tells you what to fix.
The demonstration was measured on the wrong thing. The team reported a headline accuracy number on a held-out split of a dataset that was assembled for convenience. The customer's real workload has a different class balance, different data quality, and cases the training set never contained. The number was not dishonest; it answered a question nobody in the mission actually asks. When the system meets real inputs, the useful metric is far worse than the reported one, and the team has no way to explain the gap because the useful metric was never computed.
There is no measured baseline for the current process. Nobody wrote down how well the humans do today, at what cost and at what volume. Without that, no result is interpretable. A component that is right seventy percent of the time is a triumph or a disaster depending entirely on a number nobody collected, and in the absence of that number the conversation becomes about impressions.
The pipeline was built for one pass, not for operation. The path from raw data to a trained model exists as a sequence of steps someone ran by hand, in an order they remember. It cannot be rerun reliably, its intermediate outputs are not versioned, and nobody can reproduce the model that is currently deployed. This is the failure that makes everything else unfixable, because you cannot improve what you cannot rebuild.
The component was built outside the environment it must live in. It runs on tooling that is not accredited, calls a service the boundary does not permit, or assumes access to data in a form the production system does not produce. The engineering may be fine and the thing still cannot ship.
The deliverable was defined in adjectives. The contract data requirement says the system will provide accurate automated analysis. There is no threshold, no named dataset, no test. So acceptance is a matter of opinion, and once the customer's confidence has slipped, opinion goes one direction.
What is usually actually wrong when an AI workstream goes red
Editorial weighting, illustrative rather than measured. The last row is deliberately low: the model choice is almost never the problem, though it is almost always the first thing proposed as the fix.
That last row deserves emphasis, because it governs where a recovery spends its money. When a workstream is red, the instinct in the room is to change the model. A newer architecture, a larger model, a different vendor. That change is visible, it is easy to authorize, and it almost never fixes anything, because the problem is in the data, the measurement or the path to production. A recovery that starts by swapping models buys three weeks of hope and arrives back at the same place with less schedule.
The two-week assessment
The first move is a bounded assessment with a hard stop and a written finding. Two weeks, not six. The purpose is to convert a situation everyone has an opinion about into a set of facts everyone can act on. It has five parts and they run in parallel.
Reproduce the current result. Take the deployed or demonstrated model and try to rebuild it from what exists: the code, the data, the configuration. Whether this succeeds is itself one of the most informative findings of the assessment. If it succeeds, the engineering foundation is sound and the problem is narrower than it looked. If it fails, that is the first item in the recovery plan, because everything else depends on it.
Measure the real baseline. Find out how the work is done today and how well. Sample actual cases, have the current process handle them, and record outcomes. Where a human decision is the baseline, measure agreement between reviewers as well, because a process where two qualified people disagree on a fifth of cases sets a ceiling on what any automated component can be asked to achieve, and knowing that ceiling changes the whole conversation.
Build an honest evaluation set. Not a random split of whatever data the team had. A set drawn from the production workload, covering the classes and edge cases the mission actually contains, held out and frozen. This is the single most valuable artifact the assessment produces, because from the moment it exists every claim about performance is checkable by anyone.
Measure the current component against it. Now the headline number can be replaced by a real one, broken out by the categories that matter to the mission, with the errors examined rather than summarized. Reading fifty actual failures is worth more than any aggregate statistic, and it usually reveals that the errors cluster in two or three identifiable situations rather than being spread evenly.
Trace the path to production. Where does the component have to run, what does the accreditation boundary permit, what does the release pipeline require, and which of those conditions does the current build fail. This is often where the real schedule risk lives, and it is the part most likely to have been deferred.
What the written finding contains
The output is a document the program manager can hand to the customer without rewriting it. That constraint shapes it: no internal blame, no vendor politics, no hedging. It answers six questions.
- What the current process achieves. The measured baseline, with the method and the sample described so a reader can judge it.
- What the current component achieves. Measured on the frozen evaluation set, broken out by mission-relevant category, with the dominant failure situations named.
- What is worth keeping. Specifically. Data engineering, interfaces, labeled data, domain rules encoded by the team, integration work already done. There is nearly always more here than the mood in the room suggests.
- What has to be replaced, and why. With the reason stated in engineering terms rather than as a judgment about the previous team.
- What the component would have to reach to be worth deploying. A threshold tied to the mission decision, at a review volume the staff can absorb, agreed with the people who will use it.
- A recovery plan with dates. Milestones, each with a testable exit condition, and the first one no more than three weeks out.
Where the honest answer is that the component cannot reach a useful threshold on the data available, the finding says so and proposes the alternative: a narrower scope where it can, a different decision to assist, or a data collection effort that has to happen first. A recovery that reports a problem is worth far more to a program manager than one that promises a schedule it will miss, and it is also, in the medium term, the thing that keeps the customer.
What to keep and what to replace
Recoveries fail in both directions. Keeping everything means inheriting the constraints that caused the failure. Replacing everything means spending the schedule rebuilding work that was fine, and it usually reflects an engineer's preference rather than an analysis.
| Asset | Usual verdict | Reason |
|---|---|---|
| Labeled data and annotation guidelines | Keep, and audit a sample | The most expensive artifact on the workstream; label quality is checkable and fixable |
| Domain rules and business logic encoded by the team | Keep | Represents mission knowledge that took months to extract and is not in any document |
| Integration code and interfaces to the host system | Keep where tested | Integration work is slow to redo and rarely the source of the failure |
| The training pipeline | Rebuild, usually | If the model cannot be reproduced, nothing downstream can be improved with confidence |
| The evaluation approach | Replace | The measurement that produced a misleading result is the thing that must not survive |
| The model itself | Retrain, rarely re-architect | Architecture is seldom the binding constraint; data and thresholds usually are |
One more asset belongs on that list and is frequently discarded by accident: the people on the incumbent team who understand the customer's domain. On a recovery, the engineer who spent eight months learning how the mission actually classifies its cases is a resource, not a liability, and a recovery team that treats the previous team as the problem loses that knowledge and gains an enemy inside the program.
The engineering that makes the recovery hold
Beyond the assessment, the technical work of a recovery is mostly about making the system legible and rebuildable. Five things, in order.
One command rebuilds the model. From a versioned data snapshot, through versioned code, to a model artifact with its evaluation results attached. Until this exists, every improvement is an anecdote. After it exists, improvement becomes an ordinary engineering loop, and the pace of the workstream changes visibly within two weeks, which matters politically as much as technically.
The evaluation set is frozen and versioned like code. It sits in the repository, it does not change without a recorded decision, and every result is reported against a named version of it. This is what makes the numbers in the recovery credible to a customer who has been given numbers before.
Errors are categorized, not just counted. Every failure on the evaluation set is assigned to a category, and the category counts are tracked release over release. This turns model work from guessing into targeting: the team can see that half the errors come from one document type, fix that specifically, and demonstrate the effect.
The component sits behind the host system's service layer, with the same identity, logging and failure handling as any other dependency, and every inference is written to the operational record alongside what the human did afterward. That record is the source of accuracy measurement in production, the source of drift detection, and the answer when an oversight question arrives about a specific case. Where the previous build lacked it, adding it is usually a week of work that pays for itself immediately.
The path to production is exercised in the first milestone, not the last. Even with a component that performs poorly, push it through the accredited release pipeline into the target environment behind a flag that keeps it off. Doing this early converts the largest unknown on the workstream into a known, and it removes the failure mode where a technically successful recovery cannot ship.
Presenting the recovery to the customer
The customer has been told before that this was close. Whatever is said next is heard against that history, so the presentation has to be structured to earn back credibility rather than to restore enthusiasm.
Four rules make this work. Lead with the measurement, not the plan. Open with the baseline of the current process and the honest measured performance of the current component, on a named evaluation set, with the method stated. A customer who sees a program measuring itself accurately, including where it looks bad, updates their view of the program more than any commitment could.
Name the threshold and who agreed to it. Say what the component must reach to be worth deploying, tied to the mission decision and the review capacity, and say which users agreed that threshold is the right one. This converts acceptance from an opinion into a test.
Give dates with exit conditions, and make the first one soon. A milestone three weeks out with a testable exit condition is worth more than a plausible plan that ends in six months. Then hit it. The second delivered milestone is where the customer's posture changes; the first only buys the chance.
Say what happens if it does not reach the threshold. Naming the fallback in advance, and the decision point at which the program would take it, is what separates a recovery from a promise. It also removes the customer's worst fear, which is not that the component underperforms but that they will find out too late to do anything.
What restores a customer's confidence, in order of effect
Editorial weighting, illustrative rather than measured. The last row is deliberately low: a customer who has already been given one plan does not update on a second one.
How we run a recovery inside your contract
Precision Federal builds AI, data platforms and production software and delivers them into federal agencies. On a red workstream we come in as a specialist subcontractor under the prime's contract, working inside the prime's boundary and reporting through the prime, and we work to a shape that is deliberately easy to stop.
The first two weeks. The assessment above, at a fixed price, ending in the written finding: measured baseline, measured current performance on a frozen evaluation set, what to keep, what to replace, the threshold worth deploying against, and a milestone plan with dates. The finding is written so the prime can hand it to the customer without editing it. If the finding is that the work should be descoped rather than continued, it says that plainly, and the program has spent two weeks to learn it instead of five months.
Then delivery. Fixed-price milestones tied to the exit conditions in the plan, so the prime carries no schedule risk from our side, or a committed team at agreed labor categories mapping into the prime's cost structure and the contract's rate schedule where the workstream is durable rather than bounded. The first milestone is three weeks out and includes something running in the target environment.
What the prime keeps. The code, the models, the training and evaluation pipelines, the frozen evaluation sets, the infrastructure definitions and the documentation, under the prime's data rights posture as the government contract requires, transferred by a present written assignment rather than a work-for-hire recital. The customer relationship stays entirely with the prime; our engineers speak to the government through the channel the prime designates, and bad news reaches the prime before it reaches anyone else. We are comfortable being named as a subcontractor with a defined scope or working behind the prime's brand.
How to start. One email with a one-page brief: the system and the component, what the deliverable requires, what data exists and who grants access, the environment and the accreditation it holds, the contract instrument, and the date that matters. We return a scoped, priced statement of work for the assessment and an outline of what follows it.
Five ways a recovery fails
It starts by changing the model. Visible, easy to authorize, and almost never the binding constraint. It costs three weeks and returns to the same position with less schedule remaining.
It skips the baseline. Without a measured picture of how the work is done today, no result can be interpreted and the acceptance conversation stays a matter of opinion, which is the environment the workstream already failed in.
It rebuilds everything. Discarding integration work, labeled data and encoded domain knowledge in favor of a clean start spends the remaining schedule on ground the program already covered.
It treats the incumbent team as the problem. The people who learned the mission are the cheapest source of domain knowledge available, and turning them into adversaries costs the recovery more than any technical mistake.
It defers production until the component works. The largest unknown on a red workstream is usually whether the thing can be deployed at all. Exercising the release path in the first milestone removes that risk while there is still time to act on the answer.
Bottom line
A red AI workstream is recovered by measurement before engineering. Spend two weeks reproducing the current result, measuring how the work is done today, building a frozen evaluation set drawn from the real workload, measuring the current component against it, and tracing the path into the accredited environment. Write a finding the prime can hand to the customer unchanged: what is kept, what is replaced, what threshold is worth deploying against, and milestones with testable exit conditions, the first one three weeks out. Then make the model rebuildable with one command, categorize the errors, push something through the release path early, and hit the dates. Programs recover credibility by delivering the second milestone, not by promising the sixth.
Frequently asked questions
A bounded two-week assessment with a hard stop, not more development. Reproduce the current model from source, measure how the work is done today, build an evaluation set drawn from the real production workload and freeze it, measure the current component against that set with errors examined rather than summarized, and trace what the accredited environment and release pipeline require. The output is a written finding a program manager can hand to the customer without rewriting it.
Usually because it was measured on the wrong thing. A headline number computed on a convenient dataset answers a question the mission does not ask. The production workload has a different class balance, different data quality and cases the training data never contained, and the metric that matters is often the miss rate in one important class at a review volume staff can absorb rather than overall accuracy. Building a frozen evaluation set from the real workload replaces impressions with a checkable number.
The pipeline, in most cases. If the deployed model cannot be rebuilt from versioned data and versioned code, no improvement can be verified and every result is an anecdote. Model architecture is rarely the binding constraint even though it is the most common first proposal. Keep labeled data, encoded domain rules and tested integration code; rebuild the training pipeline; replace the evaluation approach that produced the misleading number in the first place.
Lead with measurement rather than plan. Show the baseline of the current process and the honest measured performance of the component on a named evaluation set, including where it looks bad. Name the deployment threshold and which users agreed it is the right one. Give milestones with testable exit conditions and make the first one about three weeks out. Then state the fallback and the decision point for taking it, so the customer's real fear, finding out too late, is answered.
The assessment is best bought as a fixed price for a fixed two weeks ending in a written finding, which keeps it easy to stop. Delivery afterward is either fixed-price milestones tied to the exit conditions in the plan, putting schedule risk on the subcontractor, or a committed team at agreed labor categories that maps into the prime's cost structure and the contract's rate schedule where the workstream is durable rather than a bounded fix.