The failure modes are boring, and they repeat
A small delivery team on an AI project almost never fails at the math. It fails on the calendar and on the record. A credentialing request sits with a customer's security office for five weeks. The evaluation holdout gets used for tuning in month two and nobody catches it until month five. One engineer holds the entire feature pipeline in their head and then takes two weeks of leave at the worst possible moment. The final report describes a system that has only ever run on one machine. Each of these is a management failure, and each is visible weeks before it becomes expensive if the week has a shape.
The arithmetic is unforgiving on short efforts. A six-month, $250,000 effort is roughly 26 weeks and, at a typical two-person loading, somewhere near 1,000 to 1,300 direct labor hours. The first three to five weeks are usually consumed by account provisioning, data-use agreements, a security questionnaire, and getting a working environment. A team that treats those weeks as blocker-clearing weeks with parallel work queued behind them finishes. A team that treats them as ordinary working weeks and discovers in week six that nothing is unblocked has already lost a fifth of the period of performance.
This piece describes how our team runs delivery on federal, state, and commercial AI work: the weekly rhythm, the review that catches technical drift while it is still cheap, the discipline that keeps evaluation numbers honest from kickoff to closeout, the documentation set that makes handoff real instead of ceremonial, and the specific habits that stop a project from resting on one person's head.
Where short AI projects actually lose schedule
Editorial weighting from public sources and practitioner reading, illustrative rather than a measured statistic.
The week is the unit of control
A month is too coarse to steer a six-month project. A daily standup is fine for coordination and useless for schedule risk, because nobody raises a five-week credentialing delay in a fifteen-minute round of what-I-did-yesterday. The week is where both live. It is long enough that a real piece of work either moved or did not, and short enough that eight bad weeks in a row cannot happen without somebody having to say so in writing eight times.
Our cadence is fixed and short. It costs about three hours per person per week, which is roughly seven percent of loaded time, and it buys back far more than that the first time it catches a dependency going sideways.
The weekly operating rhythm
Two details do most of the work. The Monday commitment is a single demonstrable outcome per person, phrased so that Friday produces a yes or a no. "Work on the ingest pipeline" is not a commitment. "Ingest the full March extract end to end and print row counts at every stage" is. The Wednesday dependency check assigns every external blocker an age in days and a named person, which converts a vague "still waiting on the customer" into "the ATO evidence request is 19 days old and sits with a named contracting officer's representative." An aging number is the only thing that reliably makes an organization act.
The Friday demonstration matters more than it looks. Running on the target environment every week, from week two, means integration is discovered continuously instead of in the last three weeks. Teams that demo on a laptop all project long are not measuring progress toward delivery, they are measuring progress toward a laptop.
Three numbers on every weekly note
Status prose hides trouble. Three numbers do not. The first is hours booked against hours planned for the period to date, expressed as a percentage of the total. The second is total blocked-days accumulated across open dependencies. The third is the number of days since the evaluation holdout was last opened, which we explain below.
Burn tracking is not only good practice on federal work, it is a clause obligation. Under FAR 52.232-20, Limitation of Cost, a contractor on a cost-reimbursement contract must notify the contracting officer in writing when it has reason to believe that costs in the next 60 days, added to costs already incurred, will exceed 75 percent of the estimated cost. FAR 52.232-22, Limitation of Funds, sets the same 75 percent trigger against the amount allotted on an incrementally funded contract. A team that does not know its burn to the week cannot meet either notice requirement, and a missed notice can leave work performed at the contractor's own risk.
The 75 percent notice is a calendar event, not a surprise
FAR 52.232-20(b) and FAR 52.232-22(c) both require written notice to the contracting officer at the 75 percent point, looking 60 days ahead. On a cost-type contract we put the projected crossing date on the schedule at kickoff and re-forecast it every Friday. The notice then arrives as a planned deliverable rather than as bad news.
What a technical review should examine
Most technical reviews are demonstrations with an audience, which means they surface exactly the things the presenter already knows work. A useful review is the opposite: one artifact, chosen in advance, read by an engineer who did not write it, with the author answering questions rather than presenting. Sixty minutes, one artifact. The artifact rotates: this week the ingest code, next week the evaluation code, the week after the deployment configuration or the interface contract.
The reviewer's job is not to admire the design. It is to try to run it, and to ask six questions that expose the failure modes that actually end projects.
| What the reviewer examines | The question asked | A failing answer |
|---|---|---|
| Reproducibility | Can this run be rebuilt from raw inputs on a clean machine using only what is in the repository? | "It works on the analyst's workstation." |
| Evaluation integrity | When was the holdout last opened, by whom, and for what? | "We checked against it a few times while tuning." |
| Error structure | What do the twenty worst cases have in common? | "Accuracy is 94 percent." |
| Interface contracts | What breaks upstream and downstream if this schema changes? | "We will sort that out at integration." |
| Security inheritance | Which controls does this component inherit from the platform, and which does it own? | "The cloud provider covers it." |
| Continuity | Could a second engineer take this over on Monday from the written record alone? | "Only one person really knows that part." |
The third row deserves emphasis. A single aggregate metric tells a reviewer almost nothing about whether a system is fit to field. Reading the twenty worst cases takes twenty minutes and routinely reveals that half the errors come from one malformed source file, one date-parsing rule, or one class that was never represented in training. That is a fixable finding. "94 percent" is not.
Reviews produce written findings with owners and dates, tracked in the same place as the code. A review that ends in verbal agreement produces nothing an auditor, a contracting officer, or a future engineer can see.
Keeping evaluation honest across a project
Evaluation integrity decays quietly. On day one, the team splits the data, sets a metric, and means every word of it. By month four the metric has been consulted a hundred times, thresholds have been nudged, features have been added because they helped on the set everyone agreed not to touch, and the reported number no longer estimates anything about future performance. Nobody cheated. The set simply got used.
Four practices hold the line, and all four are cheap.
Freeze the holdout before the first model exists. Split the data, hash the holdout, write the hash into the repository, and give the set an access log. Every opening is a logged event with a date, a person, and a reason. When the weekly note carries "days since holdout opened," the number is visible to the customer as well as the team, and a set that has been opened eleven times in four months cannot be quietly reported as a clean estimate.
Write the metric and the success threshold into the record at kickoff. Not "high accuracy" but the specific measure, the specific population, and the number that constitutes success, agreed before any result exists. This is the single most effective defense against a metric that migrates toward whatever the system turned out to be good at. NIST's AI Risk Management Framework puts the same idea in MEASURE 2.3: performance and assurance criteria should be demonstrated under conditions similar to the deployment setting, which is a requirement about the evaluation population, not only the score.
Segment every reported number. An aggregate can be carried entirely by the easy majority class. Report by site, by document type, by time period, by whatever partition the customer's operation actually cares about, and report the worst segment as prominently as the average. Buyers who have been burned before look for exactly this, and its absence reads as a tell.
Have someone outside the build team run the final evaluation. The engineer who tuned the system should not be the one who produces the number that goes in the report. On our projects the final scoring run is executed by a bench engineer who has not touched the model, from the repository, on a clean environment. If it does not run for them, it does not run.
The same discipline applies to anything a vendor reports to you. We have written separately on reproducibility as a procurement requirement, and the short version is that a number nobody can regenerate is not evidence.
Documentation that makes handoff possible
Handoff documentation written in the last week of a project is a summary, and a summary cannot be executed. The test for real handoff documentation is blunt: a competent engineer who has never seen the project should be able to clone the repository, follow the written instructions, and reach a working system with passing tests inside a day. Anything short of that is a description of work rather than a transfer of it.
Federal contracts often specify the artifacts by name. Data Item Descriptions still in active use include DI-IPSC-81427A for the Software Development Plan, DI-IPSC-81442A for the Software Version Description, and DI-MGMT-80227 for the contractor's progress, status, and management report. Under DFARS 252.227-7014, "computer software documentation" is its own deliverable category with its own rights, distinct from the software itself, which means a delivery that ships code without the documentation that lets someone maintain it can be nonconforming even when the code works. Where transition between contractors is contemplated, FAR 52.237-3, Continuity of Services, obligates orderly phase-in and phase-out support, and that obligation is only meetable if the record exists.
The package we build during a project, not after it:
- Environment lockfiles and container image digests for every runtime, pinned by hash
- Data lineage from raw source to final table, including every filter, join, and exclusion rule and why it exists
- A model card: training data, intended use, out-of-scope use, known failure modes, measured performance by segment
- The frozen evaluation set with its access log and hash
- Runbooks for deploy, rollback, retrain, and first-hour triage of the three most likely failures
- A decision log: every architectural choice, the option it beat, and the constraint that decided it
- A test suite a new engineer can run to green on a clean machine in under an hour
- An SBOM in SPDX or CycloneDX for every delivered component
The decision log is the item teams skip and the item successors need most. Code shows what was built. It never shows what was rejected, and a successor who does not know that the obvious approach was tried and failed for a documented reason will spend three weeks rediscovering it. Two sentences written the day a decision is made are worth more than two pages written six months later. The SBOM is not optional on federal software: Executive Order 14028 and the NIST Secure Software Development Framework, SP 800-218, treat component provenance as a delivery requirement rather than a nicety, and OMB has required attestation to those practices for software supplied to federal agencies.
Killing the single-expert dependency
Every technical organization concentrates knowledge, and small delivery teams concentrate it fastest, because the natural move is to give each problem to whoever solves it quickest. Six weeks in, one engineer owns the data pipeline completely and no one else can debug it. That is a continuity risk on a commercial project and a contractual one on a federal project, where the individual may be named in the proposal and bound by a key-personnel clause.
Three habits, applied from week one, largely dissolve it.
No unpaired subsystem. Every meaningful component has a primary and a named second who has actually run it, not merely read about it. The second's qualification is behavioral: they have executed the component end to end, on their own machine, without the primary present. Reading the code is not qualification.
Rotate the review seat, not just the reviewer. The Thursday review is read by whoever did not write the artifact, and over a six-month project that puts every engineer inside every subsystem at least twice. Rotation is also how a review stops being a formality: a reviewer who will own the component next month reads it very differently from one who will not.
Write while building, and make the writing load-bearing. Runbooks that are followed during real deployments get corrected. Runbooks that live in a document nobody executes are fiction. We deploy from the runbook every time, including the times when the primary could do it from memory, precisely so the document stays true. The same logic applies to the test suite: it is the executable half of the documentation, and a suite that only passes in one person's environment is telling you the project has a continuity problem regardless of what the staffing chart says.
On the contract side, key-personnel provisions are agency-specific rather than a single FAR clause. NASA uses NFS 1852.235-71, HHS uses HHSAR 352.237-75, and many DoD solicitations write their own. What they share is a requirement for written contracting-officer approval before a named individual is substituted, usually with a resume of equal or better qualification. Our depth of bench is what makes that survivable: when a named engineer must be replaced, the replacement is a named person with comparable qualifications who is already inside the project record, not a resume found after the fact. A fuller treatment of the clause mechanics is in our piece on what key personnel clauses bind.
What a buyer should ask for before award
If you are buying AI delivery rather than performing it, the cadence above is something you can require in the statement of work, and requiring it costs you nothing at award while saving you a great deal at month five. Four provisions do most of the work.
A weekly written note with three numbers. Burn against plan, open dependency ages with named owners on your side, and days since the evaluation holdout was opened. Ask for it as a one-page deliverable. A vendor who cannot produce it weekly is not tracking it.
A demonstration on your environment starting in week two. Not a slide, a running system, however small. This is the single clause that prevents integration surprise at closeout.
A named second for every key-personnel role. Written into the staffing plan at proposal, with the qualification standard stated as having executed the component rather than having read it.
A handoff dry run in the second-to-last week. An engineer outside the build team clones the repository and stands the system up from the written record while there is still time to fix what is missing. A handoff verified after final payment is not verified.
Those four items also make good acceptance criteria on a fixed-price engagement, where the definition of done carries more weight than the level of effort. We have written on what a fixed-price AI engagement should deliver for the contracting side of that question.
Bottom line
The cadence is not administration and it is not process for its own sake. It is a set of instruments that make the three project-killing failures visible while they are still cheap: a dependency aging past the point of recovery, an evaluation number quietly losing its meaning, and knowledge collecting in one person's head. Three hours a week per person buys all three signals. On a six-month effort, catching any one of them in week six instead of week twenty is the difference between a delivery and a report about why there was no delivery.
Small teams can run this rhythm better than large ones, because the loop from signal to correction is short and there is no committee between noticing and fixing. That advantage only exists if the rhythm exists.
Common questions on running the cadence
Is a weekly technical review too much overhead for a six-month project?
It is roughly one hour per person per week, about two percent of loaded time. The comparison is not review time against zero, it is review time against the cost of finding an integration defect or a contaminated evaluation set in the final month, when the only remaining options are unpaid rework or a diminished delivery. One review that catches one such defect pays for the entire project's reviews.
What if the customer cannot give us their real data?
Then the schedule assumes it, and the cadence tracks the gap explicitly. We build against a documented synthetic or public proxy with the schema pinned to the customer's actual schema, keep the ingest layer swappable, and carry the data dependency as an aging item on every weekly note so the delay is visible on the customer's side too. The one thing we do not do is report proxy-data performance as if it were performance on the real corpus.
Does this survive an air-gapped or classified environment?
The rhythm holds, the artifacts move differently. Written logs and review findings live inside the accredited enclave, the Friday demonstration happens on the enclave rather than over a screen share, and the weekly note to the customer carries only what is releasable. What actually changes is the dependency clock: provisioning, media transfer, and review of anything leaving the enclave all take longer, so those items get more schedule and tighter tracking, not less.
Frequently asked questions
Weekly, on a rotating artifact, for about an hour. Monthly reviews are too infrequent to catch drift while it is cheap to correct, and reviews held only at milestones become presentations rather than examinations. The reviewer should be an engineer who did not write the artifact under review.
Whether the run can be rebuilt from raw inputs on a clean machine, when the evaluation holdout was last opened and why, what the worst twenty cases have in common, what breaks if an interface schema changes, which security controls the component owns versus inherits, and whether a second engineer could take it over from the written record.
Freeze and hash the holdout before the first model exists, log every access to it, write the metric and success threshold into the record at kickoff, report performance segmented by the partitions the operation cares about, and have the final scoring run executed by someone outside the build team on a clean environment.
Pinned environment lockfiles and image digests, data lineage from raw source to final table, a model card with segmented performance and known failure modes, the frozen evaluation set with its access log, deploy and rollback runbooks, a decision log, a test suite that runs to green on a clean machine, and an SBOM in SPDX or CycloneDX. Federal deliveries often name these through Data Item Descriptions on the contract data requirements list.
Give every meaningful component a primary and a named second who has executed it end to end alone, rotate the review seat so every engineer reads every subsystem, and deploy from the runbook every time so the written procedure stays true. On federal work this also protects the key-personnel position, since substitutions generally require contracting-officer approval with an equally qualified named replacement.