What the review is for
An independent review of an AI roadmap is a few weeks of someone reading the plan who has no stake in the answer, testing the assumptions underneath it, and writing down which ones are load-bearing. It is not a second opinion on the strategy. Whether the company should be doing document automation before customer service, or building a data platform before either, is the executive's call and depends on things a reviewer will never see. The review has a narrower job: find the assumptions the plan cannot survive being wrong about, say what evidence would settle each one, and report which of them nobody has checked.
That framing keeps the work bounded. A strategy engagement is open-ended and expensive; an assumption review is not. There are maybe a dozen things a three-year AI plan is betting on, most can be tested against evidence that already exists somewhere in the building, and the job is finished when the register is written and walked through. The reason to run it before the budget commits is arithmetic: a wrong assumption found in month two costs a re-plan, and the same assumption found in month twenty costs the sunk spend, the re-plan, and the credibility of whoever signed the original. Plans rarely fail at the moment the flaw appears. They fail later, once it has been built on.
The federal government wrote down the checklist
In April 2026 the Government Accountability Office published Artificial Intelligence Acquisitions: Agencies Should Collect and Apply Lessons Learned to Improve Future Procurements (GAO-26-107859). GAO conducted in-depth reviews of 13 AI acquisitions at four agencies, the Department of Defense, the Department of Homeland Security, the General Services Administration and the Department of Veterans Affairs, and interviewed senior acquisition leaders at a fifth. The report groups what it found into six recurring problem areas: access to subject matter experts, protections for government data and intellectual property rights, and traditional acquisition time frames on the strategic side; requirements definition and contract terms, early testing and continuous evaluation, and AI pricing and overall cost on the programmatic side.
The first of those deserves attention from anyone in the private sector who assumes this is a government problem. Officials from every one of the five agencies GAO looked at said that limited access to subject matter experts, specifically data scientists and software engineers, made it hard to procure AI well. GAO records that officials on one Veterans Affairs decision-support effort struggled to write effective evaluation factors for source selection because they did not have the technical expertise on hand. The people with the budget did not have, inside the building, the person who could tell them whether what was being offered was any good. Commercial buyers land in the same position more often than they say out loud, and the reason is structural.
The Office of Management and Budget's guidance on AI acquisition, M-25-22, issued April 3, 2025 and rescinding the 2024 memo M-24-18, reads almost like a review checklist. It tells agencies to test proposed solutions before award; to run independent evaluations on data the buyer defines and the vendor cannot access; to write contracts that detail how the vendor tests and that do not prohibit the buyer from internally disclosing those methods or results; to require performance standards a new version must meet before it deploys, with a rollback if it does not; to require notice before new features are integrated; and to include specific protections against lock-in. None of that is unique to federal procurement. A commercial buyer can put the same clauses in a master services agreement, and most do not.
Where a multi-year AI plan carries its risk
Our editorial weighting of how often each assumption is the one that breaks a plan, drawn from public reporting and practitioner reading. Illustrative ordering, not a measured statistic.
Assumption one: the data is ready when the schedule says it is
Nearly every AI plan has an early line that reads like data preparation and a later line that assumes it finished. The second line is where plans slip, and the slip is rarely caused by anything exotic. The join key between two systems turns out not to exist. The field everyone assumed was populated is populated in 40 percent of rows. The historical archive is real but the labels on it were applied by a process that changed twice. The system of record belongs to a team with its own roadmap and no reason to prioritize yours.
A review does not solve any of that. It counts. How many rows, over what period, with what field-level completeness. Whether a labeled set exists, who labeled it, and whether two labelers agree. Whether anyone has pulled a sample and looked at it recently, or whether the description of the data is inherited from a slide made two years ago. These questions are cheap, and they are almost never asked before the budget is set, because asking them is somebody's afternoon and nobody owns the afternoon.
GAO has a precise example of the cost of not writing this down. Federal Emergency Management Agency officials working with a geospatial imaging vendor found the model had trouble distinguishing between different types of dwellings, and expected accuracy problems on imagery from regions unlike the imagery it was trained on. Neither observation was documented. The agency then continued with that vendor for another twelve-month period, and GAO notes that other personnel using the same contract elsewhere may not benefit from what was already learned. The knowledge existed inside the building. It did not reach the next decision.
Assumption two: the model named in the plan will still exist
A three-year plan that names a specific model version has named something with a shorter published life than the plan. This is not speculation; the vendors publish the schedule. Anthropic's model deprecation policy commits to at least 60 days' notice before retiring a publicly released model, and the record shows what that looks like in practice: Claude Sonnet 3.7 was announced for retirement on October 28, 2025 and retired February 19, 2026, and Claude Opus 4.1 was announced June 5, 2026 and retired August 5, 2026. The same page notes that partner-operated platforms such as Amazon Bedrock and Google Cloud set their own retirement schedules, so the identical model can carry different end dates depending on where it runs. Every major provider publishes something comparable, with its own notice periods and its own dates.
The breakage is not only at the model level. Anthropic's documentation also records parameter-level deprecations: temperature, top_p and top_k now return a 400 error when set to a non-default value on Claude Opus 4.7 and later models. Code that ran correctly against one generation returns an error against the next. That is a small, concrete instance of a general property. The interface is not stable over the life of a multi-year plan, and a plan that assumes it is has a maintenance line item nobody costed.
The fix is not to avoid naming a model. It is to write the plan so the model is a replaceable component behind an interface the organization controls, with a frozen evaluation set that can be run against a candidate replacement in an afternoon rather than a quarter. That one design choice turns a forced migration from a project into a task. GAO records that GSA officials supporting the USAi platform test leading models against a battery of performance tests over time, precisely to judge whether a new version justifies what the report describes as often much higher prices. A newer model is not automatically better for your task, and without a standing evaluation there is no way to find out except in production.
Assumption three: the price in the plan is the price
Officials from all five agencies in the GAO review said understanding AI costs was hard, and cited uncertainty in how vendors price licenses and services. The report gives a number worth sitting with. On the Army's XM-30 effort, which puts AI-enabled targeting into minimally crewed infantry combat vehicles, proposals included AI licensing at around $300,000 per vehicle per year. GAO records that this would have come to more than $500 million each year in licensing fees alone, on top of the cost of acquiring the vehicles. The Army rejected that proposal and is exploring other licensing models.
Nothing about that failure is exotic. A per-unit price that looks reasonable in a pilot becomes a different number multiplied by a fleet, and the multiplication usually happens in a spreadsheet nobody circulated. The quieter finding in the same section: officials on the Army's Project Linchpin told GAO that buyers often underestimate total cost because they price the training of models and overlook the infrastructure needed to support the capability over time, and that programs find it hard to forecast how enterprise costs such as cloud and compute will change. GAO's conclusion is blunt, that agencies may find the AI solutions they acquire too expensive to sustain.
So the cost work in a review is mechanical and it is about the second and third year, not the first. What is the unit of pricing, seat or call or token or document or vehicle or endpoint, and what does the bill look like at the volume the plan promises rather than the volume in the pilot. What does inference, monitoring, retraining and storage cost in steady state. Is the quoted discount attached to a term commitment, a volume floor, or a consolidation of spend, and what is the renewal price once it lapses. A plan that has priced only the build has priced the cheapest part.
Assumption four: someone on your side can tell whether it worked
GAO reports that NIST officials told them well-defined, universal AI tests do not yet exist, given the variation and complexity of AI services. That is the honest state of the field, and it has a direct consequence: the measurement has to be built for your task, before you need it to settle an argument. There is no standard test to fall back on, and a benchmark the vendor ran on data the vendor chose answers a different question than the one you are asking.
M-25-22 states the discipline in three parts that translate directly into commercial contracts. Independent evaluations must use data the buyer has defined; that data should not be accessible to the vendor and should be as close as possible to what the system will see in deployment; and the contract must detail the vendor's examination, testing and validation procedures, and must not prohibit the buyer from internally disclosing how the vendor tests or what the results were. As three clauses in a master services agreement, they cost nothing before signature and are close to impossible to add after.
What a review checks is whether any of it exists yet. Is there a held-back set of real items, scored by people who know the work. Was a threshold written down before the run, or decided afterward by looking at the result. Is there a baseline for how the current process performs, so a comparison is possible at all. Are there enough scored items that a difference can be separated from noise. When all four answers are no, the plan does not have a measurement problem in year two. It has one now.
| Assumption in the plan | What the review asks to see | A weak answer sounds like |
|---|---|---|
| The data is ready | Row counts, field-level completeness, a real sample looked at this quarter, the owner of the system of record by name | "The data is in the warehouse." No one has queried it against the plan's requirements. |
| Quality is good enough | A labeled set, who labeled it, and an agreement rate between two labelers on the same items | "Our SMEs know the domain." Labels exist but no one has measured whether the SMEs agree with each other. |
| The model choice holds | The vendor's published deprecation page, the interface boundary, and an evaluation set that can be re-run against a replacement | "We standardized on one model." The model version is named in the architecture and in nothing else. |
| The price scales | A cost model at pilot, target and stretch volume, with inference, monitoring, retraining and storage in years two and three | "The vendor gave us a good rate." The rate is per unit and the unit count in year three was never multiplied out. |
| We will know if it works | A held-back scored set, a threshold written before the run, a baseline for the current process, and enough items to beat noise | "We will measure adoption." Usage is counted; correctness is not. |
| We can change our mind | What is owned at the end, what is portable, the cost and elapsed time of moving, and the renewal price | "There is an exit clause." The clause permits leaving; it does not make leaving affordable. |
Dependencies that keep their own schedule
Plans get built as if every date is inside the organization's control. Several are not, and the compliance ones tend to sit on the critical path. A review's job here is to identify which milestones depend on a calendar somebody else keeps, and to check that the plan cites the current version of it rather than the version that was current when the deck was first written.
FedRAMP is mid-transition. FedRAMP 20x has run a Low pilot that closed in the second half of fiscal 2025 with 26 submissions and a Moderate pilot in the first half of fiscal 2026 with 14 qualifying submissions. The program's published plan puts wide-scale adoption in the second half of fiscal 2026, with consolidated rules targeted for June 2026 and the public submission pipeline opening between July and September 2026, a Class D (High) pilot estimated for the first half of fiscal 2027, and Rev5 end of life estimated for the second half of fiscal 2027. Any plan whose federal revenue depends on an authorization date should say which of those doors it is walking through.
The governance references are moving. The NIST AI Risk Management Framework, AI 100-1, was published January 26, 2023, with the Generative AI Profile, AI 600-1, following July 26, 2024. NIST states the framework is being revised as part of the White House AI Action Plan, and on April 7, 2026 released a concept note for a profile covering trustworthy AI in critical infrastructure. On the policy side, OMB issued M-25-21 and M-25-22 on April 3, 2025 and M-26-04 on December 11, 2025. A plan drafted in 2024 that still cites M-24-18 is citing a rescinded memo, which is the kind of detail that survives for years because nobody re-reads the citations.
The European dates changed once already. The EU AI Act entered into force August 1, 2024, its prohibitions applied from February 2, 2025, and the general-purpose model and governance rules from August 2, 2025. Obligations for high-risk systems were then rescheduled by a simplification amendment; the Commission's published timeline now places the Annex III high-risk categories, which include biometrics, critical infrastructure, education, employment and border control, at December 2, 2027, and high-risk systems embedded in regulated products at August 2, 2028. The law here has already moved once, so a roadmap that hard-codes a compliance date should carry the source and the date it was last checked beside the number. ISO/IEC 42001:2023, for organizations pursuing certification instead, is an auditable management-system standard, and an audit cycle is a calendar commitment rather than a document.
Exit cost is a design decision made early
The most expensive property of an AI system is usually how hard it is to leave, and that property is set in the first architecture conversation rather than at renewal. M-25-22 is unusually specific about this. It directs that solicitations reflect an interest in proposals reducing lock-in risk through knowledge transfer, data and model portability, clear licensing terms and pricing transparency, and that contracts include terms giving the buyer rights to code and models produced in performance of the contract.
A footnote to the memo lists the practices worth treating as evaluation criteria: well-defined application programming interfaces that let the acquired system interoperate with the rest of the stack; documentation of model development decisions, languages, testing scripts and protocols in a form that lets the work transition from one vendor to the next; open-source licensing where it applies; pricing that is uniform and publicly available rather than shaped by bulk arrangements, tying, steering, minimum spend requirements or exclusive discounts that concentrate spend with one vendor; and disclosure of which subcontractors and system integrators were engaged and how their involvement affects price.
For a commercial buyer the translation is four questions, and a plan that cannot answer them carries an unpriced liability. What do we own at the end. What can we take with us, in what format. What does moving cost, in money and elapsed weeks. What is the renewal price once the initial discount lapses.
What the review hands back
A review that produces a slide deck of observations has failed. The output is a small set of artifacts a decision can be made against, each short enough that the sponsor actually reads it.
- An assumption register listing every load-bearing assumption, the evidence that currently supports it, and whether that evidence is measured, inferred, or absent.
- A dependency map with the items outside the organization's control marked, each carrying the date it was last verified and where the date came from.
- A measurement plan naming the held-back set, who scores it, the threshold, the baseline, and the number of items needed to separate a real difference from noise.
- A cost model at three volumes covering years one through three, including inference, monitoring, retraining, storage and the renewal price.
- An exit assessment stating what is owned, what is portable, and the cost and elapsed time of moving.
- A sequencing recommendation where the order is wrong, with the specific dependency that forces the change.
- A short list of what would change our mind, so the findings are falsifiable rather than rhetorical.
How a four-week roadmap review runs
How to buy a review that stays independent
Separate the reviewer from the builder. A firm reviewing a plan it hopes to implement is answering a different question than the one asked. If the same firm might later build, the review must be a fixed fee that does not change with the finding, and the implementation decision must be made after the report lands and by someone other than the reviewer.
Fix the fee and the duration before the work starts. An open-ended review becomes a consulting engagement, and a consulting engagement has an incentive to find more work. Four weeks and a fixed number is the right shape for a plan of any size, because the assumptions worth testing are finite.
Give access to people, not just to the deck. The plan is the summary. The engineers, the data owners, and the person who negotiated the license know what the plan smoothed over, and a review that never talks to them is reading a translation.
Ask for the disagreement in writing. The findings should name the places where the reviewer's read differs from the authors', with the reason and the authors' response beside it. A review everyone agrees with cost money and bought nothing. The disagreement is the product.
Do not ask for a rubber stamp, and do not accept one. Some sponsors want an outside document to carry into a board meeting. That document is worth having only if the reviewer was free to write the opposite. If the terms make an unfavorable finding awkward, the favorable finding is worth nothing to the board either.
What a review cannot do
It cannot make the plan work; closing the gaps it identifies is execution, and execution is a different engagement with different people. It cannot substitute for a measurement nobody has run, so where there is no held-back scored set the review will say so and recommend building one, and that recommendation costs weeks. It is not an audit and produces no opinion of the kind an audit firm signs. It says nothing about whether the market wants the product, which technical evidence cannot settle. And it cannot be both independent and comfortable: a review that tests a plan properly will find at least one assumption somebody in the room is attached to. That moment is the reason the review exists, and handling it in private, before the money commits, is cheaper than every alternative.
Bottom line
A multi-year AI plan usually fails on things that were knowable at the start: the data was not what the plan said it was, the unit price did not survive being multiplied by the real volume, the named model had a shorter published life than the plan, or nobody built the measurement that would have shown the problem in month three. Each is testable against evidence that already exists. What is missing is rarely intelligence or budget. It is a person with no stake in the answer, four weeks, and permission to write down what they find. GAO's finding across four agencies was that buyers lacked the technical people they needed at the moment of decision, and that when hard lessons were learned, nobody wrote them down. Both are cheap to fix before the money moves and expensive after.
Frequently asked questions
An architecture review examines a proposed technical design: the components, the data flow, the failure modes, whether the thing will hold together. A roadmap review examines the plan around the design: sequencing, dependencies with their own schedules, unit economics at volume, model and vendor lifecycle, and whether a measurement exists that would show the plan working. They overlap, and on a large program both are worth doing, but the roadmap review is the one that runs before budget commits.
Not on their own work, and not if the fee or the follow-on depends on the conclusion. The cleanest arrangement is a fixed fee for a fixed duration with a firm that has no implementation claim on the outcome, and a decision about who builds that is made after the report lands. If a partner who might build does the review, put the independence terms in writing before the work starts.
A vendor benchmark answers whether the system performs on data the vendor selected. GAO reports that NIST officials told them well-defined, universal AI tests do not yet exist because of the variation and complexity of AI services, so there is no neutral standard to fall back on. OMB's M-25-22 directs federal agencies to run independent evaluations using data the buyer defines that the vendor cannot access, and to keep contracts from prohibiting internal disclosure of how the vendor tests. A commercial buyer can write the same terms and should.
The plan with dates and money attached; the contracts and license terms, including pricing schedules and renewal language; a data inventory naming the systems of record and their owners; any evaluation already run, with the raw scores rather than the summary; the names and real availability of the people who would do the work; and the compliance obligations already committed to, with dates. State the decision being made and when, because that sets what the review has to settle.
Far enough that the findings can change the plan. A four-week review delivered a week before a board vote produces a document, not a decision input. Six to ten weeks ahead is the useful window: time for the review to run, for the sponsor to test a finding or two internally, and for the plan to be revised before it is presented as final.