Scoping an AI pilot that does not stall

The stall happens upstream of the model

A stalled AI pilot looks like a technical problem from the outside. Accuracy came in lower than hoped, the demo was slow, the outputs were uneven. Look at the calendar instead of the metrics and a different story shows up: six of the twelve weeks went to getting a data extract, the success criterion was written after the first results were in, and when the final readout happened there was no one in the room whose job it was to change how work gets done. The model was the only part of the project that had an owner.

Gartner predicted in July 2024 that at least 30 percent of generative AI projects would be abandoned after proof of concept by the end of 2025. Whatever the precise figure turns out to be, the abandonment pattern is consistent across sectors and it is not a modeling pattern. It is a scoping pattern. Four failures account for most of it, and all four are decided in the first two weeks, before anyone writes code.

Below is the diagnosis of those four failures and a six-field scoping page that closes them. If a sponsor cannot fill the page in, the pilot is not ready to start, and starting anyway converts a two-week disagreement into a six-month one.

Failure one: data access treated as a task instead of a dependency

A kickoff deck says "obtain representative data" as a line item in week one. In a regulated organization that line item is a chain of approvals with named humans on it, several of whom have never heard of the project. The data owner is usually not the person who sponsored the pilot. The system of record belongs to a program office; the security boundary belongs to an ISSO; the legal basis belongs to a privacy officer.

In federal work the chain is written down, which helps once you know where to look. If agency records containing personal information will be used for a purpose the published System of Records Notice does not already cover, the Privacy Act at 5 U.S.C. 552a is the gate, and amending a SORN is a Federal Register action rather than an email. A new or materially changed IT system handling personally identifiable information triggers a Privacy Impact Assessment under Section 208 of the E-Government Act of 2002. Moving data between two accredited systems normally requires an interconnection agreement of the kind described in NIST SP 800-47. Tax data carries IRS Publication 1075, student records carry FERPA, criminal justice data carries the CJIS Security Policy, and each has its own custodian.

None of this is exotic and none of it is fast. The mistake is sequencing. Teams start modeling in week one and start the access work in week three, then learn in week seven that the promised extract needs a data use agreement that takes a month. Make the data path the first deliverable and name the human who signs at each step.

Failure two: no metric that anyone agreed to in advance

Ask five people at a kickoff what success looks like and you get five answers: fewer errors, faster turnaround, less overtime, better consistency, a demo the deputy director likes. All five are legitimate. None is a threshold. Without a number fixed in advance, the readout becomes a negotiation about whether 84 percent is good, and that negotiation has no natural end.

The threshold has to be paired with an operating point, because a single accuracy number hides the trade that matters. A model that is right 92 percent of the time is a very different product depending on whether the 8 percent shows up as missed items or as false alarms. Fix the recall you need, then report precision at that recall. Fix the false-positive budget the reviewers can absorb per week, then report what recall you get inside it. A threshold without an operating point can be met by tuning in a direction nobody wanted.

Fix the unit of measurement in the same sentence. A document-extraction system reported at the field level and the same system reported at the document level can differ by twenty points, because a document is only correct when every field in it is. Reviewers work at the document level, so that is usually the honest unit. Whatever the choice, write it into the scoping page next to the threshold. Half the disputes at a readout are two people quoting correct numbers computed over different denominators.

Failure three: no owner for the decision the model informs

Every model exists to change a decision. Which claims get a second look. Which assets get inspected this quarter. Which of 400 incoming documents a human reads first. If that decision currently belongs to someone, then that someone has to be in the pilot, because they are the person who will either change their process or not.

When the decision owner is absent, the pilot produces a dashboard nobody is accountable for acting on. The demo goes well, everyone agrees it is interesting, and the work returns to the way it was done before. This is the quietest way a pilot dies, because nothing visibly fails. A pilot with a named decision owner sounds different from the first meeting: that person asks about workload, about what happens on a bad day, about whether their staff will trust the output. Those questions are the project.

Federal buyers have a formal version of this requirement. OMB Memorandum M-25-21, issued April 3, 2025, sets minimum risk management practices for what it calls high-impact AI, including pre-deployment testing, an AI impact assessment, ongoing monitoring, and human oversight with a route to remedy for affected people. Each of those practices assumes a named owner on the government side. M-25-22, issued the same day, carries related expectations into acquisition.

Failure four: evaluating on data that does not look like production

The evaluation set usually arrives as a clean export: complete records, consistent formatting, the last two years, deduplicated by someone helpful. Production is a different animal. It has the scanned fax, the record with three fields blank, the vendor whose name is spelled four ways, the case type that appears eleven times a year and matters more than the other thousand.

Two habits fix most of this. First, sample the evaluation set from the real intake stream over a real window, including the days everyone would rather forget, instead of from a curated table. Second, hold out a slice by time rather than at random. Random splits leak: if the same case appears in both halves, or if last month's patterns are in training and this month's are in test, the number you report will not survive contact with next month. A time-based split is a weaker-looking number and a far more honest one.

Then size the rare-but-important classes deliberately. If the value of the system comes from catching the 2 percent of cases that cost real money, an evaluation set holding fourteen of them cannot tell a good model from a lucky one. Count them before you start, and if there are not enough, say so in the scoping page and plan a longer collection window.

The scoping page: six fields

The whole method is one page with six fields, filled in by the buyer and the builder together before any environment is provisioned. It is short on purpose. If a field cannot be filled in with a specific answer, that is the finding, and it is better to have it in week zero.

Field	What it must state	Weak answer that predicts a stall
1. The decision	One recurring decision, its frequency, and who makes it today. "Which of ~1,200 monthly submissions goes to a senior reviewer, decided by the intake lead."	"Improve efficiency in the claims division."
2. The baseline	How that decision performs now, measured the same way you will measure the model. Include the cost of being wrong in each direction.	"Current process is manual." No number at all.
3. Metric and threshold	One primary metric, one operating point, one number that counts as success. "Recall of 0.90 on priority cases at no more than 25 false positives per week."	"High accuracy." Two metrics with no priority between them.
4. Data path	Source system, custodian by name, legal basis, environment where the data may sit, and the date the first real extract lands.	"IT will provide the data." No named custodian, no date.
5. Failure regime	What the system does when it is uncertain or wrong: abstain, escalate, flag, degrade. Who sees it. What the reviewer's workload looks like on a bad day.	Not addressed, or "a human reviews everything" with no time budget.
6. Go/no-go gate	A date, a named decider, and what happens on each outcome, including the no.	"We'll evaluate at the end and see."

Field 2 in detail: measure the baseline before you build

The baseline is the field most often skipped and the one that decides whether the result means anything. If nobody knows how accurate the current process is, no model result can be interpreted. Ninety-one percent is excellent against a human baseline of 78 and a failure against a baseline of 96.

Measuring it is usually cheap. Take a sample of past decisions large enough to matter, have two experienced staff independently re-adjudicate them without seeing the original outcome, and compare. That exercise produces three things at once: a baseline accuracy number, an inter-rater agreement number, and a labeling protocol. The agreement number is the sleeper. If two experts agree only 70 percent of the time on what the right answer is, no model will be evaluated above 70 percent against either of them, and the honest scope of the pilot changes from prediction to consistency.

Baselines also need a cost side. Write down what a false positive costs in reviewer minutes and what a false negative costs in dollars, delay, or risk. Those two numbers set the operating point in field 3 and end most arguments about thresholds before they start.

Field 4 in detail: the data path is a sequence of named approvals

Write the data path as steps with owners and dates, the way a schedule is written, and put it in front of the sponsor at kickoff. The elapsed times below are planning assumptions to use until a specific agency or company tells you otherwise. They are deliberately not optimistic.

Two shortcuts are worth knowing. Working inside an environment the customer has already accredited removes step 4 almost entirely, which is why "we will work in your existing enclave" is often worth more than any modeling claim. A de-identified or synthetic slice can unblock early engineering while the real path is still moving, provided the evaluation that decides the gate runs on real data. A pilot that only ever sees synthetic data has proven that the code runs.

Field 5 in detail: say what happens when the model is wrong

Every deployed model has a wrong answer somewhere in its future. The scope has to say what the system does at that moment. Three behaviors cover most cases, and choosing among them is a design decision, not a detail to settle later.

Abstain. The system declines to produce an output below a confidence threshold and routes the case to a human. This is the safest posture for consequential decisions, and its real cost is the abstention rate. A system that abstains on 40 percent of cases has not removed 40 percent of the work.

Rank rather than decide. The system orders a queue and a human works down it. Errors become ordering errors instead of wrong outcomes, which is often the easiest version to get approved, because the human decision authority never moves.

Decide with review. The system produces an outcome and a sample is audited. This needs the tightest evidence, and it is where documentation requirements bite: what was the input, which model version ran, what did it output, who reviewed it. Log those four things from day one of the pilot, because retrofitting an audit trail after a successful demo is more work than building it in.

The MEASURE and MANAGE functions of the NIST AI Risk Management Framework, published January 2023, are a useful checklist here. The framework is voluntary and it is increasingly the vocabulary buyers use to ask these questions, so answering in it saves a round trip.

Field 6 in detail: a gate with a real no

A go/no-go gate needs three things: a date, a named decider, and a written consequence for each outcome. The consequence for "go" is easy to write. The consequence for "no" is the one that gives the gate its force, and the one teams skip.

Write it plainly. If the threshold is missed, the work stops and the remaining funds go to the next candidate use, or the scope narrows to the sub-population where the threshold was met, or the team gets one defined extension with a specific hypothesis attached. Any of those is fine. What is not fine is an unwritten default of continuing, because that default is what converts a pilot into an indefinite research project with a budget code.

For federal buyers this matches how software is supposed to be acquired anyway. FAR 39.103 directs modular contracting for information technology: narrow scope, short successive increments, each one usable and each next one a decision rather than an obligation. A pilot with a real gate is one increment. Structured that way, a no is a normal outcome that cost a defined amount and produced a documented finding.

Kickoff readiness check

Run this before the environment is provisioned. Anything unchecked is a scoping conversation, not a build task.

• The decision, its frequency, and its current owner are written in one sentence
• The baseline has been measured, including inter-rater agreement among experts
• One primary metric, one operating point, and one threshold are agreed in writing
• The data custodian is a named person who has acknowledged the request
• The legal basis for the intended use is identified, not assumed
• The evaluation set is drawn from production intake with a time-based holdout
• The rare-but-important class has enough examples to measure
• The failure behavior is chosen and the reviewer workload is estimated
• The gate has a date, a decider, and a written consequence for a no

Common objections, and what they usually mean

What this costs and what it saves

Filling in the six fields takes two or three working sessions with the right people in the room, plus whatever the baseline measurement costs, which is typically a few days of expert time. Against a pilot budget that runs from tens of thousands of dollars in a commercial setting to a few hundred thousand under an SBIR Phase I or a small task order, that is a rounding error.

What it buys is the ability to end the project cleanly. A scoped pilot produces one of three outcomes: it clears the threshold and there is a named person ready to change a process, it misses and the reason is documented well enough to inform the next attempt, or it never starts because a field could not be filled in. All three are results. The unscoped pilot produces a fourth outcome, which is a demo, a slide deck, and a quiet return to the previous way of working.

Frequently asked questions