The pilot did not fail. It succeeded at the wrong thing.
A director tells us the same story about once a month, with the nouns changed. The team built something in ten weeks. It worked, and on the sample documents the extraction beat the analysts. Then it went to an architecture board, or a security review, or a discussion about who would own the pager, and it stopped moving. Nobody killed it. It never got a slot, and by the next budget cycle its sponsor had moved on. The company concluded out loud that the technology was not ready. The real conclusion is that the pilot was scoped as a demonstration, a production system is a different object, and the ten weeks bought almost nothing that transfers.
This is not rare. Commonly cited figures put the share of machine learning projects that never reach production between half and four fifths, the ranges wide because everyone counts differently. The shape matters more than the number, and it is stable. In nearly every case we review, the pilot cleared the technical bar and failed one nobody wrote down: ownership, data access, evaluation, security, or cost.
What follows is the list, in roughly the order the failures arrive, with the mechanism behind each. Then the part that matters more: what a second attempt should be scoped as, and how to tell inside two weeks whether it will end the same way.
Where stalled AI pilots actually die, by frequency in the engagements we review
Editorial weighting from our own review engagements and published practice. Illustrative, not a measured statistic.
Failure one: the data path was a person
The most common cause, and the least discussed, is that the pilot ran on an extract. A DBA ran a query against a replica on a Friday, or an analyst exported 4,000 PDFs into a shared drive. That extract fed everything that followed. It was clean because a human cleaned it, complete because a human checked it, and current because it was pulled the week the work started.
Production has none of those properties. The same data on a schedule brings nulls the extract filtered out, a schema that changes when the upstream vendor ships an update, records that arrive late and out of order, and a permissions model the human pull bypassed. Turning a one-time extract into a repeatable path is frequently larger than the model work, and it was never in the pilot budget because it did not look like AI.
We ask one question in the first hour of any review: show me the pilot's input data being produced by something other than a person. If the answer involves a name, the pilot proved a model can do a task and nothing about whether the company can feed it. Rebuilding that path runs six to twelve weeks per source, longer when the source is a vendor system with an export API rather than a database you control.
Failure two: nobody owned the thing after the demo
Pilots are run by an innovation group, a data science team, or an outside firm. Production systems are owned by whoever carries the pager. Those are almost never the same people, and that handoff is where most pilots quietly stop.
The receiving team has a rational objection. They are asked to operate a system they did not design, in a stack they may not use, with no runbook, no alerting, no on-call rotation, and a failure mode they cannot debug from a stack trace. Their answer is not no. It is "put it in the queue," which is the same thing with better manners.
The fix is structural and cheap at the start. Name the operating team before the pilot begins, put one of their engineers on the build at even ten percent time, and make the exit criteria include what that team will ask for: a runbook, structured logs, a health check, a defined alert with a defined response, and a rollback executed once rather than described. A pilot that ends with those gets a slot.
Failure three: the evaluation was a screenshot
Ask what the pilot's accuracy was and you get a number. Ask how it was computed and the story thins. Usually somebody reviewed output on a set of examples and counted the ones that looked right. That measures a moment, and it does not survive the first thing that changes.
The consequences arrive later and they are severe. When the provider deprecates a version, on cycles now measured in months, nothing tells you whether the replacement is better or worse on your task. When a prompt is edited to fix one complaint, nothing tells you what it broke. When the business asks whether performance has drifted, nobody can say. Each is a reason a review board declines to put the system in front of customers.
What production needs is unglamorous: a labeled set drawn from real inputs, held apart from anything used to build the system, with an adjudicated answer key and a written scoring rule. Per-field metrics rather than one aggregate, because an aggregate hides the case that matters. And it runs on every change, so a regression is a failed build rather than a customer email. That set is the highest-return item in a second attempt, at two to four weeks of analyst and engineering time.
One trap is worth naming. An evaluation set built after the system, by reviewers marking errors in system output, encodes the system's blind spots as ground truth. Mistakes it makes consistently look correct to a reviewer anchored on what the screen already says. Independence is designed in, not retrofitted.
Failure four: security arrived after the architecture was fixed
The pilot called a hosted model API over the internet, which was right for a pilot. It became wrong the moment the system was proposed for production with real customer data, and nobody noticed until a security architect read the diagram in month seven.
The specifics follow the data. Regulated health information makes the model provider a business associate, and the agreement has to exist before the data moves. Controlled unclassified information under a federal contract pulls NIST SP 800-171 onto the contractor systems that store or process it, and a cloud service touching covered defense information inherits FedRAMP Moderate equivalency through DFARS 252.204-7012. A bank customer puts the model in scope for SR 11-7 model risk management, written for statistical models and applied to this anyway, with validation and documentation expectations a pilot never produced.
Even without a regulator, an internal review asks about the OWASP Top 10 for LLM Applications: prompt injection, sensitive information disclosure, supply chain exposure in model and plugin dependencies, excessive agency. MITRE ATLAS covers the adversarial side with a technique catalog modeled after ATT&CK. NIST's AI Risk Management Framework and ISO/IEC 42001 are what an enterprise governance function is most likely to measure you against, and neither is satisfied by a working demo.
None of this is hard when scoped at the beginning. Deciding in week one whether inference happens inside your boundary, whether payloads may leave, and which control set applies changes the architecture modestly. Deciding it in month seven means redrawing it, which is the point at which a pilot stops being funded.
Failure five: the arithmetic was never done at real volume
A pilot processes a few thousand items and the inference bill is a rounding error. Production processes the whole queue every day, and the number changes character. We watched a project die in a finance review because a system that cost $180 during the pilot came to roughly $31,000 a month at full volume, against a business case that assumed a fraction of that.
The arithmetic takes an afternoon and almost nobody does it. Take the real daily volume, the average input and output size, the retry rate, and the fact that a production path usually makes more than one model call per item once validation and re-asking are counted. Multiply. Then add what is not inference: storage, the vector index, orchestration compute, egress if the corpus and the model sit with different providers, and the human review time the design still requires.
That last item reverses more business cases than any other. A system at 92 percent that requires a person to check every output has not removed the person; it added a step. The economics work only if the design routes confident cases through untouched and uncertain ones to a human, which requires the system to know which is which, which requires calibration a pilot almost never does.
| What the pilot proved | What production needs instead | Typical cost to add later |
|---|---|---|
| A model can do the task | A repeatable data path with schema handling and late arrivals | 6 to 12 weeks per source |
| Output looked right on samples | An independent labeled set, per-field metrics, regression in CI | 2 to 4 weeks, plus lost independence |
| It ran on a laptop or a notebook | A named owner, runbook, alerting, tested rollback | 3 to 8 weeks, or the project stalls |
| It called a public API | A decided boundary, a documented data flow, a control mapping | 4 weeks early; a redesign at month seven |
| Inference was cheap | Cost per item at peak, including review labor and retries | One afternoon early; a cancelled program late |
Failure six: the process around the system never changed
This is the one that kills systems after they ship, which is worse than dying before. The model works, it is deployed, and adoption is near zero because the people whose work it was supposed to change were told about it rather than designed with.
The mechanism is simple. An analyst has a workflow that works. The new system puts a suggestion somewhere they have to go look for it, offers no account of where the answer came from, and carries a personal cost if it is wrong and they accepted it. Ignoring it is the rational choice, and they will ignore it politely and forever.
Training does not change this. Three design decisions do. The output arrives inside the tool the person already uses, not a new tab. Every value traces to the source span it came from, so verification takes three seconds instead of three minutes. And accountability is explicit: who is responsible when the system is wrong and a person accepted it, agreed before launch, because that ambiguity is what makes people refuse to rely on it.
Where the effort goes on a slice that actually ships
Typical shape of a first production slice in our delivery practice. Illustrative, not a measured statistic.
What the second attempt should be scoped as
Not a bigger pilot. A thin production slice. Pick the narrowest useful piece of the workflow and build it all the way through: real users, real data arriving on a schedule, real monitoring, a real owner. Narrow in scope, complete in depth. It either runs or fails for a reason you can name, and everything in it is reusable when the scope widens.
The scoping rules are short. One task, chosen because a specific person's decision changes based on the output. One data source, ingested as a scheduled job on day one even if the volume does not require it. An evaluation set built before the system by people who are not building it. A named operating team with a member on the build. A written boundary decision in week one. A cost model at ten times the slice's volume, before the build starts.
That runs longer than a pilot and shorter than a program: twelve to twenty weeks for a first slice, with effort roughly two to one in favor of the plumbing over the model. Teams find that ratio disappointing. It is also the reason the second attempt works.
How to tell in two weeks whether it will stall again
A short diagnostic beats waiting six months. Ask five questions and require the answers in writing.
Who operates this on the day after launch, by name and team. Where does the input data come from, and does that mechanism involve a person. What is the labeled evaluation set, who built it, and was it built before the system. What is the security boundary and which control framework applies to it. What does one item cost to process at full volume, including human review. If any answer is missing at the end of week two, the effort has already acquired the shape of the last one, and the cheapest moment to fix it is now.
All five share a pattern. Each answer is owned by somebody outside the build team, and pilots fail because they are structured to avoid needing anyone outside the build team. That is what makes them fast, and exactly what stops them becoming systems.
What we do about it
Our engineers get called in at two moments. The first is after a stalled pilot, when a director needs to know whether the work is salvageable and what production actually costs. That review runs two weeks against documents and interviews and produces a written decision record: what transfers, what has to be rebuilt, the boundary and control question answered, the cost model computed, and a sequenced plan naming the dependencies outside the engineering team.
The second is the build itself, scoped as the thin production slice rather than a second demonstration. We write the ingestion as a scheduled job from day one, build the evaluation set before the system exists and hand you the answer key, put monitoring and the runbook in the same sprint as the model, and answer the boundary question in week one. The deliverable is a system your operations team will accept, with the artifacts they will ask for already written.
Bottom line
Model quality is almost never the reason a pilot stops. Three years of model improvement have not moved the stall rate much, which is the strongest evidence available that the models were never the constraint. A demonstration and a system are different objects, and a pilot scoped to produce the first will not produce the second no matter how well it goes. A narrow slice built to production depth costs more up front, moves slower for a month, and is the only version of this that ends with something running.
Frequently asked questions
Usually the task definition, the prompt or model selection work, and the error analysis transfer. The data path, the evaluation approach, and the deployment scaffolding are usually rebuilt. Expect roughly a third of the effort to carry forward, and be skeptical of anyone who promises more before reading the code.
Twelve to twenty weeks for a first one, assuming data access is approved before the start. Data access approval is the dependency that most often adds a month, so it belongs before kickoff rather than inside week one.
Yes. Public benchmarks measure general capability on public data. They tell you nothing about performance on your documents, your edge cases, or your definition of correct, and they cannot tell you whether a version upgrade improved or degraded your specific task.
Most enterprise governance functions are working from the NIST AI Risk Management Framework, ISO/IEC 42001, or both. Security review typically references the OWASP Top 10 for LLM Applications and, for adversarial threat modeling, MITRE ATLAS. Regulated financial use adds SR 11-7 model risk management, and federal data adds NIST SP 800-53 or SP 800-171 depending on the boundary.
Often, yes. If the first attempt established that the task is technically feasible, a second demonstration of that fact has no value. Go directly to the thin production slice and treat the feasibility question as answered.