What an AI proof of concept should prove

A proof of concept is a purchase of information

The right way to think about a pilot is that you are buying an answer to a question, and the price is worth paying only if the answer would change what you do next. That framing does almost all the work. It tells you what to build, how long to spend, what to measure, and — most usefully — when not to run one at all. If you already know what you will do regardless of the outcome, you are not buying information. You are buying a slide.

Demonstrations are not worthless. They align stakeholders, they make an abstract idea concrete, and they are sometimes the only way to get a budget conversation started. But a demonstration and an experiment are different purchases with different designs, and the failure mode that wastes the most money in this field is buying the first while believing you bought the second. A demonstration is built to succeed on chosen examples. An experiment is built to report a number on examples nobody chose.

The practical tell is simple. Ask the person proposing the pilot: what result would make us stop? If there is no answer, or the answer is soft, the pilot cannot produce information. Every hour of it will confirm the plan that already existed, and the project will proceed into production on the strength of forty hand-picked examples that worked in a meeting.

Name the uncertainty before you name the technology

Almost every AI project carries several risks at once, and the pilot should attack the one most likely to kill it. In our experience the candidates fall into four families, and teams routinely test the wrong one because the wrong one is more fun to build.

Can it be done well enough? The quality question. Can a system read these documents, classify these tickets, forecast this demand, at a level that beats what happens today. This is the risk everyone tests, and with current models it is increasingly the least risky of the four.

Does the data support it? The data question, and the one that kills more projects than any other. Do the records that would train or ground the system actually exist, do they cover the cases that matter, are they consistent enough to be joined, and can the team get access to them within the life of the project. A pilot run on a hand-cleaned extract answers nothing about a system that must run on the real feed.

Does the arithmetic work? The economics question. What does one unit of work cost to process, including the human review that the accuracy level implies, and is that below what it costs today. A system that is ninety percent accurate and requires a person to check all of the output has not removed the person.

Will anyone use it? The workflow question. Where does the output land, who acts on it, what happens when it is wrong, and does the person whose job it touches have a reason to prefer it. This is the least tested and it is where a large share of technically successful pilots quietly die.

Write down which of the four is most likely to end this project. Design the pilot to attack that one. If the honest answer is that you do not know which, spend the first week finding out — that week is cheaper than a pilot aimed at the wrong risk.

The decision rule, written before anything is built

This is the highest-value paragraph here. Before work starts, write one page containing the metric, the threshold, and the action attached to each outcome. Something in this shape: on a held-out set of 300 real cases, drawn at random from the last twelve months, if the system handles more than 80 percent without human correction we proceed to a production build; below 60 percent we stop and do not revisit for a year; between 60 and 80 we do one specified round of improvement and re-measure once.

Then have it signed by whoever can actually stop the project. Not agreed in a meeting — written and acknowledged, because the entire value of the rule comes from existing before the result does. Everyone is a rigorous evaluator in advance and a creative one afterward, and that is not dishonesty. It is what human beings do when they have spent three months and a hundred thousand dollars and the number comes in at 71 percent.

Two details make the rule usable. First, it must reference a set of examples that exists before the work starts, so nobody can select the cases later. Second, the middle band must have a specified action rather than “discuss it” — because the middle band is where most pilots land, and an unspecified middle is how a project drifts into production without a decision ever being made.

Uncertainty	What the pilot must produce	The result that should stop you
Quality	A number on a held-out set of real cases, next to the current process and a simple baseline	No better than the simple baseline, or worse than people at a cost that is not lower
Data	An access path to the live source, and a measured completeness rate on the fields that matter	Key fields missing on a large share of records, or no realistic path to the live feed
Economics	Cost per item processed at the observed accuracy, including the human review it implies	Cost per correctly handled item at or above the current cost
Workflow	Real users acting on real output for two weeks, with usage measured rather than surveyed	People stop opening it, or route around it, and can explain why
Latency	Response time at the volume the process actually produces, at peak not average	Slower than the decision window the work operates in

Two baselines, and almost nobody builds either

A number by itself means nothing. Eighty-seven percent accuracy is excellent or embarrassing depending on what it is next to, and a pilot report without comparisons is a pilot report without information.

The first baseline is the current process. How accurate are people today, how long do they take, and what does an error cost. Teams are frequently unable to answer this, which is itself the most valuable finding of the whole exercise, and it usually takes two or three days of sampling to fix. It also has a habit of shrinking the perceived problem: a process that everyone describes as error-prone often turns out to be 96 percent right, which sets a bar the pilot has to clear rather than a mess it can trivially beat.

The second baseline is the boring approach. Keyword rules. A regular expression. Predicting the most common category every time. A small classical model on three features. This one is deeply unpopular, because it is unglamorous and because it wins more often than anyone expects. When the simple version gets to 82 and the sophisticated version gets to 86, you have learned something important about where the remaining budget should go, and possibly that you can ship next month instead of next year.

Both baselines are cheap. Together they are usually two to four days of work, and they change the interpretation of every subsequent number. A vendor who declines to build them is telling you what their number would look like beside them.

The evaluation set is the real deliverable

If a pilot leaves behind one durable thing, it should be a set of real examples with correct answers attached, produced by people who know the domain, held out from whoever is tuning the system. Everything else — the prompts, the model choice, the interface, the vendor — can be replaced later. The evaluation set is what lets you tell whether the replacement is better.

Practical shape. Two hundred to five hundred examples is usually enough to distinguish large differences and too few to distinguish small ones; if you need to detect a two-point improvement, you need more. Draw them at random from a real period rather than curating them, then deliberately add a slice of the hard cases the business actually loses money on, and keep those two groups scored separately. Have at least a subset labeled twice by different people, because the rate at which your own experts disagree is a hard ceiling on any accuracy you can ever claim — and finding out that two senior staff agree only 80 percent of the time is one of the most useful things a pilot can tell you.

Budget it honestly. Expert labeling of a few hundred non-trivial items is typically one to three weeks of somebody's part-time attention, which prices at roughly $5,000 to $25,000 in internal cost depending on who has to do it. That is real money and it is the least regrettable line in the whole pilot, because it is the only artifact that keeps its value when the technology underneath changes, which it will, twice, before you finish.

Put the economics inside the pilot

The number that decides a deployment is rarely accuracy. It is cost per correctly handled item, and it is a function of accuracy, of what a mistake costs, and of how much human review the accuracy level forces you to keep.

Work it through with real figures during the pilot rather than after. If the system handles 85 percent of cases correctly and a mistake is expensive, you will review everything, and then your cost is the model plus the reviewer, which is higher than the reviewer alone. If a mistake is cheap and reversible, you can accept 85 and route only the low-confidence cases, and the arithmetic looks entirely different. The same accuracy number is a good business in one case and a worse-than-nothing business in the other, and nobody can tell which from the accuracy alone.

Two related quantities belong in the same page. Latency, measured at peak rather than average, against the window the process actually operates in. And the confidence behavior — whether the system knows when it is unsure. A system that is 85 percent accurate and reliably flags its own uncertain cases is far more valuable than one that is 88 percent accurate and confidently wrong, because the first one can be deployed with a review queue sized to the flagged share and the second one cannot be deployed at all.

What a proof of concept should not try to prove

Scope creep in pilots comes from trying to prove things that are not in doubt, and each one adds weeks.

That it scales. Almost nothing at pilot volume tells you about production volume, and the engineering to find out is a production project. Note the volume assumption and move on.

That it is secure and compliant. Worth a paragraph of design review during the pilot, not a workstream. Run the pilot on data you are permitted to use, and put the real review in the production phase where it belongs.

That the integration is complete. A pilot can read an export and write to a spreadsheet. Building the two-way integration to the system of record before you know whether the thing works is the most common way to spend production money on a question.

That the interface is right. Enough interface to let a real user do the real task, and no more. Polish is a production concern and it is also, quietly, the thing that makes stakeholders confuse a demonstration with a result.

Cost, duration, and when to skip it entirely

A well-scoped pilot runs three to six weeks and costs somewhere between $40,000 and $120,000 at commercial rates for the outside work, plus the internal labeling and subject-matter time discussed above. Longer than six weeks and it has usually stopped being a pilot and become a slow production build without a production plan.

There are three situations where the honest recommendation is to skip it. The first is when the problem is well trodden — document classification on clean text, a standard forecast on a clean series, retrieval over a document set you already own. The uncertainty is low enough that the pilot cost approaches the build cost, and you should simply build the smallest useful version and put it in front of users.

The second is when you already know the answer. If three people in the company can predict the result and agree, the pilot is a way of documenting a decision rather than making one, and there are cheaper documents.

The third is when the real blocker is not technical. If the data cannot be released, if the process owner has not agreed to change the process, if there is no budget for a production build regardless of the outcome — then the pilot answers a question that is not the one standing in the way. Fix the actual blocker first. That advice costs us work fairly often and it has never once been wrong.

The mistakes we see most

• No stopping condition, so the pilot cannot produce information
• Curated examples instead of a random draw from real cases
• No baseline, so the headline number cannot be interpreted
• Hand-cleaned data that the production system will never receive
• Accuracy reported without the review burden it implies
• The evaluation set left in the vendor's account rather than delivered to you
• Success declared in a meeting where the demonstrator drives the keyboard
• A pilot aimed at the quality risk when the project will actually die on data access or adoption

Reading the result honestly

Three habits protect a result at the end. Have somebody other than the builder run the evaluation, on the held-out set, without the builder present — not because anyone is dishonest, but because the person who tuned the system knows which inputs it likes. Look at the errors individually rather than only in aggregate; twenty minutes reading actual failures tells you more about deployability than the summary statistic, and it produces the error taxonomy that the production build will need anyway. And treat the middle band as the expected outcome rather than an inconvenience, because it usually is.

The last habit is the hardest and the most valuable: be willing to publish a stop. A pilot that ends with a documented, well-evidenced “no” has done its job perfectly and has saved the production budget for something that will work. Organisations that can do this run more pilots, learn faster, and waste less. Organisations that treat a stop as a failure quietly stop running honest experiments, and every pilot after that succeeds.

Before you fund a pilot

• The riskiest uncertainty is named, and the pilot attacks that one
• The decision rule is written and signed before work starts
• The middle band has a specified action, not a discussion
• The evaluation set is drawn at random from real cases, before tuning begins
• Both baselines are in scope: the current process and the simple approach
• Cost per correctly handled item is a required output
• The evaluation set, labels and guidelines are contracted as deliverables to you
• Somebody other than the builder will run the final evaluation
• There is a real budget for the production build if the answer is yes

Bottom line

A proof of concept earns its money when it changes a decision, and it can only change a decision if it was capable of coming back with bad news. Name the uncertainty that would kill the project, write the threshold and the stopping action before anything is built, measure against both the current process and the boring approach, and price the unit economics inside the pilot rather than after it. Keep the evaluation set — it will outlive the model, the vendor and probably the strategy. And if the honest answer is that the question is not really in doubt, build the small version instead and spend the pilot money on something you actually do not know.

Frequently asked questions