Buying AI when you cannot evaluate AI

Most organizations buying AI have no one internally who can assess an architecture, and they know it. The usual response is to hire an advisor or to trust the demo. There is a better option, which is to design a test that does not require expertise to interpret.

Why demos are uninformative by construction

A demo runs on data the vendor selected, on questions the vendor anticipated, in an environment the vendor controls. Nothing about it is dishonest. It is simply a measurement of the best case, and the best case is not the thing being purchased.

Every demo is a measurement of the vendor's preparation, not the product's behavior.

The four tests, none requiring ML expertise

Your data, your questions, sealed. Provide a sample the vendor has not seen, with questions written by your own staff, and require the run to happen in front of you. Refusal to do this is itself the finding.

The unanswerable question. Include several questions the data genuinely cannot answer. A system built correctly declines. A system that answers all of them has demonstrated that it will also answer the ones it gets wrong.

The trace. Take any figure in the output and ask to see the records that produced it. Rows are a pass. A paragraph explaining the figure is a fail, and this distinction is legible to anyone regardless of background.

The repeat. Run the same question three times. Materially different answers mean the output is not reproducible, which matters enormously if anything downstream is audited.

Questions worth asking, and what a real answer sounds like

• What is your error rate on data like ours, and how was it measured? — a number and a method, or an admission that it was not measured
• What does the system do when it does not know? — a described behavior you can trigger in the test
• Where does our data go, and is it retained? — a data flow, not a policy statement
• What happens when the underlying model is updated? — a versioning and revalidation practice
• Who else has deployed this at our scale? — a reference you may contact directly

Structure the deal to match the uncertainty

The strongest protection is not a technical assessment. It is a paid evaluation phase with written acceptance criteria, tied to the tests above, run on your data, with a clean exit if the criteria are not met.

Vendors confident in the product agree to this readily because it shortens their sales cycle. Reluctance to accept measurable acceptance criteria is the most reliable negative signal available to a buyer, and it requires no technical knowledge to read.

Running the four tests properly

The tests are simple to state and easy to run badly. A few details determine whether they produce information.

Mixing the unanswerable questions in among ordinary ones matters. Presented as a block, they signal what is being tested and invite a cautious posture that will not persist in production.

The false-premise question

Worth adding as a fifth test because it is the most revealing single question available and almost nobody asks it.

Ask about something that did not happen. Why did the shipment in March fail inspection, when there was no failed inspection in March. What drove the increase in the third quarter, when there was a decrease.

A grounded system says the premise does not match the data. A generative system explains the event, fluently, with detail, sometimes with citations to real documents that do not say what it claims. Watching this happen once is more persuasive than any amount of architectural discussion, and it requires no technical background to interpret.

Reading the answers you get

The questions listed earlier are only useful if you can distinguish a real answer from a fluent one. The pattern is consistent.

• "What is your error rate?" — Real: a number, a dataset description, a definition of error, and where it is worse. Fluent: a benchmark on public data, or a claim of high accuracy without a measurement
• "What does it do when it does not know?" — Real: a described behavior you can trigger. Fluent: "it is trained to acknowledge uncertainty"
• "Where does our data go?" — Real: a data flow with named regions, retention periods, and subprocessors. Fluent: a link to a security page
• "What happens when the model updates?" — Real: versioning, notice, and a revalidation practice. Fluent: "we continuously improve"
• "Who else runs this at our scale?" — Real: a reference you may contact unaccompanied. Fluent: a logo slide

The pattern across all five is that a real answer contains something checkable and a fluent answer contains a reassurance. That distinction requires no ML knowledge, which is the point.

Structuring the agreement around the uncertainty

The most effective protection available to a non-technical buyer is commercial rather than technical: a paid evaluation phase with written acceptance criteria tied to the tests above, run on your data, with a defined exit.

Criteria should be specific enough to settle a disagreement. Not "acceptable accuracy" but a stated performance level on a named test set, a stated maximum rate of confident answers to unanswerable questions, and a demonstrated trace for every figure in a sampled output.

Beyond the evaluation phase, four terms consistently matter and are consistently absent from standard software agreements: indemnification for harm caused by output, warranties on intellectual property in generated content, notice before material model changes, and the right to audit or receive evidence of the data handling described in the answers above.

The signal that costs nothing to read

Vendors confident in what they built agree to unseen-data evaluation and measurable acceptance criteria readily, because it shortens their sales cycle and it is how they win against competitors who cannot.

Reluctance is the most reliable negative signal available to a buyer with no technical staff. It requires no expertise to interpret, it arrives early, and in practice it correlates better with deployment outcomes than any technical assessment a non-specialist could perform.

That is worth internalizing, because organizations often assume they need to hire expertise to buy responsibly. Frequently what they need is a well-designed test and the discipline to insist on it.

Total cost is rarely the license

Buyers compare quoted prices and are surprised by the second year. The costs that dominate are mostly on the buyer's side and mostly absent from the proposal.

Asking a vendor to help you estimate the unquoted rows is itself a test. One that engages seriously has deployed before; one that treats them as your problem probably has not.

The build alternative, honestly framed

Organizations with engineering capacity ask whether to build instead, and the honest answer depends less on capability than on what the system is made of.

The generic parts — model access, retrieval plumbing, orchestration — are commodity and buying them is nearly always right. The parts specific to your data — the entity semantics, the schema, the evaluation set, the provenance chain — are where your advantage lives and where a vendor's generic implementation fits worst.

Which suggests a middle path more often than either extreme: buy the platform, own the semantics. The failure mode to avoid is buying a system that encodes your domain logic in a form you cannot inspect, export, or take with you, because that is the arrangement that becomes impossible to leave.

Reading the reference call

References are offered because they will be positive, so the useful questions are not about satisfaction. Ask what broke, what took longer than expected, what they staffed that they had not planned to staff, what they would scope differently, and what happens when the vendor ships a model update.

A reference who answers those concretely is describing a real deployment. One who cannot is describing a pilot, and a pilot reference tells you nothing about the thing you are buying.

Frequently asked questions