Diligence fails because it asks about the company instead of the system
A standard AI diligence packet has three parts. A live demo, two or three reference customers the vendor selected, and a security questionnaire with a couple hundred yes-or-no rows. Each part is answerable by a company whose system does not work. The demo runs on inputs the vendor curated. The references were chosen because they are happy. The questionnaire measures whether someone in the company knows what the right answer is, which is a different fact from whether the control exists. You can complete all three, sign, and discover in month four that the accuracy number in the deck was measured on a test set the vendor built after the model.
The fix is not more diligence. It is different diligence. Instead of asking the company to describe itself, ask it to produce the artifacts a working system generates as a side effect of working. A team that runs evaluations has evaluation history. A team that operates in production has incident records and a cost per unit. A team that handles your category of data has a data flow map and a signed subprocessor list. These things exist or they do not, and a vendor cannot write them the weekend before your meeting without it being obvious.
What follows is the list I would use if I were a director at a ratings firm, a health-tech company, or a systems integrator, and I had four weeks and one engineer's attention to spend before signing something that will run for three years.
Ask who built the test set, and when
This is the single highest-yield question in AI diligence and almost nobody asks it. Every vendor has an accuracy number. The number is meaningless without knowing the provenance of the data it was measured on.
There are three answers and they are worth wildly different amounts. The best answer is that a held-out set was defined before the model was built, labeled by people who were not on the modeling team, and has been frozen since. The middle answer is that the set was assembled from production data after the fact, with sampling documented. The worst answer, and the most common, is that engineers pulled examples they considered representative, which means the test set encodes the same assumptions as the model. A system evaluated that way will score well on exactly the cases it was already good at, and the score tells you nothing about the cases you care about.
Follow it with a second question: what fraction of the evaluation set does the model get wrong, and can you see fifty of those errors. A vendor with a real evaluation practice can produce the error set in an afternoon because they look at it every week. A vendor without one will offer to schedule a session. The offer to schedule is the answer.
The version of this that costs nothing and settles the matter is a blind run. Send 200 to 500 examples from your own data, labeled by your people, held back from the vendor. Ask for predictions, not a demo. The gap between the deck number and the number on your data is the real finding of the entire diligence process, and I have seen it run 15 to 30 points on document extraction and classification work where the vendor was not being dishonest at all. Their data was simply cleaner than yours.
Diligence Evidence — What Each Artifact Actually Proves
Editorial weighting from practitioner experience. Illustrative ranking, not a measured statistic.
Read the evaluation history, not the current score
A single accuracy figure is a photograph. What you want is the film. Ask for the evaluation results for the last six model versions, on the same test set, with dates. Three things fall out of that table immediately.
You see whether the team measures at all between releases, or only when a customer asks. You see whether performance moves in one direction or bounces, and bouncing usually means the evaluation set is too small for the differences being claimed. And you see whether they track anything besides the headline metric. A team that reports only aggregate accuracy is not looking at the subgroups where the system fails, which is where your regulatory exposure lives if you are in lending, hiring, insurance, or clinical decision support.
If the vendor sells into financial services, the useful reference frame is SR 11-7, the Federal Reserve and OCC supervisory guidance on model risk management. It predates the current wave of AI by more than a decade and it is still the sharpest short document on the subject. Its core demand is effective challenge: review by people with the incentive and standing to say the model is wrong. Ask the vendor who plays that role in their organization. If model developers also own model validation, there is no effective challenge, and every number you have been shown was graded by the person who took the test.
Find out what happens when the model is wrong
Every AI system is wrong some of the time. The question that separates a product from a science project is whether the system knows it. Ask what the confidence score means, and specifically whether it has been calibrated. A calibrated system that outputs 0.8 is right about 80 percent of the time at that threshold. An uncalibrated one outputs 0.8 because of how a softmax normalized, and the number carries no operational meaning. You can check this in an hour with a reliability plot on the blind run you already sent, and vendors are frequently surprised by the result.
Then ask what the system does at low confidence. The good answer is that it abstains, routes to a queue, and that the queue has an owner and a service level. The bad answer is that it returns its best guess with the same interface as a high-confidence answer, which pushes the entire burden of catching errors onto whichever of your employees is least equipped to catch them. That design decision determines the real staffing cost of the system, and it never appears in the pricing.
For a generative system, add the retrieval question. If the product answers from your documents, ask what happens when the retrieval step returns nothing relevant. A system that says it does not know is engineered. A system that answers anyway from the base model is a liability wearing your logo, and it will be confidently wrong in front of a customer within the first month.
Price the thing at your volume, not at pilot volume
Ask for cost per unit of the thing you pay for: per document, per case, per claim, per seat per month, at your projected production volume rather than at demo volume. Then ask for the same figure at three times that volume, and ask what drives the difference.
Unit economics is where AI companies quietly break. Inference cost scales with usage in a way that seat-based software does not, and a vendor that has only run pilots may not know its own curve. Two specific numbers to extract: the token or compute cost per transaction at the model tier they actually run in production, which is often a smaller model than the one in the benchmark, and the human-in-the-loop cost, meaning what fraction of transactions get reviewed and what that review costs. If 20 percent of outputs need a human and each review takes four minutes, that is the real price of the system, and it dwarfs the license.
The failure mode to name out loud: a vendor priced on pilot volumes discovers its own gross margin at your scale and comes back at renewal with a number that is not a negotiation. Anchor a multi-year price now, with volume tiers written into the agreement.
Where the Real Cost Sits — Typical Document-Processing Deployment
Editorial ordering from practitioner experience. The license is rarely the biggest line. Illustrative, not a measured statistic.
Check the security posture against a framework, not a logo
The security questionnaire tells you what the vendor knows the answer should be. What you want is evidence of a practice. For AI systems specifically, two public frameworks give you a real checklist. The OWASP Top 10 for Large Language Model Applications names the concrete attack classes: prompt injection, insecure output handling, training data poisoning, model denial of service, supply chain vulnerabilities, sensitive information disclosure, excessive agency. MITRE ATLAS is the adversarial-ML analogue of ATT&CK and catalogs real observed techniques against ML systems.
Ask which of those the vendor has tested against and what they found. The answer "we have not seen prompt injection as an issue" from a vendor shipping an agent with tool access and write permissions is the most important sentence in your entire diligence process. Excessive agency is the failure mode where an LLM has a tool it should not have, and the mitigation is architectural rather than a filter you add later.
If the system touches regulated data, the questions get specific. For a health-tech buyer that means the business associate agreement and where de-identification happens. For anything touching federal customers, the relevant control sets are NIST SP 800-53 for federal systems and NIST SP 800-171 for controlled unclassified information on a contractor network, and the question is which controls the vendor implements versus inherits from a cloud provider. A vendor claiming FedRAMP inheritance should be able to name the authorized service and its impact level. Inheriting controls from an authorized platform is legitimate and common. Claiming the platform's authorization as your own is not, and the distinction is easy to check at the FedRAMP Marketplace.
Read the data rights clause before the technical review
Four questions decide what you are actually buying, and they live in the contract rather than the architecture. Does the vendor train on your data, and if so, on what legal basis and with what opt-out. If they do train on it, does a model derived from your data survive termination, and who owns it. What third-party corpora and model weights are in the stack, read at the license itself rather than a summary. And what rights in the output are you granted, which matters if your product incorporates the output into something you resell.
The clause that catches people is the one permitting training on customer data "to improve the service." It sounds like housekeeping. It means your proprietary corpus is improving a model your competitor will also use. Strike it or scope it to your tenant. This is a fifteen-minute redline that people skip because it sits in the section everyone assumes is boilerplate.
Also ask where inference happens. Data residency and subprocessor lists matter, and the subprocessor list for an AI vendor is often longer than they volunteer, because a model API, a vector database, an observability platform, and an evaluation tool may each see your content. Ask for the list in writing and for notification rights when it changes.
Test whether the company can rebuild its own product
Reproducibility is the cheapest proxy for engineering maturity that exists. Ask a simple question: if the model that is in production right now were deleted, how long until you have it back, and what exactly do you re-run.
A mature answer describes versioned training data, a pinned dependency set, a training script under source control, and a registry entry linking the model artifact to the commit and the data snapshot that produced it. An immature answer involves a named person, a laptop, and a notebook. The second is more common than the industry admits, and it converts key-person risk into an existential risk you have now taken on. If the person who trained the production model leaves and nothing is versioned, the vendor cannot retrain, cannot fix a drift problem, and cannot answer a regulator asking what the model was trained on.
The same question exposes drift monitoring. Ask what they measure in production and what triggers a retrain. A system with no drift monitoring is a system whose accuracy is a historical fact rather than a current property, and input distributions move for ordinary business reasons: a new customer segment, a form redesign, a policy change upstream.
Governance frameworks worth naming in the contract
Two references are worth writing into an agreement because they turn vague assurances into checkable commitments. The NIST AI Risk Management Framework, and its companion Playbook, structure the questions of who owns AI risk, what the system is for, and what is measured. It is voluntary, and its usefulness in a contract is that its Govern and Map functions give you a shared vocabulary for the artifacts you are requiring.
ISO/IEC 42001 is the certifiable AI management system standard, and it is the closest thing to an auditable AI governance credential currently available. Whether a vendor holds it matters less than whether they can describe their AI risk process in terms it would recognize. A vendor that has never heard of either framework is not disqualified, but you are now the governance function for that relationship, and you should price that work into your side of the project.
What to write into the agreement: the evaluation set and cadence, the performance floor below which remediation is triggered, notification obligations when the underlying model version changes, and the right to run your own evaluation against the production endpoint on a schedule. That last one is the most valuable clause in an AI contract and the least frequently requested. Model behavior changes when a vendor swaps a base model, and without a testing right you will find out from a customer complaint.
The four-week version
Week one is the paper read: contract, data rights, subprocessor list, security documentation, and the claim inventory, meaning a written list of every sentence the vendor has published asserting the system does something. Each sentence becomes a testable proposition. Week two is the blind run, which you should start early because assembling labeled data from your own operation is slower than anyone expects. Week three is the technical session, working from the results of the blind run rather than from a deck, and covering evaluation history, calibration, failure handling, and reproducibility. Week four is unit economics at your volume, reference calls to customers you sourced yourself rather than the vendor's list, and the redline.
Two people can run this: one who can read a contract and one who can read an evaluation. It costs a few thousand dollars of internal time against a decision that typically commits several hundred thousand, and it changes the answer often enough to be worth doing every time.
Bottom line
Diligence on an AI company is not about auditing the company. It is about producing one piece of independent evidence, on your own data, that the system does the thing. Everything else in this article exists to protect the value of that number: evaluation history tells you whether it will hold, calibration and failure handling tell you what the system costs to operate, unit economics tell you what it costs at scale, data rights tell you what you are buying, and reproducibility tells you whether the vendor can still deliver it in two years. Run the blind test first. The rest of the process is a lot faster once you know whether the thing works.
Frequently asked questions
Who built the evaluation set and when, relative to the model. If the modeling team assembled it afterward from data they considered representative, the accuracy figure measures self-consistency rather than performance, and it will not survive contact with your data.
For most classification and extraction work, 200 to 500 labeled examples drawn from your real distribution is enough to see a meaningful gap. Include the hard cases and the edge cases rather than a clean sample, since the clean cases are the ones the vendor already handles.
It establishes that someone at the vendor knows what the correct answers are. Evidence of practice is different: incident records, penetration test results with the findings and remediation dates, and specific answers about the OWASP LLM Top 10 and MITRE ATLAS technique classes relevant to the architecture.
A right to evaluate the production endpoint on a schedule, notification when the underlying model version changes, a performance floor that triggers remediation, and a scoped or struck clause on training with your data. Model behavior changes silently when a vendor swaps a base model.
The evaluation and calibration questions apply identically, and the data rights and residency questions get more important because the surface is thinner. The reproducibility question shifts from the vendor to you, since you now own the pipeline built on top of the API.