Why your AI pilot died in legal review

The pattern repeats across industries. A pilot exceeds its targets, the business sponsor is enthusiastic, the vendor is ready — and the project spends five months in legal and risk review and quietly does not proceed. It is rarely a rejection. It is a set of unanswerable questions.

Question one: what happens when it is wrong

Not the error rate. The consequence path. Who is accountable, how is the error detected, what is the remediation, and what is the exposure to the affected party.

Pilots measure accuracy and almost never define this. A pilot that arrives with a documented error taxonomy — categories of failure, detection method, remediation, and owner — passes a review that an accuracy number does not.

Legal is not asking whether it works. They are asking what happens the day it does not.

Question two: where did the data go

Whether customer or employee data left the boundary, whether it was retained, whether it trained anything, and whether the contract with the data subject permitted it.

Pilots frequently run on a copy of production data under a handshake because the pilot is small. Legal has to evaluate the production arrangement, and if the pilot's data handling was informal there is nothing to evaluate — which reads as a finding rather than an absence.

Question three: can you show what it did

For any past output, can the organization reconstruct the inputs, the version, and the reasoning path. This is where most pilots fail outright, because logging was scoped for debugging rather than for reconstruction.

• Input, output, model version, prompt or configuration version, timestamp, user
• Retrieved records where retrieval occurred
• Retention long enough to cover the relevant dispute or examination window
• Storage that satisfies the same protection rules as the underlying data

Adding this after the fact is expensive and produces no history. Adding it during the pilot costs little and produces the evidence that the review needs.

Question four: what did the vendor agree to

Indemnification for output-caused harm, intellectual property warranties on generated content, notice before model changes, and audit rights. Standard software terms address none of these adequately.

Negotiating them after a successful pilot is the weakest possible position, because the business has already committed. Raising them at the pilot contract stage costs nothing and reveals early whether the vendor can meet them at all.

The correction is cheap and early

Bring legal and risk into the pilot design rather than the pilot review, and ask a single question: what would you need to see to approve this for production. Then build the pilot to produce those artifacts alongside the performance results.

It adds a small amount of work to the pilot and removes the months. It also filters out the pilots that were never going to be approvable, which is worth knowing in week two rather than month seven.

The artifacts that answer the four questions

Each question has a corresponding artifact. Produced during the pilot they cost days; reconstructed afterward they cost months, and some of them cannot be reconstructed at all.

The third row is the one that cannot be recovered. Logging added after a pilot produces no history of the pilot, so the very period the organization wants to reason about is the period with no evidence.

The error taxonomy is more useful than the error rate

A single accuracy figure tells a risk committee almost nothing, because it aggregates failures with completely different consequences. A taxonomy separates them and turns an abstract concern into a set of manageable items.

For each failure category the taxonomy states what it looks like, how it is detected, who is affected, what remediation is available, and who owns it. A category with no detection mechanism is immediately visible as the gap it is — which is uncomfortable and considerably better than discovering it during an incident.

Committees respond to this well, and the reason is worth understanding: it demonstrates that the team has thought about failure concretely. A high accuracy number with no failure analysis reads as optimism. A modest accuracy number with a complete taxonomy and remediation paths reads as engineering, and it is far more likely to be approved.

Logging for reconstruction, not debugging

Application logging answers "why did this request fail." Governance logging answers "what did the system do, for whom, on what basis, eight months ago." These have different contents and different retention, and the first does not become the second by increasing verbosity.

• Request input as received, and output as returned to the user
• Model identifier and version, plus prompt or configuration version
• Retrieved record identifiers where retrieval occurred
• The acting user, and the authorization under which the request ran
• Timestamp, and any human override with its actor and rationale
• Retention covering the relevant dispute or examination window, with protection matching the underlying data

The retrieved-records item is frequently omitted and frequently the one that matters most, because a wrong output caused by retrieving the wrong document is indistinguishable from a wrong output caused by the model unless the retrieval was recorded.

The terms, and why timing decides them

Four contract terms consistently matter for AI components and consistently absent from standard software agreements.

Indemnification for output-caused harm. Standard agreements indemnify for infringement and breach, not for consequences of what the system produced. IP warranties on generated content, which matter wherever output is incorporated into products or filed externally. Notice before material model changes, without which revalidation cannot be scheduled. And audit or evidence rights sufficient to verify the data handling the vendor described.

All four are negotiable at the pilot contract stage, when the vendor is competing and the business has not committed. None of them are meaningfully negotiable after a successful pilot, when the business has planned around the capability and the alternative is starting over. The timing is the whole game, and it costs nothing to raise them early.

The one-meeting fix

The intervention that prevents nearly all of this is a single conversation before the pilot is designed. Bring legal, risk, and privacy in and ask one question: what would you need to see to approve this for production.

The answer is a list. Build the pilot to produce the items on it alongside the performance results, and the review becomes a review rather than a discovery process.

There is a second benefit that teams underestimate. Sometimes the answer reveals that no achievable artifact would make the use case approvable — the data cannot leave, the decision cannot be automated under existing policy, the vendor arrangement cannot satisfy a binding requirement. Learning that in week two is a good outcome, even though it does not feel like one, because the alternative is learning it in month seven after the business has planned around a capability that was never going to ship.

Reviving a pilot that already stalled

Most teams reading this are not designing a pilot; they are holding one that has been in review for months. The situation is recoverable and the approach differs from starting clean.

The first move is to get the actual objection in writing. Stalled reviews frequently persist because nobody has stated a specific blocker — the project is "under review" and each round surfaces a new question. A written list of what would be required to approve converts an open-ended process into a task list, and sometimes reveals the list is shorter than the delay suggested.

The second move is to accept that the pilot's own history may be unrecoverable. If logging was never in place, the pilot period cannot be reconstructed, and arguing about it wastes the remaining goodwill. Proposing a short instrumented re-run — same system, proper logs, a few weeks — is usually faster than litigating the original.

The third is to separate what needs approval from what does not. Frequently a narrower version of the use case clears review immediately while the full version needs more work, and shipping the narrow one preserves the momentum the program is losing.

Who owns what, stated plainly

The last row is the one that stalls reviews longest, because it has no natural owner and everyone assumes someone else has thought about it. Naming an owner in the first week is a five-minute act that removes a five-month delay.

The version of this that is not bureaucracy

None of this argues for heavier process. It argues for producing four artifacts during work that is happening anyway, at a cost of a few days, instead of reconstructing them under pressure at a cost of months.

A team that runs a pilot with proper logging, a written data flow, an error taxonomy, and pilot-stage contract terms has not slowed down. It has front-loaded the questions that were going to be asked regardless, and it arrives at review with answers rather than with a promise to find them.

Frequently asked questions