The two jobs, and why conflating them fails
A company doing a few hundred to a few thousand third-party agreements a year has two contract problems that share a name and nothing else. Before signature, someone is reading vendor paper and customer redlines against positions the company will and will not accept. After signature, someone is trying to find out what is in four thousand live agreements, usually at the worst possible moment — a diligence request, an insurance claim, a renewal that already auto-extended. These need different data, different tooling and different measures of success. Most disappointing contract projects are one product aimed at both.
Pre-signature is a comparison problem: this clause against our position. Post-signature is a retrieval and inventory problem: which of our agreements say something about this, and are we sure we are looking at the operative version. The first is bounded and fast. The second is where the money hides and where the document set will defeat you.
You are probably here because
- Sales is complaining that a routine agreement takes four days to turn around
- A renewal auto-extended for twelve months and nobody had it diaried
- Someone asked which of your contracts restrict assignment on a change of control, and the honest answer was three weeks
- You watched a demo where the tool found a limitation of liability clause and could not tell whether that was impressive
Finding the clause is the part that already works. The parts that decide whether this pays are the playbook, the amendments, and what happens on the third false alarm.
The playbook is sixty percent of the project and it is not software
Ask a general counsel what the company's position is on limitation of liability and you will get a good answer. Ask for the written version with the fallbacks and you will usually get silence, or a memo from four years ago that no longer reflects what actually gets signed. That gap is not a criticism of anyone. Positions accrete through negotiation, and writing them down never makes it above the day's fires.
But it is the whole foundation. A model cannot flag a deviation from a standard that does not exist, and a reviewer cannot be consistent with a standard held in one person's head. The artifact you need is unglamorous: for each clause type, the preferred language, one or two acceptable fallbacks, the point at which it escalates to a named human, and who can approve an exception. Thirty to fifty clause types covers most commercial paper.
Writing that down takes a few weeks of a lawyer's part-time attention, it is the single highest-value thing on this list, and it delivers value with no software at all. Companies that stop there and hand a two-page playbook to their commercial team often capture a third of the benefit of the whole program. If a vendor's pitch does not begin with "who is writing your playbook," you are being sold a search engine.
What the model is actually good at, and where it stops
Modern language models are genuinely good at three things here, and it is worth being precise about which three. They find the clause even when it is titled something else or buried in a schedule. They quote the operative span verbatim rather than paraphrasing. And they compare that span to a reference position and characterize the difference in a sentence.
What they are unreliable at is the judgment that follows: whether this particular deviation matters for this particular counterparty on this particular deal. A mutual limitation at one times fees is a routine acceptance for a small software purchase and an unacceptable term on an agreement where you hold the counterparty's customer data. The clause text is identical. The correct answer is not, and it depends on facts that are nowhere in the document.
That boundary is the design. Extraction and comparison go to the machine. The commercial judgment stays with a person, and the system's job is to put the right five things in front of that person with the source text attached, rather than a summary they have to trust.
Tier the clauses by consequence, not by frequency
The instinct is to treat every clause the same and report every deviation. That produces forty findings on a document with six that matter, and by the third document the reviewer is skimming. Precision, not recall, is what keeps a reviewer engaged on routine terms.
The exception is a small set of clauses where a miss is expensive and a false alarm costs thirty seconds. On those, tune hard toward recall and accept the noise. Every company's list is slightly different, but it usually includes uncapped or carved-out indemnities, assignment restrictions triggered by a change of control, automatic renewal with a long notice window, most-favored-customer pricing, and unilateral amendment or policy-incorporation-by-reference.
| Tier | Examples | Tune for | What happens on a finding |
|---|---|---|---|
| Consequential | Uncapped indemnity, change-of-control assignment, most-favored pricing, unilateral amendment | Recall. Accept false alarms | Always surfaced with the span; a person signs off explicitly |
| Negotiated | Liability cap and carve-outs, IP ownership, data handling, termination for convenience | Balance | Compared to playbook, deviation described, routed by fallback tier |
| Routine | Notice addresses, governing law, counterparts, severability | Precision. Silence is fine | Recorded in the abstract, not raised unless outside the accepted set |
| Commercial | Price, term, volume commitments, service levels | Extraction accuracy | Written to the record for reporting; never interpreted |
The tiering conversation is also the fastest way to find out whether the legal team is actually behind the project. If nobody will commit to a list of five clauses that are worth a false alarm, the project is not ready.
The document set beats the extraction problem
Everyone budgets for extraction accuracy. Almost nobody budgets for the discovery that the contract repository is not what it claims to be. In the sets we have worked with, a meaningful share of what is filed as an executed agreement is a draft, an unsigned version, a partially executed copy, or a duplicate under a different filename. Executed versions live in a signature platform, in an inbox, or in a folder someone took with them.
Amendments are the specific killer. An agreement that has been amended three times has an operative answer that lives in the fourth document, and any system that reads the base agreement and returns a confident clause has produced a wrong answer with a citation attached. That is worse than no answer, because the citation makes it credible. Assembling the family — base, amendments in order, order forms, statements of work, incorporated policies — is a prerequisite, not a refinement.
Incorporation by reference is the quieter version. Vendor paper routinely binds you to terms on a web page, and the operative language is whatever was published on some date. If your record does not capture that page as of signature, you do not actually have the agreement.
Practical sequence: before extracting anything, run a census of the repository. How many files, how many families, how many have an executed signature page, how many are duplicates, how many reference a document you do not hold. Show that census to the general counsel. It is uncomfortable and it is the most useful single deliverable in the first month.
Where post-signature projects actually lose time
Our planning split from delivery experience, not a published statistic. The last row is the one every proposal is written about.
Refuse to answer, out loud
The most important behavior in a contract system is the one that produces nothing. If the clause is not present, the system must say the clause is not present. If the answer depends on a document not in the set, it must say which document is missing. If the language is genuinely ambiguous, it must show the text and decline to characterize it.
This is harder than it sounds, because a language model asked a direct question will nearly always produce a direct answer, and a plausible wrong answer about an indemnity is a liability wearing the costume of a productivity gain. The engineering that prevents it is not clever prompting; it is structure. Every extracted value carries a span pointer to the source document, page and character range. A value with no span is not returned. A field has three distinct states — found, absent, unresolved — and the interface shows the difference rather than rendering all three as an empty cell.
Then measure the refusal behavior explicitly. On a test set where you have deliberately removed the relevant clause, how often does the system correctly say so? That number is rarely reported and it is more predictive of whether lawyers will keep using the tool than any accuracy figure.
Measure on a gold set, not on a demo
A demo on a document the vendor selected proves nothing. Build a gold set: one hundred fifty to three hundred of your own agreements, abstracted by a person who is willing to defend each answer, spanning your real mix of counterparty paper. It takes a few weeks and it is the only way you will ever know whether anything changed.
Score per clause, not overall, because an eighty-nine percent average conceals a clause type that is right forty percent of the time. Separate the error classes: wrong value, missed a present clause, invented a clause that is not there, and got the right answer from the wrong document. That last one is the amendment failure and it is invisible in an accuracy number.
Then measure the thing the business actually cares about, which is minutes. Time a reviewer on twenty agreements without the tool and twenty with it. If manual review runs twenty-five to forty minutes on a routine agreement and assisted review runs twelve to eighteen, that is a real and defensible claim. It is also a much more modest claim than the one in the sales deck, and it is the one that will still be true in a year.
Post-signature: obligations, dates and the questions you cannot answer today
Once the families are assembled and the abstraction is trustworthy, the return is not review speed. It is that a category of question becomes cheap. Which agreements auto-renew in the next ninety days and what is the notice deadline for each. Which counterparties can terminate for convenience on thirty days. Where do we owe a most-favored price. Which agreements would require consent if the company were sold.
The auto-renewal one is worth stating plainly because it is the easiest money in the entire program. A long notice window on a mid-six-figure annual subscription that nobody diaried is a full year of spend the company did not choose. Most legal teams can name at least one. A calendar built from abstracted notice windows, with alerts to a named owner rather than to a shared mailbox, pays for a good deal of the work by itself.
The change-of-control question is the other one, and it arrives on a deadline. A company that can answer it in an afternoon rather than three weeks is not saving legal hours; it is removing a rate-limiting step from a transaction. That is a different kind of value and it does not show up in any per-document time study.
What this costs, roughly
Ranges rather than a number, because the document set dominates. A pre-signature playbook plus assisted review for a defined set of agreement types is typically a two-to-four month build for a small team, and the playbook work runs in parallel and belongs to the legal group. A post-signature program over several thousand agreements is longer, and the schedule is set by repository condition rather than by engineering.
The honest comparison for a one-time abstraction is outsourcing. Managed abstraction services price roughly in the tens of dollars to a few hundred dollars per agreement depending on depth. If you have two thousand agreements and no ongoing inflow, buying the abstraction once is very likely cheaper than building anything, and we will tell you so. Building wins when there is a continuing stream, when re-abstraction will be needed as standards change, or when the ability to click from a number back to the clause is itself a requirement.
When you do not need software
Some cases where the answer is no, or not yet.
Under about forty third-party agreements a month with a stable template set. A written playbook, a one-page clause summary that the commercial team fills in, and a fifteen-minute weekly triage will get most of the benefit. Adding a system adds a system to maintain.
Your paper wins most of the time. If ninety percent of your agreements are on your own template with light edits, the review burden is already small, and the gain is in tightening the template and the approval matrix rather than in reading faster.
Nobody will own the playbook. If the legal team cannot commit the hours to write positions down, buying software will not produce them, and the project will stall four months in with an expensive search interface and no standard to compare against.
The repository is the problem and everyone knows it. Then the first project is a file census and a single place for executed documents. It is unglamorous, it takes a couple of months, and every later thing depends on it.
A sequence that works
First cycle, pre-signature and post-signature in order
Step one before step four is the whole argument of this article. A team that extracts before it knows what it has produces a clean-looking database of answers drawn from superseded documents, and the failure will not surface until somebody relies on it.
The mistakes we get called in to fix
- Clause answers pulled from a base agreement that three amendments had already superseded
- Forty findings on every document, so reviewers approve without reading by the second week
- Summaries instead of spans, leaving lawyers to re-open the source to trust anything
- Absent, unresolved and blank rendered identically as an empty cell
- Accuracy reported as one average, hiding a clause type that is right four times in ten
- Renewal alerts sent to a shared mailbox that nobody owns, which is the same as no alert
Before you buy or build
- A written playbook exists, with fallbacks and a named escalation owner per clause
- Five clauses are designated as worth a false alarm, and legal agreed to the list
- The repository census is done and the executed-version rate is known
- Amendments and incorporated documents are resolved into families before extraction
- Every value carries a span; a value without one is not returned
- Found, absent and unresolved are three visibly different states
- A gold set of your own agreements exists and is scored per clause
- Review minutes were measured before, so the after number means something
Bottom line
Contract review at volume is two projects. The pre-signature one is limited by whether anyone has written the company's positions down, and that work needs a lawyer and a few weeks, not a purchase order. The post-signature one is limited by whether you can tell which document is operative, and that is a records problem that extraction accuracy cannot rescue. Solve those two and the technology is straightforward and genuinely useful: find the clause, quote it, compare it, say plainly when it is not there. Skip them and you will buy a fast way to produce confident answers from the wrong version of the agreement.
Frequently asked questions
No, and the framing causes trouble. It can find the clause, quote it, and tell you how it differs from your standard position, which removes most of the reading. Whether a given deviation is acceptable depends on commercial facts that are not in the document, and that judgment stays with a person.
Resolve documents into families before extracting anything, order the amendments, and answer every clause question against the family rather than the base agreement. If an amendment is referenced but missing from the set, the system should report the field as unresolved instead of answering from the base document.
Often yes. If you have a fixed backlog of a few thousand agreements and no continuing inflow, managed abstraction is usually the lower total cost. Building makes sense when the flow is ongoing, when standards will change and force re-abstraction, or when clause-level provenance is itself a requirement.
Ask instead for per-clause numbers on your own gold set, split into wrong value, missed clause, invented clause and right answer from the wrong document. A single average is not usable, and the last error class is the one that causes real damage while looking like success.
Usually the renewal and notice-window calendar. Abstract the term, renewal mechanics and notice period across live agreements, then route each alert to a named owner. One missed notice window on a mid-size annual subscription frequently covers a large share of the program cost.