The thing you are actually buying
Nobody sets out to buy rule extraction. They set out to replace a system that a small number of people understand, that runs the pricing or the eligibility or the settlement, and that cannot be turned off for a weekend. The rewrite quote comes back at three years. Somebody asks the reasonable question: could AI just read the old code and tell us what it does? The answer is yes, partially, and the partial is the whole story.
What a legacy system holds is not one artifact but three, tangled together. There is intent, the policy somebody decided in 1997. There is implementation, the code that mostly carries out that policy. And there is accident, the behavior that emerges from a rounding choice, a field width, a sort order, a patch applied under pressure on a Friday. Downstream consumers have been calibrating against all three for decades without distinguishing them. Your replacement has to reproduce the accidents too, or somebody's reconciliation breaks and it becomes your fault.
AI tooling is good at the second of these and blind to the first. Given a COBOL paragraph, a modern code model will produce a fluent English description of what the code does. That description is a hypothesis. It is often right. When it is wrong it is wrong in a way that reads exactly like being right, which is the specific property that makes an unharnessed extraction project dangerous rather than merely disappointing.
What Recovers Cleanly, and What Does Not
Practitioner reading of where static analysis plus code models land on typical batch systems. Directional, not a measured benchmark.
What the tooling genuinely recovers
Start with the parts that work, because they are worth real money and they are unglamorous. A parser plus a call-graph builder gives you the shape of the system in days rather than months: which programs call which, which copybooks are shared, which modules have no live caller. On the systems we see, somewhere between a fifth and a third of the source is unreachable. Establishing that early removes it from every estimate that follows, and it is a deterministic result, not a model's opinion.
Data layout recovery is the second reliable win. COBOL copybooks and their VB6 and PL/I equivalents encode types, widths, sign conventions, packed-decimal choices, and redefinitions. Extracting them mechanically produces a schema that the new system needs anyway. A code model adds value here by proposing human names for fields called WS-FLG-7, which is a naming suggestion a person can accept or reject in seconds. Low risk, high throughput.
The third is program slicing. Ask "what affects the value of this output field" and a slicer walks the data-dependence graph backwards to the subset of statements that matter. This is decades-old compiler technology, not machine learning, and it is what makes a code model useful at all: instead of summarizing a 12,000-line program, the model summarizes a 200-line slice that provably contains the logic for one field. Same model, far better answer, because the context is correct.
Where it fails, and the shape of the failure
The failures are not random. They cluster in four places, and each has a specific tell.
Rules that live between programs. Batch systems distribute a single business rule across a job stream: step 40 writes a flag file, step 70 reads it, and the actual policy is the interaction. Neither program contains the rule. A model summarizing each program separately will describe two mechanisms and never mention the policy, and nothing in its output signals the omission.
Arithmetic that is not what it looks like. COBOL COMP-3 packed decimal with an explicit PICTURE clause does fixed-point arithmetic with truncation rules that differ from IEEE 754 binary floating point. Reimplement a premium calculation in a language with double-precision floats and you get answers that agree to six decimals and disagree at the cent, on maybe one record in four thousand. That is a defect nobody finds in user acceptance testing and everybody finds in the first month-end close.
Dead branches described as live. A model reads the source, so it describes every branch as though the branch runs. A condition gated on a product code retired in 2011 gets written into your requirements, priced, built, and tested. The tell is that the specification is larger than the observed behavior of the system.
Confabulated coherence. This is the expensive one. Asked to explain a messy stretch of code, a model produces a clean story, because clean stories are what its training distribution rewards. The narrative smooths over the exception that a maintenance programmer added in 2013 for one customer segment. Everything reads plausibly. It is wrong only where it matters.
| Failure | How it shows up | What actually catches it |
|---|---|---|
| Cross-program rule | Two accurate summaries, policy missing | Job-stream dataflow across steps, not per-program reading |
| Fixed-point drift | Cent-level mismatch on a small fraction of records | Shadow run on production volume, exact-equality diff |
| Dead branch in spec | Requirements larger than real behavior | Production data coverage per branch before it is written down |
| Confabulated summary | Reads well, fails one segment | Every rule carries a line citation a reviewer opens |
| Undocumented intent | Rule is correct and nobody can approve it | Named business owner per rule, or it stays quarantined |
Every rule cites a line, or it does not exist
One rule kills most of the confabulation risk. A rule statement is only admitted to the catalogue if it carries a machine-checkable citation: program, paragraph, line range. A reviewer clicks it, sees the source, and confirms in about ninety seconds. Rules without citations are not rules. They are notes.
This sounds like process overhead. It is the opposite. It converts review from "read the model's essay and judge whether it feels right", which is slow and unreliable, into "check that these eleven lines say what this sentence says", which is fast and reliable. A domain expert who cannot read COBOL can still do it, because the slice is small enough to reason about with a colleague present.
It also fixes the audit problem later. When a regulator, a customer, or your own internal audit asks why the new system applies a particular threshold, the answer is a pointer into the system of record that produced it, not a recollection of a workshop. On anything touching financial reporting under SOX or a model-risk regime shaped by SR 11-7, this traceability is what makes the modernization defensible rather than merely finished.
Production data outranks source code
The source tells you what could happen. The data tells you what does. Before a single rule is written down, instrument the legacy system or mine its logs and outputs to establish which paths actually execute, at what volume, for which segments.
Three things fall out. Branches with zero production traffic over a full business cycle get quarantined rather than specified, which typically removes 20 to 40 percent of apparent scope. Branches with tiny traffic get flagged for a decision by a named person, because those are usually where a single large customer or a regulatory exception lives. And the distribution of real inputs becomes the test corpus, which is worth more than any generated one because it contains the shapes nobody would think to invent.
A full business cycle means what it says. Annual true-ups, quarterly filings, and open-enrollment paths fire on a calendar most sampling windows miss. Thirty days of traffic will confidently tell you a year-end reconciliation routine is dead code. Where a full year is not available, sample the calendar deliberately rather than taking a contiguous month, and treat the un-observed branches as unknown rather than absent.
The harness that makes it contractual
The extraction is not the deliverable. The comparison is. Build a shadow-run harness before you build anything else: production inputs feed both the legacy system and the new implementation, and outputs are compared field by field with exact equality on anything monetary or identity-bearing. Not sampled. Not tolerance-based. Exact, on real volume, every night.
This is what turns extraction from a judgment call into a measurement. You stop arguing about whether the rules were captured correctly and start reading a number: 4,812,003 records processed, 41 mismatches, here they are grouped by pattern. Each mismatch is either a bug in the new code, a rule that was missed, or an accident of the old system that somebody now has to decide about. All three are progress, and all three are visible.
Make the number the acceptance criterion. "Rule extraction complete" is not a milestone anybody can verify. "Thirty consecutive days of shadow running at zero unexplained mismatches on production volume, with every explained mismatch signed off by a named owner" is a milestone that either happened or did not. If a firm proposing this work resists writing that clause, the reason is worth understanding before you sign.
Decision tables, not prose
Rules recorded as English paragraphs cannot be tested, cannot be diffed, and quietly contradict each other. Record them as decision tables instead: conditions across the top, outcomes down the side, one row per combination. The format is old and it works, because it makes two properties mechanically checkable.
Completeness: does every combination of input conditions map to an outcome, or is there a hole where the legacy system does something undefined that production has been relying on. Consistency: do two rows claim different outcomes for the same inputs, which usually means an ordering dependency exists in the old code that nobody documented. Both checks are automated and both find real defects in the first week.
The tables also generate their own tests. Every row becomes at least one case in the shadow harness, so coverage of the rule catalogue is a computed number rather than an assertion. And when a business owner wants to change a rule after cutover, they edit a row, and the change has a diff, a date, and a name attached to it.
What this costs and how long it takes
For a single subsystem in the range of 150,000 to 400,000 lines with a bounded set of outputs, discovery and structural analysis run four to eight weeks. Rule extraction with the citation and review discipline runs another eight to sixteen. Shadow harness construction overlaps discovery and takes three to six weeks, mostly spent on getting a production data feed approved rather than on writing code. Budget the engineering at roughly 300,000 to 900,000 dollars for that scope, and expect the range to be driven by data access, not by lines of code.
The number that matters more is the ratio. Full-system extraction on a multi-million-line estate is a program, not a project, and the honest sequencing is subsystem by subsystem behind a strangler boundary. Attempting the entire estate at once reproduces the failure mode that made the original rewrite quote three years long, with an AI budget line added.
Where the Effort Goes on a Bounded Subsystem
Relative calendar effort on a 150k-400k line subsystem. The cheapest line item is the one the sales deck is about.
Two things reliably blow the schedule, and neither is technical. Getting production data into a place where the harness can use it takes longer than any engineering step, because it crosses legal, privacy, and security review. Start that on day one. And rules that are correctly extracted but that nobody in the business will own remain open indefinitely; the fix is to name owners in the kickoff, per rule family, before the extraction starts.
Where the data is sensitive
Shadow running means real records in a test environment, which is where most of these programs meet their compliance constraint. If the payload is PHI, the harness sits inside the HIPAA boundary or it does not run. If the system touches federal data, the relevant control families in NIST SP 800-53 and, for controlled unclassified information on a contractor system, the requirements in NIST SP 800-171 govern where records may be processed and what has to be logged.
Three design choices keep the harness inside the boundary without gutting its value. Compare hashes rather than values for fields you do not need to inspect, so a mismatch is detectable without the record leaving the enclave. Keep the model out of the data path entirely, since it reads source code, not records, and there is no reason for production values to reach an inference endpoint. And if a hosted model reads proprietary source, confirm the retention and training terms in writing, because source code for a system that runs your pricing is a trade secret whose disclosure has no remedy.
When those constraints make a hosted model unusable, an open-weight code model on your own hardware handles slice summarization at acceptable quality. The slicing and citation discipline is what carries the accuracy, so the harness tolerates a weaker model far better than it tolerates a weaker process.
Questions to ask a firm before you hire one
Ask how a rule gets into their catalogue, and listen for whether a citation is mandatory or aspirational. Ask what happens when the model and the shadow run disagree, and whether the answer is a defined procedure or an improvisation. Ask how they establish that a branch is dead, and whether the evidence is production traffic over a full cycle or an inspection of the code.
Ask what they do about fixed-point arithmetic specifically. It is a narrow question with a right answer, and it separates people who have shipped a financial cutover from people who have read about one. Ask them to name the acceptance number they are willing to sign, on production volume, for how many consecutive days.
Then ask what they expect from you. A firm that has done this will ask for production data access, a named business owner per rule family, and a decision path for accidents that turn out to be load-bearing. A firm that asks only for source code has not thought past the demo.
Bottom line
AI-assisted rule extraction is real and it moves the schedule. It moves it by making a person's review fast rather than by removing the person. The tooling recovers structure, data layouts, and per-slice logic well; it does not recover intent, it is unreliable on arithmetic as executed, and it will describe dead code as live behavior with complete confidence. Every one of those failures is caught by the same two mechanisms: a line citation on every rule, and a shadow run against production volume with exact-equality comparison.
Buy the harness, not the model. The harness is what you can put a number in a contract about, and the number is the only part of this that a year from now will still be true.
Frequently asked questions
It produces a fluent draft quickly, and the draft contains rules that are wrong in ways that read as correct: logic spanning job steps, branches dead for a decade, arithmetic that differs at the cent. Useful as input to a review process with line citations and a shadow run behind it. On its own, not something anyone can safely act on.
Not from the code, because both look identical there. You surface the behavior in a shadow run, attribute it to a named owner, and ask whether it is required. Some accidents are load-bearing because a downstream consumer has calibrated against them for years, which is why they need a decision rather than a deletion.
Usually, with the harness inside the existing compliance boundary. Where records cannot be copied, compare cryptographic hashes of output fields so a mismatch is detected without the values leaving the enclave, and reserve full-value inspection for the small set of records that actually differ.
On the batch systems we see, a fifth to a third of source has no reachable caller, and a further slice is reachable but receives no production traffic across a full business cycle. Establishing both numbers early is the cheapest scope reduction available, and it is deterministic rather than a model judgment.
Decision tables. They support automated completeness and consistency checks, they generate their own test cases into the shadow harness, and a business owner can change one row with a diff and a date attached. Prose rules cannot be tested and contradict each other silently.
A dead-code map inside four to eight weeks on a bounded subsystem, and first shadow-run diffs on real volume within roughly a quarter if data access started on day one. That first diff report is the earliest honest signal.