Which source of truth wins: the code, the documentation, or the operator's memory

Three witnesses, three stories

You are about to modernize something that already exists. A claims adjudication engine, a ratings pipeline, a device-data ingest path, a scoring service that four downstream teams depend on. Before anyone writes a line of replacement code, someone has to answer a simple question: what does the current system actually do? You will get three answers. The repository says one thing. The design document and the control narrative say another. The operator who has run the thing for nine years says a third. All three are given in good faith. None of them is complete. The project's schedule is decided by how quickly you work out which to trust for which kind of question.

The instinct in engineering circles is to say the code wins, because the code is what runs. That instinct is right about a narrower set of questions than people assume, and it is dangerously wrong about the rest. The code tells you what happens. It does not tell you what was supposed to happen, what the business agreed to, what an examiner was told, or which of the forty branches has been dead since a migration in 2019. Those answers live in the other two witnesses, and a team that treats the repository as the whole truth will faithfully reimplement bugs, preserve dead code, and lose the one constraint that mattered.

What follows is the method we use on discovery engagements, and it is deliberately mechanical. Not a philosophy of documentation. A procedure for cross-examining three witnesses on a deadline.

What the code is a reliable witness to

The repository is authoritative about mechanism and only about mechanism. It will tell you exactly how a value is computed, in what order operations run, what the retry behavior is when a call fails, and what the field types are. If a question is of the form "what happens when," the code answers it and nothing else does.

It is a poor witness to four things, and each one has burned a project we were later called to rescue. It cannot tell you which paths are live. In a fifteen-year-old codebase, somewhere between a fifth and half of the conditional branches have not executed in years, and reading them as requirements inflates scope enormously. It cannot tell you intent, so a rounding rule that exists to satisfy a regulator looks identical to a rounding rule someone added to make a test pass. It does not contain the configuration and the reference data, which in most enterprise systems carry more business logic than the source does. And it says nothing about the environment: the cron entry on a server nobody documented, the file drop from a partner, the manual step someone performs on the third business day.

The practical rule we work by is that the code is the source of truth for behavior, and the runtime is the source of truth for which behavior matters. Those are different, and conflating them is the single most expensive mistake in legacy discovery.

What the documentation is a reliable witness to

Documentation gets a bad reputation it half deserves. Design documents rot because nothing forces them to change when the code does, and a document that has drifted is worse than none because it is confidently wrong. But one class of document does not rot in the same way, and teams routinely ignore it: the document somebody signed.

A control narrative attached to a NIST 800-53 assessment, a model risk document written to SR 11-7, an ISO 42001 statement of applicability, a validation report filed with a regulator, a data processing agreement. These carry a commitment. If the code has drifted away from one of them, that drift is not a documentation problem to be fixed by updating the document. It is a finding. The gap between the signed narrative and the running system is often the most valuable output of a discovery engagement, because it is the thing an auditor will find eventually, on a worse day, with less time to fix it.

The distinction is worth making explicit to your own team. Unsigned documents are hypotheses about the system. Signed documents are constraints on the system. A hypothesis that disagrees with the code loses. A constraint that disagrees with the code means somebody has a problem, and the first job is to work out who.

What the operator knows that nothing else records

The operator is the witness engineering teams undervalue most and the one whose testimony is hardest to reconstruct after they leave. What they hold is causal history. Not what the system does, but why it does it, and what happened the two times somebody changed it.

Ask a nine-year operator why the reconciliation job runs at 4:10 a.m. rather than 4:00, and you will get an answer about a partner file that arrives late on the last business day of the month, which is not in any document and is not inferable from the code. Ask why a particular customer segment is excluded from an automated path and you will hear about a complaint in 2021 and a decision made in a meeting. Ask what breaks first under load and they will name the component, correctly, in about four seconds.

Their limits are equally specific. Human memory reconstructs rather than replays, so operators are confidently wrong about ordering, about thresholds, about which release changed what. They describe the system as it was when they last had to think about it hard, which may be four years ago. They know their own surface and describe adjacent surfaces from hearsay. So the rule is: the operator is authoritative about why and about the existence of things, and unreliable about exactly. Take the pointer, verify the number.

The reconciliation, done in two weeks

Here is the procedure. It is not elaborate, and its whole value is that it produces disagreements early, when they are cheap.

Start by writing down between fifteen and thirty questions whose answers would change the design of the replacement. Not an inventory of the system. Questions with consequences: what determines whether a record goes down the automated path, what is the actual latency requirement as opposed to the stated one, what happens to a partially processed batch when the job dies at 3 a.m., which downstream consumers read this table directly rather than through the API. That list is the scope of discovery, and it should fit on two pages.

Then answer every question three times, once from each witness, and record the three answers side by side without trying to resolve them yet. This feels wasteful for about a day and then starts paying. The questions where all three agree are settled and you stop spending time on them, which is usually half the list. The questions where they disagree are the project's real risk register, and you now have it in week two instead of month five.

Question type	Ask the code	Ask the document	Ask the operator
Computation and ordering	Authoritative. Read the path, then confirm with a trace.	Treat as a hypothesis to test	Directionally right, wrong on specifics
What is actually used	Silent. Coverage data and logs answer this, not source.	Usually overstates scope	Knows which screens people open
Why a rule exists	Silent, unless a comment survived	Best when the document was signed	Authoritative. This is the whole reason to interview.
External commitments	Silent	Authoritative when signed and dated	Remembers the negotiation, not the wording
Failure behavior	Shows the handler that exists	Describes the handler somebody intended	Knows what actually happens at 3 a.m.
Manual and out-of-band steps	Silent by definition	Almost never recorded	The only witness. Ask directly and specifically.

The third step is the one that settles arguments: for every disagreement, design a cheap observation that decides it. Not a debate, an experiment. Add temporary logging to the branch in question and watch a week of production. Query the audit table for how many records took each path last quarter. Replay a day of real inputs through the current system and record the outputs. Pull the runtime configuration and diff it against what the document claims. Each of these costs hours and settles a question that could otherwise consume a quarter of rework.

The observation methods that pay for themselves

Four techniques do most of the work, and none of them requires the vendor's cooperation or a large budget.

Production coverage. Instrument the running system to record which code paths execute, then let it run through a full business cycle, which for most enterprise systems means one month plus a quarter-end. What comes back is usually startling. On systems of this age, it is routine to find that a large minority of the branches never fire, and the replacement scope shrinks accordingly. This one measurement has moved more estimates than any conversation we have ever had.

Golden-record replay. Capture real inputs and the current system's outputs for a representative window, including the ugly cases: the reversal, the backdated correction, the record with a null where nulls are supposedly impossible. That corpus becomes the acceptance test for the replacement, and it is worth more than any written specification because it encodes behavior nobody remembered to write down. Build it during discovery, not during the build, so the schedule reflects what it teaches you.

Configuration and reference data extraction. Pull every configuration file, database-held rule table, feature flag and lookup, and treat them as first-class artifacts. In claims systems, ratings engines and eligibility platforms this is frequently where the majority of the actual business logic lives, and it is the part most likely to be missing from both the repository and the design document.

Structured interviews with the exceptions as the agenda. Do not ask an operator to describe the system. Ask about the last five times something went wrong, what they do that is not in a runbook, and what they would tell their replacement in the final hour of their final day. Two ninety-minute sessions with the right person, recorded and transcribed with permission, yields more than a month of reading.

Note the bottom row. Pure "we cannot work out what this code does" is the rarest problem. The expensive findings are almost always about what surrounds the code rather than the code itself, which is exactly why a discovery approach built only on reading the repository misses them.

When the model is the system

Machine learning adds a fourth witness that the other three cannot substitute for: the training data and the artifact itself. The code shows the training script as it exists today, which may not be the script that produced the model in production. The document describes the model as validated. The operator knows when it started behaving oddly. None of them tells you what the deployed weights actually encode.

The reconciliation questions are specific here. Which artifact is serving right now, and can you tie it to a commit and a dataset version. What was the evaluation set, and does it still resemble current traffic. What did the validation report claim, and does the live system still meet it. Firms working under SR 11-7 have a head start because model risk management already demands most of these answers, and the documentation practice around AI systems is converging on the same questions through the NIST AI RMF Map and Measure functions. If you have those artifacts, use them as the fourth witness. If you do not, the honest finding is that the model's behavior is only knowable by measurement, and you plan a measurement.

What this costs and what it buys

For a single substantial system of the kind we have been describing, discovery of this shape is typically three to six weeks of one or two senior engineers, plus roughly four to eight hours total of each operator's time and access to production telemetry. At the rates senior engineers on this work command, generally $150 to $250 an hour however they are billed, that lands somewhere between $35,000 and $90,000. Add a week if the system is regulated and the signed documents have to be read carefully rather than skimmed.

What it buys is an estimate you can defend. The replacement projects that go badly are almost never the ones where the engineering was hard. They are the ones where month five produced a surprise that invalidated the plan: a consumer nobody knew about, a manual step nobody mentioned, a regulatory commitment nobody read. Every one of those is findable in week two for a few thousand dollars of instrumentation, and the asymmetry is not close.

There is a version of this that is not worth doing. If the system is small enough that one engineer can read it end to end in a week, read it. If it is being retired rather than replaced, capture the golden records and the interviews and skip the rest. And if the real decision is whether to build at all, a shorter engagement aimed at that single question serves better than a full discovery. We say so when that is the case.

Stopping the disagreement from coming back

Once you have reconciled three witnesses, the useful question is why they diverged, because the replacement system will diverge the same way unless something changes. The mechanism is always the same. Each source of truth drifts at the speed of whatever forces it to be correct, and documents have no forcing function at all.

So put one there. The techniques are unglamorous and they work. Generate what can be generated, so interface documentation and data lineage come from the artifacts rather than from someone's discipline. Make the runbook executable, so the steps are scripts that run in the incident rather than prose somebody reads and interprets. Put the operator's causal knowledge into tests, so the reason the job runs at 4:10 becomes a test that fails when someone moves it. And keep a short, deliberately small file of things that genuinely cannot be derived, review it quarterly, and treat its growth as a warning that something which should be emitting evidence is not.

Bottom line

Three witnesses, three jurisdictions. The code is authoritative about mechanism and silent about intent. The signed document is authoritative about commitment and unreliable about current behavior. The operator is authoritative about why and about the existence of things nothing else records, and imprecise about numbers. Nobody wins in general. The team that gets this right is the one that stops arguing about which witness to trust, writes down the twenty questions that would change the design, answers each one three times, and spends a few hours of instrumentation to settle every disagreement before the build starts. That work is cheap, it is fast, and it is the difference between an estimate that holds and one that discovers its own error in month five.

Frequently asked questions