The audit trail your ML pipeline does not have yet

The question that breaks the pipeline

Somebody outside your engineering organization asks a narrow question. On March 14 of last year, this applicant was scored 0.41 and declined. Show me why. Not what the model does in general. Why that record, on that day, produced that number. Teams that have built serious ML infrastructure often discover in the following week that they cannot answer it, and that the gap is not in their tooling budget but in what their tooling was designed to record.

The pipeline has an experiment tracker with three thousand runs. It has a feature store, a model registry, a lineage graph rendered nicely in a web console, and a data warehouse holding every row that ever came in. What it usually does not have is a durable link between one production inference and the exact set of feature values, model weights, threshold, and code path that produced it. The parts exist. The join does not.

This shows up as a specific failure. An engineer reconstructs the decision by re-running the record through today's pipeline, gets 0.47 instead of 0.41, and spends two weeks establishing which of eleven changes since March explains the gap. That reconstruction is not evidence. Nobody outside the team will accept a number produced by a system that has changed since the event, and the engineers know it, which is why the two weeks feel so bad.

Reproducibility and auditability are not the same property

Most MLOps investment buys reproducibility: given the same inputs and the same code, produce the same output again. That is a training-time property, and the tools are good at it. Auditability is a serving-time property. It asks what actually happened to one request in production, and it has to answer without re-running anything, because re-running proves what the system does now.

The distinction has a clean test. Unplug your training infrastructure entirely. Delete the notebooks, revoke access to the cluster, assume the two engineers who built the model have left. Can you still describe, from stored records alone, why a particular customer got a particular answer? If answering requires executing code, you have reproducibility. If it requires only reading rows, you have an audit trail.

That sounds like an academic distinction until the calendar gets involved. Regulatory look-back periods run long. Fair lending records under Regulation B are kept for 25 months. HIPAA documentation runs six years. Model risk management practice under SR 11-7 expects records that outlive the model itself. Federal contract closeout can pull records years after delivery. In every one of those windows, the training cluster has been rebuilt twice, the feature pipeline rewritten once, and the library versions no longer resolve.

What one inference record has to contain

The unit of an audit trail is not the model and not the run. It is the single served decision. Each one needs enough written down that a reader with no access to your infrastructure can reconstruct the reasoning. In practice that is seven things, and most production loggers capture two of them.

Element	What it must pin	The usual gap
Decision identity	A stable id for this decision, tied to the business entity and the timestamp of the event, not the log write	The id lives only in the application database and never reaches the ML logs
Resolved feature vector	The literal values fed to the model, after all defaults, imputations and clipping	Only the raw request is logged, so imputed and derived values are unrecoverable
Feature provenance	For each feature: source system, materialization timestamp, and the fact that a fallback fired	A null that was silently filled with a training-set median looks identical to a real zero
Model identity	Artifact content hash, not a version label; plus the container image digest that ran it	A tag like v3 that was rebuilt against a patched base image in June
Decision policy	Threshold, calibration mapping, tie-break, business rules applied after the score	Thresholds edited in a config map without producing a new version record
Raw and final output	Model score, the action taken, and every downstream rule that changed it	Only the final action is stored, so nobody can tell whether the model or a rule caused it
Human interaction	Who saw it, what they were shown, what they did, whether they overrode	Overrides recorded in a ticketing system that has no key back to the inference

The feature provenance row is the one people push back on and the one that pays for itself first. When a score looks wrong in retrospect, the cause is almost never the weights. It is that a feature arrived stale, or missing, or from a fallback path that quietly returned a default. A record that stores the value but not where the value came from cannot distinguish a correct low number from a broken pipeline, and that distinction is usually the whole question.

Point-in-time correctness is the hard part

Feature values change. A customer's account age, transaction count, and risk band all move over time, and most feature stores are built to serve the current value fast. Ask what a feature was fourteen months ago and you get one of two bad answers: the current value, or a recomputation against a source table that has since been corrected, deduplicated, or backfilled.

This is the same problem that causes training-serving skew, seen from the other end. Teams learn to do point-in-time joins for training data so the model does not learn from the future. Far fewer apply the same discipline to the audit path, which needs the reverse capability: reconstructing the past exactly as the system saw it, including the parts that were wrong at the time.

Two mechanisms handle it. Log the resolved vector at inference, which is cheap, direct, and immune to any later change in the upstream data. Or keep bitemporal source tables with both a valid-from and a recorded-at column, so a query can ask what the system believed on a given date rather than what turned out to be true. The first is what we build by default. The second is worth adding when regulators will want the underlying data and not just your record of it, which is common in credit, insurance and clinical work.

The important part is that the choice gets made deliberately. A pipeline with neither will produce a confident, reasonable, and wrong reconstruction, and the fact that it is wrong will be discovered by someone else.

What the frameworks actually ask for

Nobody publishes a schema, but several frameworks converge on the same requirements from different directions, and reading them together is more useful than reading any one.

SR 11-7, the Federal Reserve and OCC guidance on model risk management, is the oldest and still the most concrete. It expects documentation detailed enough that a knowledgeable third party can understand and evaluate the model without help from its developers, plus ongoing monitoring and a record of overrides. That third-party sentence is the operative test, and it predates modern ML by more than a decade.

The NIST AI Risk Management Framework covers similar ground in the Map and Measure functions, with more attention to what gets documented about intended use and measured performance. ISO/IEC 42001, published in 2023, wraps it in a management-system structure that certification auditors already know how to sample: they will pick records and ask you to produce them. The EU AI Act's Article 12 is the most explicit about logging, requiring automatically recorded events over the lifetime of high-risk systems.

None of these dictate a table design. All of them assume the records exist and are queryable by someone who does not work on your team. That is the requirement worth engineering to, and it is stable across all four.

Storage is not the reason you skipped this

The objection is always cost, and the arithmetic rarely survives contact. A resolved feature vector with 200 features, provenance flags, model identity and outputs runs roughly 3 to 6 KB uncompressed, and columnar formats with dictionary encoding routinely take that below 1 KB, because most fields repeat heavily across rows.

At one million inferences a day, three years of retention, and 1 KB compressed, that is about one terabyte. In object storage on a standard tier, roughly 250 to 300 dollars a month, and materially less with lifecycle rules that move records past 90 days into infrequent-access or archive tiers. At ten million a day it is ten terabytes and a few thousand dollars a year. Compare that to two engineers spending three weeks on one reconstruction, which is the cheap version of the alternative.

The genuine costs sit elsewhere. Serving latency, if logging is synchronous, which is why it should be a fire-and-forget write to a queue with the decision id as the key. Schema evolution, because a log written for three years must be readable by a reader written next year, which argues for Parquet or Avro with a schema registry over hand-rolled JSON. And privacy, since a record of exactly what the system knew about a person is precisely the kind of data a retention policy and an access-control review need to cover.

The override table nobody builds

Any ML system with a human in the loop has a second decision layer, and it is almost always invisible. The model scores, a person reviews, sometimes the person disagrees. If that disagreement is not captured with a key back to the inference, three separate things break at once.

You cannot report the real decision rate, because the model's outputs and the organization's actions have diverged by an amount nobody has measured. You cannot retrain honestly, since labels derived from final outcomes are contaminated by override behavior the model never saw. And you cannot answer the fairness question at all, because the pattern of overrides is where disparate treatment usually enters, not the weights.

The fix is small. An override table with the inference id, the reviewer, the timestamp, the original and final outcomes, a structured reason code, and free text. What it costs is discipline in the application layer, where the reviewer tooling has to carry the inference id through the workflow. What it buys is the one artifact that turns a fairness review from an argument into a query.

Retrofitting without stopping the line

Nobody gets to pause production for a quarter. The sequence that works starts at the end and moves backward, because the newest records are the ones most likely to be asked about first.

Start with inference logging on the highest-consequence model, written asynchronously, containing the resolved vector and the model content hash. That is usually two to four weeks and it stops the bleeding, since every day after it ships is a day with a real record. Next, pin model identity: replace tags with content hashes and image digests, and make threshold changes produce a new version rather than an edit. Then wire the override join, which is application work and the piece most likely to need another team's calendar.

Only then go backward. Historical reconstruction is the expensive part and is worth doing selectively, scoped to the retention window that actually binds you and the models that actually carry consequence. A full-fidelity backfill of five years of every model is a project that gets cancelled halfway. A reconstruction of the two models under regulatory scope, for the 25 months that matter, finishes.

A first pass on a single serious model runs six to ten weeks with two engineers. A platform-level build covering several models, point-in-time features, override capture, and a query surface an outside reviewer can use runs three to five months. Those ranges hold when the application team is available. When the override join has to wait for someone else's roadmap, that dependency is the schedule.

Make it queryable by someone who does not work here

The last mile is the one teams skip. Records that only an engineer with cluster access can extract will, in practice, be extracted by an engineer under deadline pressure, formatted into a spreadsheet, and defended in a meeting. That is expensive every time and it never gets cheaper.

What works is a small, boring, read-only surface with three queries behind it. Given a decision id, return the full record. Given an entity and a date range, return every decision about them. Given a model version, return what it served, over what period, with what override rate. Auditors ask variations of those three. Building them takes days once the underlying records exist, and it converts a two-week scramble into a link you send.

Access control on that surface matters as much as the surface. It holds a complete record of what your systems concluded about identifiable people, and the review that granted it should be the same one you would run for the production database itself.

Bottom line

Reproducing a training run and reconstructing a served decision are different engineering problems, and the tooling market has solved the first one thoroughly while leaving the second to individual teams. The gap does not announce itself. It sits quietly until an auditor, a regulator, a customer's counsel, or a plaintiff asks a narrow question about one decision on one day, and it is discovered at exactly the moment it is most expensive to close.

The work is not exotic. Log the resolved vector, pin identity by content hash, capture the human layer, decide deliberately about point-in-time reconstruction, and put a read-only surface in front of it. Do that before the question arrives and it is a few months of engineering and a few hundred dollars a month. Do it after and it is the same engineering, done badly, under a deadline set by someone else.

Frequently asked questions