A model registry that earns its keep

The registry most teams already have

There is a table somewhere with a model name, a version number, a link to a file in object storage, and a column called stage that currently reads Production. It was created in an afternoon during the first deployment, appended to ever since, and nobody has asked what it is for. That table is not a registry. It is a deployment log with ambitions, and the difference shows up on one occasion: the day a model serving live traffic starts behaving differently and somebody has to decide quickly what to do.

The test we use is blunt. When a production model misbehaves, does anyone open the registry? In most of the systems we get called into, no. The on-call engineer opens the deployment tool to find which container is running, then the training repository to guess which commit produced the weights inside it, then chat to ask what the threshold was set to. Three sources, none authoritative, one of them asleep. The registry sat there and answered none of it.

That is not a tooling failure. MLflow, SageMaker, Vertex AI and Weights & Biases all ship a registry capable of answering every one of those questions. The failure is that the registry was set up to record which artifacts exist, while every question that arrives during an incident is about which artifact is running and what produced it. Only the second set is worth designing around.

Five questions, in the order they arrive

A registry earns its keep by answering five questions. They are worth stating precisely, because they determine the schema and almost everything else is optional.

What is serving right now. Not what was promoted, not what the pipeline last pushed. The exact artifact behind the endpoint taking traffic, identified by something stronger than a name.

What produced it. The training data, the code, the configuration, the environment. Enough to rebuild it, or to know precisely why you cannot.

What was last known good, and how fast can I put it back. Rollback is the only mitigation available in the first ten minutes, before anyone understands the cause.

What changed. A diff between the running version and the one before it: data, code, configuration, measured behavior.

Who decided this could go live, on what evidence. Not for blame. Because in six months somebody will ask whether a rule was followed, and the answer needs to live in a system rather than a memory.

Answer the first question and nothing else and you have a deployment log. Answer the first two and you have an inventory. Answer all five and it becomes the thing people open first during an incident, which is the only durable measure of whether it was worth building.

What a model version actually is

The most common design error is treating the artifact as the version. The weights file is the smallest part of what has to be captured, because weights alone do not determine the output. Everything between a request and a prediction is part of the model version, and anything versioned elsewhere leaves a record incomplete in a way only an incident reveals.

We call the full set the closure: everything that together determines what the system predicts for a given input. Write it down for one model and you will usually find two or three elements living in another repository, a cluster config map, or nobody's repository at all.

Closure element	What it pins	How it usually breaks
Model artifact	The trained weights, addressed by content hash	A mutable path like models/fraud/current/model.pkl that each deploy overwrites
Preprocessing and features	The transform applied at inference, versioned with the model	Training-time transforms in a notebook, serving-time transforms in the API repo, quietly diverging
Training data snapshot	One immutable state of the source data	A SQL query with a date range and no snapshot id, returning different rows next month
Training code and config	Commit SHA plus the fully resolved configuration, seeds included	A branch name, or a config assembled from environment variables nobody recorded
Runtime environment	Base image by digest plus the resolved dependency lock	An image tag like :latest and a requirements file with open version ranges
Decision policy	Thresholds, calibration, tie-breaks, fallback behavior	A threshold in a cluster config map, edited without producing a new version
Serving contract	Input schema, output schema, null and out-of-range handling	An implicit contract that matches whatever the first caller happened to send

The threshold row deserves its own paragraph. A threshold change is a model change: it alters what the system does for a nontrivial share of inputs, it is the easiest thing in the stack to change under pressure, and it is almost never versioned. We have seen more production surprises come from an unrecorded threshold edit than from a bad retrain. Put the decision policy inside the version record, make changing it produce a new version, and a whole class of unexplainable behavior disappears.

Immutable versions, one mutable pointer

The structural rule that makes everything else work: version records are append-only, and the only mutable object in the system is a named pointer.

A version, once written, never changes: not the artifact hash, not the lineage, not the metrics. If something about it was wrong, write a new version and mark the old one deprecated with a stated reason. Promotion is never an edit to a version. It is a pointer moving between immutable versions, recorded with a timestamp and an actor.

This is why the major tools converged on aliases. MLflow deprecated its fixed Staging and Production stage names in favor of arbitrary named aliases and tags, which was the right correction: a stage baked into the version row means promotion mutates the version, and two teams with different environments cannot both be right about what "Production" means. Vertex AI uses versions with aliases, including a default alias that endpoints resolve against. SageMaker keeps versioned model packages in a group with an approval status, the same idea reached from the governance side.

A schema that fits on one screen

We have built this several times and it collapses to five tables. Below is the shape, with the columns that carry weight and none of the ones that do not.

model
  id, name, owner_team, task_type, created_at, description

model_version                     -- APPEND ONLY
  id, model_id, version_int
  artifact_uri, artifact_sha256
  train_commit, train_config_json, dataset_snapshot_id
  image_digest, dependency_lock_sha256
  decision_policy_json            -- thresholds, calibration, fallback
  input_schema_json, output_schema_json
  created_at, created_by
  status                          -- draft | candidate | deprecated
  deprecated_reason

evaluation_run                    -- APPEND ONLY
  id, model_version_id
  eval_set_id, eval_set_sha256
  metrics_json, slice_metrics_json
  harness_commit, ran_at, ran_by

alias                             -- THE ONLY MUTABLE TABLE
  name, model_id, model_version_id, updated_at, updated_by
  -- ('prod', 4, 118), ('prod-previous', 4, 117), ('shadow', 4, 119)

alias_history                     -- APPEND ONLY
  alias_name, model_id, from_version_id, to_version_id
  moved_at, moved_by, approval_id, reason

approval
  id, model_version_id, gate_results_json
  approved_by, approved_at, waived_checks_json

Eleven columns do all the work: the artifact hash, the three lineage pointers, the decision policy, the two schemas, and the alias with its history. Everything else is convenience. If a product will not store those fields natively, store them as tags rather than standing up a second system, because the second system is the one that goes stale.

Two absences are deliberate. There is no free-text notes column, because people will use it instead of the structured fields and then nothing can be queried. And there is no is_production boolean on the version row, because a mutable flag on an immutable record is a contradiction somebody will eventually resolve the wrong way.

Lineage that still resolves in eighteen months

Lineage is the field that most often looks complete and is not. The test is not whether the column is populated. It is whether the value still resolves to the same thing eighteen months later, after the branch was deleted, the bucket lifecycle-expired, and the image tag was rebuilt on a newer distribution.

Pin data by snapshot, never by query. Iceberg snapshot ids, Delta Lake table versions, DVC content hashes, or a written manifest of file hashes all work. A SQL query with a date range does not, because the table underneath is still being written to and corrected, and the same query run twice is two different datasets.

Pin code by commit SHA, never by branch. And record the fully resolved configuration rather than the file that produced it, since the file becomes the real config only after environment substitution, defaults, and whatever the launcher injected.

Pin the environment by image digest, never by tag. A tag is a pointer somebody else controls; a sha256 digest is the image. Store the resolved dependency lock beside it, since a rebuild from an unpinned lock produces a different environment out of an identical Dockerfile.

The check is cheap to automate and almost nobody does it. A nightly job walks recent version rows and resolves every pointer: does the commit exist, does the snapshot exist, does the digest still pull, do the bytes at that URI still hash to the recorded value. Broken lineage found by a nightly job is an annoyance. Found during an incident, it is the incident.

Evaluation results belong in the registry

Metrics scattered across tracking runs, notebooks and pull request comments cannot be compared, because nothing records whether two numbers came from the same evaluation set through the same harness. That is not a documentation problem. It is the mechanism by which a worse model gets promoted.

Store, per evaluation run: the eval-set id and its content hash, the harness commit, the aggregate metrics, and the sliced metrics. The slices protect you. Aggregate accuracy moves slowly and hides the failure that matters, which is usually confined to one segment. Choose slices from how the business is organized rather than what is convenient to compute, and require them at promotion with a written tolerance each.

Hold the evaluation set separate from the training pipeline and version it on its own cadence. It outlives every model in the registry, and its stability is the only reason a metric recorded two years ago means anything today. When it does have to change, that is a new eval-set version and the comparison starts over.

The promotion gate

Promotion is the moment a registry either does work or becomes decoration. The gate is a list of conditions that must hold before an alias moves, checked by a program rather than a person, with waivers recorded rather than banned.

• The version record is complete — every closure field populated, every pointer resolving
• Evaluation ran against the current eval-set version, with slices, attached to the version
• No slice regressed past the tolerance the owning team wrote down in advance, not after seeing the numbers
• The serving contract is unchanged, or the change is recorded and every caller acknowledged it
• The artifact loads in the target runtime image and reproduces the recorded predictions on a fixed sample
• Latency and memory at expected request rate are inside the envelope, measured rather than assumed
• The previous alias target is retained and known good, so rollback has a destination that still exists
• A named human approved, or an automatic rule fired that a named human wrote and owns

The last item gets argued about. Automatic promotion is fine, and for models that retrain daily it is the only workable option. What is not fine is automatic promotion with no recorded owner of the rule. Waivers work the same way: a check can be waived, the waiver lands on the approval row with a reason and a name, and nobody argues about whether the gate is too strict at ten at night during a release.

Rollback is a registry operation

The reason to build any of this is one moment. Something is wrong in production, the cause is unknown, and the fastest safe action is to put back the version that was working. If that takes a rebuild, a retrain, or a conversation with the person who trained it, the registry did not earn its keep.

Four minutes from decision to traffic on the previous version is a reasonable target, reachable when three things are true. The previous artifact still exists and is pullable. The previous runtime image still exists at its digest. And the decision policy travels with the version, so rolling back the weights rolls back the thresholds too.

That third condition catches people. A team rolls back the model, the thresholds stay where the new version needed them, and the old weights now run under a policy nobody has measured them against. The system lands in a state that has never been evaluated, during an incident. Keep a prod-previous alias updated on every promotion and make rollback a single pointer move to it.

Then test it on a schedule. A rollback path not exercised in three months is a hypothesis. Pick a low-traffic window, move the alias, watch the metrics, move it back. Twenty minutes a quarter buys you a measured recovery time.

What belongs in the registry, and what does not

Half the disagreements about registries are scope disagreements with a neighboring system. The boundaries are stable once written down.

System	Owns	Does not own	The question it answers
Experiment tracker MLflow Tracking, W&B	Every training run including the failures, with parameters and metrics	What is serving	Which of the forty things we tried actually worked
Model registry	The small set of versions that are candidates or have served, with closure and lineage	The exploration history	What is running, what produced it, what do we roll back to
Artifact / object store	The bytes, addressed by content	The meaning of the bytes	Where is the file and is it intact
Feature store	Feature definitions and the guarantee that training and serving compute them identically	Model identity	What was this feature's value at that timestamp
Deployment system	Containers, replicas, routing, rollout strategy	Which model version those containers hold	Is the service healthy

One boundary is worth defending hard: the registry does not hold every training run. Promoting the experiment tracker to registry duty is the common shortcut, and it produces a registry with eleven thousand rows in which nobody can find the four that matter. A team shipping weekly should produce tens of versions a year. If the number is much larger, something is registering runs by accident.

Choosing the tool

The decision that matters is not which product. It is whether the registry sits on the deployment path. A registry written by the same code that deploys the model cannot drift. One that somebody updates after deploying is already wrong, and the gap widens every quarter until it is a table nobody reads.

MLflow Model Registry. Open source, self-hostable, aliases and tags, works with whatever you are serving on. You own the backing database and the artifact store, which is the point. It costs you a Postgres instance, object storage, and somebody who patches it.

SageMaker and Vertex AI model registries. Model packages with an approval status in one, versions with a default alias in the other, both wired tightly into their own pipelines and endpoints. Strong when training and serving both live there, awkward the moment half your inference runs somewhere else.

Weights & Biases Model Registry. Best fit when the team already lives in the experiment tracker, since promoting a run to a registered version is one step and the lineage comes along.

Postgres and an object store. The five tables above, a small client library, and the CI job that writes them. Unfashionable, and the right answer more often than it gets picked, especially when serving is split between a managed service and a Kubernetes cluster and neither vendor registry can see both.

What every one of them requires from you is identical: the closure fields, the promotion gate, the alias discipline. No product supplies those. They supply the table.

How this goes wrong

• A stage column edited by hand is the source of truth for what is running.
• Artifacts at a path like models/fraud/current/model.pkl, overwritten by every deploy.
• Thresholds and calibration in a cluster config map, changed without producing a new version.
• Preprocessing versioned with the serving repository instead of with the model.
• Training data recorded as a SQL query rather than a snapshot id.
• Base image pinned by tag, so the environment changes underneath a fixed version record.
• Metrics stored without the eval-set version, so two incomparable numbers look comparable.
• One registry per team, three different schemas, and no way to answer a question across them.
• No prod-previous alias, so rollback begins with an archaeology exercise.
• The registry updated by a human after deployment rather than by the deployment itself.

Introducing one to a system already in production

Nobody builds this on an empty repository. The usual situation is a dozen models already serving, most undocumented, at least two of which nobody owns. Reconstructing closure for all of them is not worth doing and reliably stalls the project in week two. Do this instead.

Two rules keep this bounded. Legacy versions are recorded as incomplete rather than reconstructed from guesses, because a blank field prompts a question and a wrong field ends the investigation. And the gate stays in report-only mode until the team has fixed what it finds, or the first blocking run lands mid-release, the gate is disabled that afternoon, and it never comes back on.

What it costs

For a team with a handful of models and one deployment path, this is two to three weeks of one engineer, most of it spent on the training-side plumbing that emits complete version records rather than on the tables. Multiple serving paths, a mixed cloud and Kubernetes footprint, or an audit requirement pushes it to five or six weeks. Ongoing cost is close to zero, because the registry is written by CI and read by the deployment, and neither forgets.

The return is concentrated in incidents and audits, which makes it hard to justify in advance and obvious in hindsight. The honest argument is not efficiency. It is that without it, the answer to "what is running and what produced it" depends on a particular person being reachable, which is not a dependency any team should accept for a system taking production traffic.

Bottom line

A model registry is not an inventory of files. It is the record of what is serving, what produced it, and what you can put back. Make version records immutable. Make one named alias the only mutable object in the system. Put the decision policy inside the version so thresholds roll back with the weights. Pin data to snapshots, code to commits, environments to digests, and check nightly that all three resolve. Then require the gate to pass before the pointer moves, rehearse the rollback quarterly, and measure the recovery time instead of estimating it.

Everything on that list is a week or two of work. What it replaces is a message to somebody who may not answer, at the moment answering matters most.

Frequently asked questions