Getting a model into production: the work that happens after the notebook

The gap between a working model and a working system

Most teams that cannot ship a model do not have a modeling problem. They have a model that works, a notebook that produces it, an AUC number somebody screenshotted for a briefing, and no path from that artifact to a decision anyone makes on a Tuesday morning. The model is finished and the system does not exist. Six months later the notebook no longer runs because a library moved, the analyst who wrote it changed jobs, and the source table got a new column. We have watched this pattern across defense, health, energy, transportation and public-sector data programs, and the shape of it barely changes between them.

The reason is structural. Training is a closed problem. You have a fixed dataset, a metric, and a laptop or a training cluster, and you can iterate against that metric all day without touching anything anyone depends on. Production is an open problem. You inherit somebody else's data contract, somebody else's uptime expectation, somebody else's audit requirement, and a user who will stop using the thing the first time it is confidently wrong at a bad moment. Nothing about the first problem prepares a team for the second.

So the useful question at the start of a project is not "can we build a model that predicts this." It is "what is the sequence of engineering steps between that model and a decision that changes." That sequence is knowable. It is mostly the same every time. Our engineers work it in a fixed order, because every step you skip early becomes a launch blocker later, and launch blockers get discovered at exactly the moment when the program has the least schedule left.

Packaging: the environment is part of the model

A model artifact by itself is not a deliverable. The deliverable is the artifact plus the exact code and environment that can load it and produce the same number twice. Teams underrate this because on a single machine the environment is invisible. It stops being invisible the moment a second machine tries to run it.

The practices that hold up: pin dependencies to exact versions with a lockfile rather than a loose requirements list; build a container image and reference it by content digest, not by a mutable tag like latest; store weights in a format with a stable loader (ONNX, TorchScript, safetensors) rather than a pickle of a class definition that only exists in one repository; and record the training run's inputs, code commit, random seeds and resulting metrics in a registry entry that the artifact points back to. If someone asks in eighteen months which data produced the model that made a specific call, that chain is the only honest answer.

Reproducibility is not only hygiene in federal work. It is increasingly a contract term. Software producers selling to the government attest to secure development practices drawn from NIST SP 800-218, the Secure Software Development Framework, under OMB M-22-18 and the deadlines set in M-23-16, using the CISA attestation form issued in March 2024. Build provenance and a software bill of materials in SPDX or CycloneDX are part of that conversation. A model pipeline that cannot say what went into a build is not going to produce a defensible attestation, and the government-side reviewer will find out.

Pick the serving pattern from the decision, not the framework

The most consequential architectural choice comes early and is often made by accident. Someone wraps the model in a REST endpoint because that is the tutorial, and the program spends the next year paying for always-on inference capacity to answer forty requests a day. The serving pattern should be derived from how the decision is made, how fresh the inputs need to be, and how much the wrong answer costs.

Serving pattern	Fits when	Typical budget	What it costs you
Batch scoring	The decision is reviewed on a cycle: nightly worklists, weekly prioritization, monthly risk ranking	Minutes to hours per run	Staleness. Scores age between runs and nobody sees an input that arrived at noon.
Scheduled micro-batch	Freshness matters but seconds do not: queue triage, dispatch prioritization, alerting	1–15 minutes	Orchestration complexity and a scheduler that becomes a single point of failure.
Synchronous request/response	A human or a system is waiting on the answer inside a transaction	50–500 ms at p99	Always-on capacity, tail-latency engineering, and a hard availability requirement.
Streaming / event-driven	Inputs arrive continuously and the decision is per-event: telemetry, sensor feeds, transactions	Sub-second to seconds	Ordering, replay and exactly-once semantics become your problem, not the model's.
Embedded / edge	The data cannot leave the platform, or the network cannot be assumed	Fixed by the hardware envelope	Compression, update logistics, and no ability to hotfix a deployed unit.
Human-in-the-loop queue	The model ranks and a person decides, with the decision recorded	Whatever the reviewer's rhythm is	Interface design and the discipline of capturing the human's answer as a label.

A large share of federal and commercial workloads that get built as real-time APIs are batch problems wearing a costume. If the output feeds a worklist a supervisor opens each morning, batch is the correct answer, it is an order of magnitude cheaper to run, it is far easier to accredit, and it fails in ways that are survivable. Reserve synchronous serving for the cases where a person or a system is genuinely blocked waiting.

Latency and cost budgets get written before the model

Write the numbers down at the start. Not "fast" but a p99 latency ceiling in milliseconds, measured end to end from the caller's perspective, including feature retrieval and network hops, because the model's own forward pass is often the smallest term. Not "affordable" but a dollar ceiling per thousand inferences, or per month, that the program office or the CFO will actually approve.

Those two numbers constrain everything downstream. They decide whether a large model is admissible at all, whether you can afford a feature lookup against a live store or must precompute, whether GPU capacity is reserved or on demand, whether responses are cached, and how much room is left for retries. Little's Law is the arithmetic that connects them: concurrency equals arrival rate times service time. At 200 requests per second and 250 ms of service time you need capacity for 50 requests in flight, always, plus headroom for the tail. Teams that skip this arithmetic discover it during load testing two weeks before a go-live date.

Cost discipline is mostly about what runs when nothing is happening. Reserved accelerator capacity bills the same whether it serves ten requests an hour or ten thousand. Scale-to-zero, request batching, quantized or distilled models, and moving the heavy work to an off-peak batch window are the levers that matter. A pilot that costs a few hundred dollars a month at ten users and quietly becomes six figures a year at agency scale is a common way for a successful pilot to die at the budget review.

Training and serving have to compute the same features

This is the failure that costs the most and looks the least like itself. A model trained on features computed one way, then served features computed another way, degrades silently. Offline metrics stay beautiful. Live performance is mediocre. Nobody can find the bug because there is no bug, only two implementations of the same idea that disagree at the margins.

The classic sources are worth naming because they recur. Aggregation windows that differ between the training SQL and the serving code. Timezone handling that is UTC in one path and local in the other. Null-fill and default values chosen differently. Category encodings fitted on the training set that silently map an unseen value to a bucket that means something else. Scaling parameters recomputed at serving time instead of being loaded from the training run. And the worst one, label leakage: a training feature computed from a table that is only populated after the outcome is known, which does not exist at the moment the prediction is actually needed.

The fix is architectural, not procedural. Compute a feature once, in one implementation, and have both paths call it. That is what a feature store buys you, whether it is Feast, a managed service, or a plain shared library plus a materialized table. Every feature carries an as-of timestamp so training joins reproduce exactly what was knowable at prediction time. Then prove it: score the same set of entities through both paths and assert the vectors are identical to a tolerance, as a test that runs in CI. When a real deployment goes sideways, this test is the first thing we reach for, and it is usually the answer.

Monitoring: data quality first, drift second

Model monitoring gets talked about as drift detection. In practice, most production incidents are data quality failures upstream, and they happen far more often than the world genuinely changing. A column starts arriving as a string. A vendor silently changes a code set. An ETL job half-fails and delivers 40 percent of yesterday's rows. The model does not crash. It confidently scores garbage.

So the first layer is input validation on every batch and every request: schema and type conformance, null rates against a baseline, cardinality and range checks, row counts, freshness of the source partitions. Tools such as Great Expectations or dbt tests handle the declarative part. The rule that matters is that a validation failure stops the pipeline and pages someone, rather than logging a warning nobody reads.

The second layer is distribution monitoring. Track the input distribution per feature against the training baseline using population stability index, Kolmogorov-Smirnov, or Jensen-Shannon divergence. The credit-risk convention on PSI is a useful starting point: below 0.10 is stable, 0.10 to 0.25 warrants a look, above 0.25 means investigate now. Watch the prediction distribution too, since a shift in output with stable inputs usually means something changed in the serving path. The third layer is outcome monitoring, which is the only one that measures whether the model is right. Ground truth arrives late, sometimes months late, so the design work is instrumenting the feedback loop at launch: capture the human's decision, capture what actually happened, and join them back to the prediction by ID.

For regulated buyers, this is not optional engineering taste. NIST's AI Risk Management Framework (AI 100-1) puts continuous measurement in the MEASURE and MANAGE functions. OMB Memorandum M-25-21, issued April 3, 2025, requires agencies to apply minimum risk management practices to high-impact AI, including pre-deployment testing, an AI impact assessment, ongoing monitoring for performance degradation, and human oversight. Financial-sector buyers apply the Federal Reserve and OCC model risk guidance SR 11-7, which has required ongoing monitoring, outcomes analysis and independent effective challenge since 2011. If a buyer sits in any of these regimes, monitoring is a deliverable with an owner, not a stretch goal.

Rollback and shadow deployment: earn the switch

Nobody who is accountable for an operational system will let a new model touch live decisions on the strength of an offline metric. The way to get authorization is to make the change reversible and to demonstrate behavior on real traffic before it counts.

Shadow deployment is the strongest tool available. Mirror live inputs to the new model, log its outputs, and act on none of them. Compare against the incumbent, whether that incumbent is another model, a rules engine, or a person. Two weeks of shadow traffic tells you things no held-out set will: what the real input distribution looks like, where the serving path disagrees with training, what the p99 latency is under actual load, and how often the two systems disagree on the cases that matter most. Replay is the cheaper cousin: score a historical window and reconcile predictions against recorded outcomes.

Rollback has to be a rehearsed procedure, not a theory. That means the previous model version stays loadable and reachable, the switch is a configuration change rather than a redeploy, the feature definitions the old version needs are still computed, and someone has actually executed the rollback in a lower environment with a stopwatch running. Under NIST SP 800-53 Rev. 5, configuration change control (CM-3) and system monitoring (SI-4) are exactly where an assessor will ask how this works. Having a real answer shortens an authorization conversation considerably.

The human decision the model actually feeds

Every deployed model terminates in a decision. Someone approves or denies, dispatches or holds, inspects or passes, escalates or closes. If the team cannot name that decision, the current holder of it, and what they do differently when the score changes, the model has no path to value and no path to adoption.

Naming the decision settles design questions that are otherwise argued forever. It sets the threshold, because the threshold is a business trade between the cost of a false positive and the cost of a false negative in that specific workflow, and it is almost never 0.5. It determines what has to accompany the score: a claims examiner needs the contributing evidence, a maintenance planner needs the estimated time window, a contracting officer needs the citation. It sets the volume the model may produce, since a queue that generates more work than the team can process will be ignored inside a month. And it defines the override path, which is both a governance requirement and the cheapest source of labeled training data any program will ever get.

This is also where accessibility and record-keeping obligations land. If the output surface is a federal system, Section 508 applies to the interface. If the decision affects a person's rights or benefits, the reasoning has to be reconstructable later by someone who was not there. Designing the audit record at the same time as the model is far cheaper than retrofitting it after the first appeal.

The pre-launch checklist

• The decision, the decision owner, and the threshold are written down and agreed
• Serving pattern chosen from the decision cadence, with a p99 latency ceiling and a monthly cost ceiling
• Environment pinned by lockfile and container digest; model artifact in a registry linked to its training run
• Feature parity test between training and serving paths running in CI
• Input validation that halts the pipeline and pages a human on failure
• Drift and prediction-distribution monitors with alert thresholds, plus an outcome feedback loop instrumented at launch
• Shadow or replay evidence on real inputs, compared against the incumbent
• Rollback rehearsed end to end, with the prior version still loadable and a kill switch behind a config flag

Why most of the effort is not modeling

Add the pieces up. The data contracts and quality checks, the feature layer with a single implementation, the packaging and provenance chain, the serving path, the budgets, the monitoring stack, the feedback loop, the rollout mechanics, the interface a human touches, and the documentation an assessor reads. Model selection and tuning is one item in a list of ten, and it is usually the item with the best tooling, the clearest literature, and the fastest iteration loop. That is precisely why it absorbs attention out of proportion to its share of the work.

The teams that ship treat the model as a component inside a system and staff accordingly: data engineering, platform, security and the domain expert who owns the decision, working the sequence together from week one. The teams that stall keep improving a number in a notebook while the ten other items stay unowned. Precision Federal builds AI and data systems for federal, state and commercial customers, and this sequence is what we bring to a program that has a promising model and no route to production. Our engineers work the serving path, the feature layer, the monitoring and the rollback mechanics, alongside the licensed engineers and domain specialists on our bench who know what the decision at the end of the pipeline actually costs when it goes wrong.

Frequently asked questions