The problem is almost never the modeling
A pattern shows up in company after company. The data science group is genuinely good. They read papers, they win internal bake-offs, they produce a churn model or a defect classifier or a document extractor that beats the current process by a wide margin offline. Leadership funds them for another year. At the end of that year, the honest answer to "what is running" is still a notebook, a slide, and a shared drive full of pickled artifacts. The instinct is to conclude the team is not senior enough, or that the tooling was wrong. Both conclusions are usually wrong, and both send a real budget in the wrong direction.
What has happened is that a group hired to answer a research question has been asked to deliver an operations product, and nobody drew the line where the job changed. Finding signal in data and running a service other systems depend on are different skills, with different failure modes and different definitions of done. A team can be excellent at the first and never once have been accountable for the second. That is a scope problem, not a talent problem, and it is fixable in a quarter if it is named correctly.
Four blockers wear the same costume
"We cannot get it into production" sounds like one condition. It is four, with different owners, costs and cures. Before spending anything, find out which one is in the room. Most stalled programs have two at once.
The runtime data is not the training data. The model was built on an extract: a table someone pulled once, cleaned by hand, joined to a spreadsheet, with a label column that came from a process nobody has to repeat. At runtime none of that exists. Fields arrive late, arrive dirty, arrive in a different unit, or do not arrive. This is the most common blocker, and it is an access and engineering problem rather than a modeling one.
The model has no consumer. Nobody has agreed which system calls it, at what moment in a workflow, and what a person or downstream process does with the answer. A prediction with no decision attached is a report, and reports do not need to be deployed. This is a product management failure, cheap to fix, but only once someone with authority over the calling system is in the room.
Nobody owns it after launch. The data science team does not carry a pager and should not be asked to. The platform team does not want a model whose behavior it cannot explain. Without an owner, launch is the moment the organization acquires a liability, so a sensible engineering leader quietly refuses the handoff. The project is not killed; it is slow-walked.
There is no path. No build, no test, no packaging, no environment, no deployment, no rollback, no monitoring. The team is asked to cross a bridge nobody built, and it never gets built because bridge-building is invisible on a roadmap made of use cases.
The ratio that should reset your budget conversation
In 2015 a group of Google engineers published Hidden Technical Debt in Machine Learning Systems at NIPS. The most quoted thing in it is a figure caption: "Only a small fraction of real-world ML systems is composed of the ML code, as shown by the small black box in the middle. The required surrounding infrastructure is vast and complex." Later the authors put a number on it, writing that "a mature system might end up being (at most) 5% machine learning code and (at least) 95% glue code."
Read that as a staffing statement rather than a technical one. If the whole team is trained and measured on the modeling work, the organization has hired for a small share of the system and left the rest unassigned. The glue is not menial. It is feature computation that must produce identical values in two codepaths, schema checks, backfills, a serving surface, versioning, access control, and the ability to put yesterday's model back at three in the afternoon without a meeting.
The same paper names what fills the vacuum: pipeline jungles, data preparation grown into "a jungle of scrapes, joins, and sampling steps, often with intermediate files output." Every stalled program we have looked at has one. It usually lives on one engineer's laptop, and it is why the team cannot say how a number was produced.
What actually changes between a notebook and a service
A notebook is a good instrument. It is an argument written in code, read top to bottom by a person who wants to be convinced. A service is a different object with a different contract, and the gap is not about code quality.
| Dimension | Notebook (research artifact) | Service (production artifact) |
|---|---|---|
| Input | An extract the author pulled once, already joined and cleaned | A live feed with late records, nulls, schema drift, and an upstream owner who agreed to nothing |
| Definition of correct | A metric on a held-out split, judged by the author | A written threshold on live traffic, judged by whoever gets paged |
| Failure | A traceback the author reads and fixes | A silent wrong answer a downstream system acts on for a week |
| Change | Edit a cell, re-run, keep going | Version, review, test, canary a small share of traffic, hold a rollback ready |
| Reproducibility | Implicit in one machine's environment | Explicit: pinned dependencies, seeded training, a registered versioned artifact |
| Ownership | The author, until they change teams | A named on-call rotation with a runbook, indefinitely |
| Evidence it works | The chart in cell 42 | Monitoring that proves it works today, not that it worked in March |
There is measured evidence for how fragile the first column is, with a caveat. Sheeba Samuel and Daniel Mietchen re-ran the Jupyter notebooks attached to biomedical publications indexed in PubMed Central: 27,271 notebooks from 2,660 GitHub repositories linked to 3,467 articles. Of those, 22,578 were Python, and 15,817 declared dependencies in a standard requirements file and could be attempted automatically. For 10,388 every declared dependency installed cleanly. Of those, 1,203 ran to completion without an error, and 879 produced results identical to the ones the notebook itself reported.
Those are research notebooks published alongside papers, not corporate notebooks behind a firewall, and the study measures automated re-execution rather than whether a person could get them working. It is not a claim about your team. What it establishes is that a notebook producing a correct result on one machine on one day is weak evidence the same computation runs again for anyone else, which is the property production requires.
An instrument, not an opinion: the ML Test Score
The most useful thing a VP can do here is stop arguing about readiness and measure it. The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction, by Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib and D. Sculley, was presented at IEEE Big Data in 2017. It sets out 28 tests, drawn from running production machine learning at scale, in four groups of seven.
Features and data asks whether feature expectations are captured in a schema, whether every feature earns its cost, whether the pipeline has privacy controls, and whether feature code is tested. Model development asks whether model specs are reviewed, whether offline metrics correlate with the online outcome anyone cares about, whether staleness impact is known, whether a simpler model would do, and whether quality holds on the slices that matter rather than only in aggregate. ML infrastructure asks for reproducible training, unit-tested model specs, an integration test across the full pipeline, validation before serving, debuggability, canarying and rollback. Monitoring asks for dependency-change notification, data invariants, no training and serving skew, staleness limits, numerical stability, and alerts on compute and prediction-quality regressions.
The scoring rule is worth copying exactly. Half a point for running a test manually with the result documented and distributed. A full point when a system runs it automatically on a repeated basis. Sum each of the four sections separately, and the final score is the minimum of the four totals. The authors chose the minimum deliberately, because all four areas have to be present for a system to be dependable.
What the score means, using the interpretation the authors calibrated against real internal systems:
| Score | What it means | What a leader should do next |
|---|---|---|
| 0 | More of a research project than a productionized system | Stop calling it a delivery program. Fund one use case and build the path |
| Above 0, up to 1 | Not untested, but assume serious holes in reliability | Find the holes before a customer does. Start with data and monitoring |
| Above 1, up to 2 | A first pass at productionization; more investment needed | Name an owner for the running system and give them budget, not just responsibility |
| Above 2, up to 3 | Reasonably tested; more of it could be automated | Convert documented manual checks into scheduled ones. Cheapest points on the board |
| Above 3, up to 5 | Strong automated testing and monitoring, fit for mission-critical systems | Hold the line. Add scope, not exceptions |
| Above 5 | Exceptional levels of automated testing and monitoring | Write down how you did it. Other teams should copy it |
Two findings make the exercise less embarrassing to run. Across a survey of several dozen teams at Google, none of the 28 tests was implemented by more than 80 percent of teams. And interviewing the maintainers of 36 real systems, the authors found the test for training and serving skew was "perhaps the most important and least implemented test." A low first score is normal. Having a first score is the point.
The failure modes have names, and naming them changes the meeting
Half the value of those two papers is vocabulary. Once a failure has a name, it stops being a personality dispute between modeling and platform.
Training and serving skew. A feature is computed one way in training and another at inference, usually because the training version was written for flexibility and the serving version for latency. The values diverge, and the production model is quietly being asked questions in a dialect it never learned. Detecting it takes logging a sample of live serving traffic and comparing feature values for the same example.
Staleness. Nobody has measured what a week, a month or a quarter of age does to prediction quality, so nobody knows how urgent retraining is. Build the age-versus-quality curve once, then monitor against it.
Undeclared consumers. A model writes predictions to a table or a log, and other systems start reading them without telling anyone. The 2015 paper calls this visibility debt, and it is what makes a harmless-looking model change break something three teams away. Access controls and stated service levels are the fix.
Correction cascades. Rather than retrain for a slightly different problem, someone trains a correction on top of the model, then another on top of that. The paper calls the result an improvement deadlock: making any one component more accurate makes the whole worse.
Configuration debt, and CACE. In a mature system the lines of configuration can exceed the lines of code, and configuration is usually treated as beneath review or testing. Sitting behind it is the authors' shorthand CACE, Changing Anything Changes Everything, which applies to input signals, hyperparameters, sampling and thresholds alike. It is why "just swap the model" is never just swapping the model.
This is an organization design question before it is a tooling question
The tempting response to all of this is to buy a platform. Platforms help, and we will get to when. But the 2015 paper puts the root cause elsewhere: glue code and pipeline jungles, it says, "are symptomatic of integration issues that may have a root cause in overly separated 'research' and 'engineering' roles." Its recommendation is to embed engineers and researchers on the same team, often as the same people.
The industry has converged on that answer from several directions. Continuous Delivery for Machine Learning, written by Danilo Sato, Arif Wider and Christoph Windheuser and published on Martin Fowler's site in 2019, defines the practice as a cross-functional team producing machine learning applications "based on code, data, and models in small and safe increments that can be reproduced and reliably released at any time." Its central observation is that an ML application changes along three axes rather than one: the code, the model, and the data. An organization with release discipline for only the first axis will keep being surprised by the other two.
The 2025 DORA research reaches a similar conclusion about AI adoption generally, describing AI's primary role as "an amplifier, magnifying an organization's existing strengths and weaknesses." A company with weak delivery discipline does not acquire delivery discipline by adding models to it. It gets a faster version of what it had.
Three documents worth putting in front of your team this week
Hidden Technical Debt in Machine Learning Systems, Sculley et al., NIPS 2015, for the vocabulary. The ML Test Score, Breck, Cai, Nielsen, Salib and Sculley, IEEE Big Data 2017, for the 28-test rubric and the minimum-of-four rule. Continuous Delivery for Machine Learning, Sato, Wider and Windheuser, 2019, for the practice model. All three are public and short, and a team that has read them argues about different things.
What has to exist before anything is called production
This is the shortest useful gate. If an item here is missing, the system is a demo with a URL, and calling it production moves risk onto people who never agreed to it.
- A named calling system and a named decision. Which service invokes the model, at what point, and what changes because of the answer.
- A written definition of good on live data. Not the offline metric. The threshold that, if crossed, means someone acts.
- Reproducible training. Pinned dependencies, seeded randomness, and a recorded pointer to the code and data that produced the artifact.
- A registered, versioned model artifact. A registry gives every model a name, sequential versions, tags, and a mutable alias such as a champion pointer, so "which model produced this decision" has an answer. The MLflow Model Registry, maintained under the Linux Foundation, is the common open-source implementation.
- An integration test across the whole pipeline. Data assembly, feature generation, training, validation and deployment, exercised end to end on a schedule. The ML Test Score authors found this among the least-adopted tests, because training is so often an ad hoc set of scripts.
- Feature parity between training and serving, monitored, not asserted.
- Canary and rollback, both practiced. New models take a small share of traffic first, and the team has reverted to a previous version once outside an emergency.
- Monitoring with an owner. Data invariants, model age, numerical stability, compute performance and prediction quality on served data, each routed to someone expected to respond.
- A runbook and an on-call rotation the modeling team is not silently carrying.
A ninety-day path to one shipped service
Trying to productionize a portfolio is how programs die. One use case, all the way through, builds the path every later use case reuses. The durations below suit a mid-size company with a platform team.
One model from notebook to owned service
Note what is absent. There is no model improvement work. The first ninety days ship the model the team already has, because the path is the deliverable. Model quality is the second lap, and it is far cheaper once the first lap exists.
Four ways to close the gap, and the honest cost of each
Retrain the team you have. Send the data scientists into the delivery discipline: code review, testing, on-call, release process. Cheapest on paper, and it works with an early-career team. The real cost is a year, and the risk is that people hired for research leave because the job changed.
Hire platform and ML engineers. The right long-term answer for a company that will run many models, and the slowest to start. The hire is competitive, the first one needs someone to review their design, and a live service is two quarters out. One person hired into an organization with no path produces one heroic person and no path.
Buy a managed platform. Managed training and serving removes real work, and integration testing in particular gets easier when training runs on a framework rather than a pile of scripts. What a platform cannot buy is the definition of good on live data, the agreement with the upstream data owner, or the on-call rotation. Buying tooling to solve an ownership problem is how this budget gets spent twice.
Bring in an engineering partner to build the path with you. The case is narrow: the work is bounded, mostly one-time, and meant to leave a pipeline your team runs rather than a service someone else operates. That is the work we take on. Insist on the handoff artifact in the contract, and judge the partner by whether the score rises on tests your own people can maintain afterwards.
These are not exclusive. What works most often is a partner building the first path against a scored rubric while a platform hire is recruited into the seat that will own it.
What a regulated or federal buyer adds on top
If the model touches a regulated process, or the company sells into government, the readiness bar gains a documentation dimension teams consistently underestimate. The NIST AI Risk Management Framework 1.0, released January 26, 2023, organizes that work into four functions: govern, map, measure and manage. NIST published a companion generative AI profile, NIST AI 600-1, on July 26, 2024, and a concept note in April 2026 for a further profile on trustworthy AI in critical infrastructure. The framework is voluntary and under revision, so confirm the live text before committing against a section number.
The practical translation is that a buyer in this class asks questions only a versioned, monitored system can answer. Which model version produced this decision. What data trained it. Who approved it. What the measured performance was on the population it affected, not just in aggregate. A team with a registry, slice-level evaluation and live monitoring answers all four in an afternoon. A notebook answers none, which is why readiness work and compliance work are the same work in different vocabulary.
Common objections you will hear from the team
"The infrastructure work is not our job."
Correct, and that is the finding rather than the counterargument. If the modeling team should not build the path, name who does and fund them. The failure is leaving the 95 percent unassigned and then treating the delay as a data science performance issue.
"We need a better model before it is worth deploying."
The rubric tests whether a simpler model would do as well, precisely because teams invest in sophistication before dependability. Ship the model you have, then improve it. An improvement you cannot deploy has no value, and a path makes every later one cheap.
"Our tooling is the problem. We need a platform."
Sometimes true, and cheap to test. Score the four sections first. If the low section is data or monitoring, a platform will not move it much, because those scores are set by agreements with data owners and by someone agreeing to be paged. If the low section is infrastructure, a platform genuinely helps.
"We already monitor it. There is a dashboard."
A dashboard nobody is required to look at is documentation, not monitoring. The difference is whether a regression finds a person or waits for one.
Bottom line
A data science team that cannot ship is usually a team that was never given the second half of the job, and the second half is most of the system. Find out which of the four blockers you have. Score the 28 tests and publish the number, low sections included. Pick one use case with a real calling system and build its whole path in a quarter. Then decide whether the seat that owns it is a hire, a partner, a platform, or all three. Measurement is the step most organizations skip, and it is what turns a stalled program into a plan with a date.
Frequently asked questions
Score the system against the four sections of the ML Test Score rubric and see where the minimum sits. If model development scores well and infrastructure or monitoring sit near zero, the team is doing the work it was hired for and the organization left the rest unassigned. That is an ownership problem. A team scoring low on model development too is a different conversation.
The same feature is computed differently in the training pipeline and in the live serving path, so the model receives values it was never trained on. It rarely causes a crash; it causes quietly degraded predictions. The ML Test Score authors, after interviewing the maintainers of 36 production systems, called it perhaps the most important and least implemented test in the rubric. Detecting it means logging a sample of real serving traffic and comparing feature values for the same example.
Partly, and only for one of the four blockers. A platform makes reproducible training, artifact versioning, deployment and integration testing much easier. It does not obtain runtime data access, decide which system calls the model, or volunteer anyone for on-call. Score the four sections before purchasing, so the spend lands on the section that is actually lowest.
For one use case with an existing calling system and platform team, a quarter is reasonable: two weeks to measure and choose, a month to settle runtime data access and rebuild feature computation once for both paths, a month for reproducible training and validation gates, and the rest for canary deployment, rollback rehearsal and live monitoring. Programs that productionize several models at once usually deliver none.
A named artifact list your team can run without the partner: pipeline code, tests, deployment configuration, runbook and monitoring. A measurable target, such as the readiness score at the end and which tests are automated. And a handoff period where your engineers operate the system while the partner is available. If the deliverable is a service only the partner can run, the ownership problem has been rented, not solved.