From notebook to production without a rewrite

The handoff that turns six weeks into six months

The pattern is the same in every company we walk into. Someone builds a model in a notebook. It works. The numbers are good, the plots are convincing, and a decision gets made to put it in front of users. The notebook goes to a platform team, and four months later a service exists that does approximately the same thing, written from scratch, producing a number nobody can reconcile with the original. Everyone calls this normal. It is not normal. It is a bill, and most of it comes due for reasons that were set in the first week of the project, before anyone had thought about deployment at all.

The rewrite is not a language problem. The same Python runs in both places. It is not a skill problem either, and framing it that way is how organizations end up with two teams that resent each other. The rewrite happens because a notebook is not a specification. It is a transcript of a conversation between one person and a dataset, and the parts of that conversation that mattered most were never written down: which cells got re-run, which file the data actually came from, which library versions were installed that morning, and which of the twelve variants printed the number that ended up in the deck.

Sculley and colleagues made the general point in "Hidden Technical Debt in Machine Learning Systems" at NeurIPS 2015, and the diagram from that paper has been on a hundred conference slides since: the box labeled "ML code" is a small square inside a much larger picture of configuration, data collection, serving infrastructure, and monitoring. Everything in that larger picture is what the port has to build. The notebook contains the small square, and it contains it in a form that resists being extracted.

What a notebook is, and what a production process needs

A notebook is a long-lived process holding mutable global state, driven interactively, with a document attached that records some of what happened. Cells run in whatever order the author clicked. Variables persist after the cell that created them is edited or deleted. Outputs are cached in the file and can outlive the code that produced them. The environment is whatever happened to be installed on one machine. None of that is a defect. Those properties are exactly what makes a notebook good for exploration, which is a genuinely different activity from running a system.

A production process needs the opposite set of properties. It starts from nothing. It reads its inputs from a named, versioned location. It runs to completion or fails loudly with a stack trace someone can act on. It produces the same output from the same input, or it records precisely why it does not. And it leaves enough evidence behind that a person who was not there can reconstruct what it did six months later.

Those two lists are not opposites so much as two ends of a short distance. The distance is about a dozen working habits. Adopt them while the work is still exploratory and they cost roughly a day of effort spread across the first two weeks. Retrofit them onto a mature notebook and they cost one to three weeks. Skip them entirely and you pay for a rewrite, which is the most expensive option and also the one that loses the original result.

The four disagreements

Every expensive thing in that chart traces back to one of four places where a notebook and a running system hold incompatible assumptions. Naming them is useful, because each has a fix that takes hours rather than weeks when applied early.

Where the code lives. In a notebook, the code and the record of having run it are the same file. There is no import path, so nothing can be tested in isolation, reused by a second entry point, or reviewed as a diff. A function that exists only inside a cell is a function that has to be moved before it can be used, and moving it is where behavior quietly changes.

Where the state lives. In a notebook, state lives in the kernel and accumulates across an afternoon of clicking. In production, state lives in explicit arguments and explicit storage. The gap between those is where "it worked yesterday" comes from: a variable defined in a cell that was later edited, a dataframe filtered in place three cells above, a config value typed once and never written down.

How data is addressed. A path like ~/Downloads/customers_final_v3_REAL.csv is not data access. It is a bookmark on one laptop pointing at a file whose provenance is a memory. Production addresses data by source, name, and version, and it can answer the question "which exact rows produced this model" without asking anyone.

What "done" means. In a notebook, done means the last cell printed a number the author believed. In production, done is defined by a test that another person can run, that fails when the result gets worse, and that runs without a human deciding whether the output looks right.

Rule one: the notebook imports, it does not define

The single highest-return habit is also the cheapest. On day one of a project, before any modeling, create a package. A pyproject.toml, a src/ directory with your package inside it, and one command to install it into the environment in editable mode. In the notebook, turn on autoreload so edits to the module take effect without restarting the kernel. That is the entire setup, and it takes under an hour including the first commit.

From then on the rule is mechanical: any function that survives its second use moves out of a cell and into a module. The notebook keeps four things, and only four. It loads data, calls functions, draws plots, and carries the narrative that explains what the author was thinking. Everything else lives somewhere it can be imported, tested, reviewed, and called by a scheduled job.

There is a one-question test for whether this is working. If you deleted every notebook in the repository right now, what would stop working? If the answer is "the model", the code is in the wrong place and the rewrite is already scheduled, whether or not anyone has put a date on it. If the answer is "we would lose some plots and some commentary", you are in good shape.

This habit also fixes the review problem. Nobody reviews a notebook properly, because the diff of a .ipynb file is JSON with base64 image blobs in it. Code that lives in modules gets read by a second engineer, which catches the class of error no test catches: a transformation that is defensible but not what the business meant.

Rule two: no cell may depend on another cell having been run

Restart the kernel and run every cell top to bottom. That is the acceptance test, it is built into every notebook interface, and running it once a day is the difference between a port and a rewrite. When it fails, the failure is the future rewrite arriving early, in a form that costs twenty minutes to fix instead of three weeks.

Two tools make this durable. nbstripout, installed as a git filter, removes outputs from notebooks before they are committed. That makes diffs readable, and it closes a data-exposure path that catches a lot of teams by surprise: a dataframe preview in a cell output is customer data serialized into a JSON file in your version control, replicated to every clone of the repository. Second, jupytext pairs each notebook with a plain .py file in percent format, so code review happens against Python rather than against JSON, and merge conflicts become tractable.

When a notebook genuinely needs to run on a schedule, papermill executes it with injected parameters and writes an output copy, which is how Netflix described running notebooks as production jobs at scale. That is a legitimate pattern and worth knowing about. It works precisely because the notebook it executes has already been made restart-clean. Papermill does not rescue a notebook with hidden state, it just automates running it.

Rule three: data has an address, not a path

Replace every literal path with two things: a configuration object read from the environment, and one loader function that takes a dataset name and a version. The loader resolves where the data actually lives, whether that is object storage, a warehouse table, or a local cache during development. Nothing else in the codebase knows about filesystems.

Then add the snapshot rule. Do not train against a live query. Run the extract once, write the result to object storage under a versioned key, and read the snapshot from then on. Two things improve immediately. Reproducibility becomes real, because the exact rows that produced a model still exist. And your exploratory loop stops issuing heavy queries against a database that is also serving customers, which is a conversation with the platform team that nobody enjoys having twice.

With each snapshot, record four fields: the source, the query or extraction logic, the row count, and a content hash. The hash is the cheap one everybody skips and the one that settles arguments. When a model result cannot be reproduced, the first question is whether the data changed, and a hash answers it in a second rather than in a two-day investigation.

Rule four: time and randomness are inputs, not ambient facts

Set seeds and record them alongside the result. Python's hash seed, the NumPy generator, and whichever framework you are training with all need explicit values. This does not buy bitwise determinism, particularly on GPUs where reduction order varies, and chasing bitwise determinism is usually a poor use of a week. What it buys is a bounded, measurable amount of run-to-run variation.

Measure that variation before you trust any comparison. Run the pipeline three times with different seeds and record the spread of the metric. If the spread between runs is larger than the improvement you are claiming, you have not improved anything yet, and shipping that change moves a number around without moving the outcome. We have seen a quarter of modeling work evaporate under this one check, which is unpleasant on the day and much cheaper than finding out in production.

Time is the other ambient input, and it is more dangerous because it fails silently. A call to the current clock inside a feature function means the feature computed during training is not the feature computed at serving, and worse, it usually means the training feature saw information that did not exist at the moment being predicted. Every feature gets computed as of an explicit timestamp passed in by the caller. Point-in-time correctness sounds like a data warehousing concern until the first time a model scores beautifully offline and lands at chance in production.

The environment is part of the model

A trained model is a function of three inputs: the code, the data, and the library versions. Teams pin the first two and treat the third as background. Then a rebuild six months later produces a different number and a week disappears into finding out why.

Pin the environment with a real lockfile, meaning a resolved, hash-bearing file produced by a resolver. A requirements.txt with loose version specifiers is a wish list, not a lock. Build the container from a base image pinned by digest rather than by tag, because tags move underneath you and the change arrives on a random Tuesday with no diff to review. And set thread counts explicitly. Threaded linear algebra changes the order of floating-point reductions, so the same model on the same input can differ in the last few digits between a laptop and a server. The differences are tiny and they still generate a two-day argument about whether the deployment is broken.

All of this belongs in the repository from the first week, because the cost of adding it is nearly zero at the start and it grows with every dependency somebody installs in a cell. An install command inside a notebook is the clearest signal that an environment has stopped being reproducible, and it is worth treating as a small alarm rather than a convenience.

Notebook habit	Why it breaks in production	The change that closes it
Functions defined in cells	No import path, so nothing can be tested, reused, or reviewed as a diff	Move to a package, install editable, import into the notebook
Out-of-order execution	The result depends on click history that nobody recorded	Restart-and-run-all as a daily gate, plus stripped outputs in version control
Hard-coded local paths	Runs on exactly one machine, and provenance is a memory	Config from the environment, one versioned loader, snapshot before training
Installing packages in a cell	The environment cannot be rebuilt, so results cannot be reproduced	A resolved lockfile and a base image pinned by digest
Pickle as the delivery format	Loads only under the exact library set that wrote it, and it executes code	Versioned artifact with library metadata, or a stable export format
Transformations written twice	Training and serving drift apart with no error to catch it	One transformation module imported by both paths
Accuracy printed in a cell	No regression detection, so quality is whoever looked most recently	A frozen evaluation set and a test that fails when the metric drops

The skew that costs the most: two implementations of one transformation

Here is the failure that produces the worst production incidents, because it is invisible to every offline test. During training, features are computed over a whole dataframe with pandas. At serving time, one record arrives and the same feature has to be computed again, in different code, usually written by a different person, often in a different language. The two implementations agree for a while. Then one of them gets a bug fix and the other does not.

Certain features break more than others. Rolling windows and lag features, because a single record has no history attached. Normalization, because the mean and standard deviation used at training time are not present at serving time unless somebody shipped them. Category encodings, because an unseen category has to map to something and the two paths pick different somethings. Imputation, because the fill value was computed from the training distribution. Anything relative to the dataset rather than to the record is a candidate.

The fix has two halves and both are non-negotiable. First, exactly one implementation of each transformation, in a module imported by the training pipeline and by the serving path. If it cannot be computed from a single record plus a lookup, then it is not a serving feature, it is a batch feature, and it needs to be precomputed and stored where the serving path can read it. Second, treat fitted statistics as model parameters. Means, variances, category vocabularies, and quantile boundaries are learned on training data and shipped inside the artifact. Recomputing them at serving time from whatever traffic happens to be present is a silent accuracy loss that no test will catch, and it degrades gradually enough that nobody links it to a deployment.

Batch first, and often batch only

Most requests for real-time inference are requests for a shorter schedule. The useful question is what decision the prediction drives and how quickly a human or a system can act on it. If an analyst reviews a queue that refreshes hourly, an hourly batch job is the correct engineering. It deletes the entire serving tier, along with its autoscaling, its latency budget, its warm-up behavior, and its 3 a.m. pages. Teams skip this question because a service feels more finished than a scheduled job, which is an aesthetic preference with a real operating cost attached.

When online serving is genuinely required, budget the latency before writing the code, and measure where it goes rather than guessing. On tabular models the pattern is consistent and it surprises people: the model forward pass is rarely the problem. Fetching features and joining them is. Optimizing the model when the feature store is the bottleneck is a week spent making the wrong number smaller.

Two more things about latency. The median is not the number your users experience, so set the budget on the 99th percentile and hold it there under realistic concurrency. And if the caller can send records in groups, offer a batch endpoint, because per-request overhead amortizes and throughput improves without touching the model at all.

The evaluation set travels with the code

An accuracy figure printed in a cell is a fact about one afternoon. What production needs is a frozen evaluation set, a scoring script, and a test that fails when the metric drops below an agreed threshold. That test is the contract. When it goes red, either the code got worse or the world moved, and both of those are worth waking up for.

Keep two tiers. A small fixture set, maybe fifty to two hundred rows, checked into the repository so continuous integration can run the entire pipeline end to end in under a minute on every commit. That test does not measure model quality. It catches the schema change, the renamed column, the exception on an empty group, and the transformation that silently returns all nulls, which together account for most of what actually breaks. Then the full evaluation runs on a schedule against the real set, with the metric history stored so regressions are visible as a trend rather than as a surprise.

Labeled examples with agreed correct answers are the most durable asset produced by a machine learning project. Models get retrained, frameworks fall out of favor, and vendors get acquired. The evaluation set outlives all of it, and it is the only thing that lets you swap a component and know within a day whether the swap made things worse.

What to log, and the week of shadow traffic

A deployed model that logs only its output is a system you cannot debug. Log the request identifier, the model version, the feature or schema version, a hash of the input, the prediction, the latency, and a flag for every feature that was missing and defaulted.

That last field earns its place more than the rest combined. Missing-feature defaulting is the most common silent failure in production machine learning. An upstream table stops updating, the join returns nothing, the serving code fills zeros because someone wrote a sensible-looking default eighteen months ago, and the model produces confident predictions from a feature vector that is mostly zeros. Nothing errors. Latency is fine. The dashboard is green. Quality degrades for weeks until a business metric moves far enough that somebody investigates. A serving path that quietly substitutes a default for a missing feature will produce confident, wrong predictions and never raise an error, so make the substitution visible in the log line and alert on its rate.

Before cutover, run the new system in shadow for a week. Real traffic goes to both paths, only the incumbent's output is acted on, and the two prediction distributions get compared. This is inexpensive and it catches the training-serving skew that no offline test finds, because it is the first time the new code sees the actual shape of production input. When the distributions disagree, you learn it during a week you planned for rather than during an incident you did not.

Five habits that guarantee a rewrite

• Exploration and the final training run in the same notebook. The file that produced the shipping model should contain nothing but the path that produced it. Fork the exploration off and keep it, but keep it separate.
• Notebooks committed with their outputs. Unreadable diffs, bloated repositories, and customer records serialized into version control where nobody thinks to look for them.
• Treating "it ran on my machine last Tuesday" as reproducibility. If it does not rebuild in a clean container from a lockfile, the result is an anecdote.
• Evaluation that exists only as a printed number. Quality then depends on who looked most recently, which means quality is unmanaged.
• Deferring the package structure until after the model works. This is the expensive one. By the time the model works, the code has grown roots into the kernel, and pulling it out changes behavior in ways nobody can characterize.

Before you call a notebook production-ready

• Restart-and-run-all passes from a clean kernel on a fixed data snapshot
• No function used more than once is defined inside a cell
• The environment rebuilds from a lockfile, in a container pinned by digest
• Every input is addressed by dataset name and version, with a content hash recorded
• Seeds are set and recorded, and run-to-run spread has been measured at least once
• Every feature is computed as of an explicit timestamp passed in by the caller
• Training and serving import the same transformation module
• Fitted statistics ship inside the model artifact rather than being recomputed
• A frozen evaluation set and a regression test run in continuous integration
• The artifact loads in a clean container from the lockfile, and somebody has proved it

A two-week path when the notebook already exists

Most teams reading this are not starting a project. They have a working notebook and a deadline. The order below is deliberate: it front-loads the changes that make every later change safe, and it produces something deployable at the end rather than a refactor that never lands.

Step three is where teams discover how bad the situation is, and it is worth doing early for exactly that reason. Rebuilding the model from a clean container and comparing it against the original number is a two-hour job when the environment was pinned and a multi-day investigation when it was not. Better to learn that in week one than in the week of the launch.

Step six is where the argument usually happens. Somebody will want the endpoint because the roadmap says "real-time". Ask for the decision latency, in writing, from the person who owns the outcome. Half the time the honest answer is measured in hours and the endpoint disappears from the plan along with a quarter of the operating burden.

What this costs against what it saves

Adopted at the start of a project, these habits cost roughly a day of engineering across the first two weeks, and they cost nothing after that because they change how the work is done rather than adding a stage to it. Retrofitted onto a mature notebook, they run one to three weeks depending on how much duplicated transformation logic exists. A full rewrite runs two to four months on a nontrivial model, and it carries a cost the estimate never includes: reproducing the original result becomes a research project, because the specification was the notebook and the notebook was ambiguous.

There is a second, quieter saving. Systems built this way can be handed to another team. The reason handoffs fail is almost never that the receiving engineers are not capable. It is that they inherit an artifact with no tests, no environment, no data lineage, and no way to tell whether a change made things worse. Give them a package, a lockfile, a versioned dataset, and a failing-on-regression test, and the handoff takes a week. That is the same property that makes an on-call rotation survivable and a vendor change tractable, and none of it requires a platform or a purchase.

Bottom line

The notebook is not the problem and the rewrite is not inevitable. Put the code where it can be imported, make the kernel restart cleanly, address data by version, pin the environment, write each transformation once, and freeze an evaluation set with a test attached. Six habits, adopted in week one, cost about a day. Skipped, they cost a rebuild and the original result along with it.

Frequently asked questions