Python packaging and deployment that holds

The artifact you are actually shipping

Ask a team what they deploy and most will say "the service." What they deploy is an interpreter, a resolved set of third-party packages, a set of compiled native libraries those packages link against, a process supervisor, and a start command. The application code is usually the smallest and most stable part of that stack. Every rebuild re-derives the rest, and unless something forces the derivation to come out the same way twice, it will not.

That is the whole problem, and it explains most of the symptoms. A build that passed on Tuesday fails on Thursday because a transitive dependency published a minor release. A container that runs locally crashes on the cluster because the base image is slimmer and a shared object is missing. A rollback restores old code against a migrated schema and the old code cannot read it. A deploy drops in-flight requests because nothing forwards SIGTERM to the worker pool. None of these are exotic. All of them are cheap to prevent and expensive to diagnose at 2 a.m.

We work on Python systems in two modes: building new services, and taking over ones that were built quickly and then had to survive. The list below is what we check first in either case. It is ordered by how often the control is missing and how much damage its absence does.

Application or library: the split that decides your dependency policy

Before any tooling argument, settle one question. Is this thing installed by other people, or is it deployed by you? The answer inverts every rule that follows.

A library declares ranges and pins nothing. Its job is to coexist with whatever else the consumer already has. If your library pins requests==2.32.3, you have just handed every consumer a version conflict. Express compatibility honestly with PEP 508 specifiers in pyproject.toml, use lower bounds you have actually tested and upper bounds only where you know a breaking change exists, and let the consumer's resolver do its job.

An application pins everything, transitively, with hashes. Its job is to run the same way on every machine that starts it. There is no consumer to accommodate. The application repository holds both files: pyproject.toml with the ranges you intend to support, and a generated lock file with the exact resolution you are shipping. The first is intent. The second is fact.

Teams get into trouble by treating a service like a library, usually because the same engineers write both and the habits transfer. A service with loose ranges and no lock is a service whose runtime is decided by whichever index mirror answered fastest on the day of the build.

What a lock file locks, and what it does not

Python's packaging standards separate metadata from installation deliberately. PEP 621 defines project metadata in pyproject.toml. PEP 517 and PEP 518 define how a build backend is invoked and how its own build-time requirements are declared. PEP 440 governs version identifiers and specifiers. None of those produce a lock. A lock is a tool-level artifact, and until recently every tool invented its own.

PEP 751 changes that by standardizing a lock file format, pylock.toml, so that a lock produced by one tool can be installed by another. Tool support is still arriving, so pick a tool now and treat the standard as the exit path rather than a reason to wait.

Whichever tool you pick, three properties matter and they are independent:

Transitive completeness. Every package that will be installed appears in the file, not just the ones you named. If a dependency of a dependency is absent from the lock, it is unpinned, and it is exactly the kind of package that publishes on a Friday.

Hashes. Each entry carries the digest of the artifact. Without hashes you have pinned a name and a version, which the index can serve differently to different clients, and which a compromised or mirrored index can serve differently on purpose.

Fail-closed installation. The install step must refuse to proceed when the environment does not match the lock. With pip that is pip install --require-hashes -r requirements.lock, which errors if any requirement lacks a pin or a hash. That flag is the point of the exercise. A lock file that the installer is free to ignore is documentation.

Tool	Lock artifact	Hashes	Fits when
pip-tools pip-compile + pip-sync	requirements.txt compiled from .in or pyproject.toml	Yes, with --generate-hashes	You want the smallest change from what you already run, and installation stays plain pip
uv	uv.lock, plus export to requirements format	Yes	Build minutes matter, you have many environments, or you want resolver and installer from one binary
Poetry	poetry.lock	Yes	The team wants one tool for dependency management and publishing, and accepts its opinions
PDM	pdm.lock	Yes	You want standards-forward behavior and multi-platform locks in one file
conda / mamba	environment.yml, explicit spec files	Partial, via explicit locks	Native scientific stacks where the binary dependency graph reaches outside Python

We use pip-tools on systems where the install path must stay boring and auditable, and uv where build time is a real cost and the team is comfortable adopting a newer tool. Both produce hashed, transitively complete locks, which is the part that matters. The tool argument is much less important than the property argument, and teams spend the ratio backwards.

One more rule that saves a category of confusion: lock per target, not per developer. If production runs Linux on x86-64 with CPython 3.12, the lock that gates the deploy is generated for that target, in a container matching it, in CI. A lock generated on a laptop running macOS on Apple silicon can resolve to different wheels and different platform-conditional dependencies, and it will pass review because the file looks the same.

Wheels, tags, and why the build works on your laptop

The single most common "it worked in CI" failure has nothing to do with Python code. It is a wheel tag mismatch, and understanding tags removes a whole class of mystery.

A built distribution filename encodes what it will run on. numpy-2.1.0-cp312-cp312-manylinux_2_28_x86_64.whl says CPython 3.12, that ABI, and a Linux with glibc 2.28 or newer on x86-64. PEP 600 defines those perennial manylinux tags against a glibc version rather than a distribution name, which is why the numbers appear where a distro codename used to. If no wheel matches your target, pip does not fail. It falls back to the source distribution and tries to compile, which is why an install that took 20 seconds on one machine takes 11 minutes on another and then fails on a missing header.

Two consequences worth internalizing. First, Alpine is not a smaller Debian. Alpine uses musl instead of glibc, so manylinux wheels do not apply. PEP 656 defines musllinux tags and coverage has improved, but it is still thinner, and choosing Alpine for a Python service frequently trades 40 MB of image size for a compiler toolchain, a longer build, and a different set of runtime bugs. For most services, a slim Debian-based image is the better default.

Second, add --only-binary=:all: to the install in CI. It converts a silent source build into a loud failure at the moment you introduced it, instead of a surprise the first time the build runs on a machine without the toolchain. If a package genuinely has no wheel for your target and must be compiled, you want that decision made deliberately, documented, and built once in a builder stage rather than rediscovered on every deploy.

The image: layer order, digests, and the cache you keep destroying

Most Dockerfiles we inherit for Python services have the same defect. They copy the whole source tree, then install dependencies. That invalidates the dependency layer on every code change, so a one-line fix reinstalls the full dependency set. On a service with a scientific or machine-learning stack that is several minutes per build, paid on every commit.

Copy the lock file first, install from it, then copy the source. Dependencies change weekly and code changes hourly, so ordering the layers by rate of change is what makes the cache useful. Pair it with a real .dockerignore, because COPY . . with no exclusions pulls in .git, local virtual environments, test fixtures, and any credential file a developer left in the working tree.

The second structural improvement is a multi-stage build. Stage one carries the compiler toolchain and any headers needed to build packages that lack wheels for your target. It creates a virtual environment and installs the locked dependency set into it. Stage two starts from the same slim base with no toolchain, copies that virtual environment whole, copies the application source, drops to a non-root user, and sets the entrypoint. The runtime image never contains a compiler, which cuts both size and attack surface without changing what the application can do.

Pin the base image by digest, not by tag. FROM python:3.12-slim resolves to a moving target. FROM python:3.12-slim@sha256:... resolves to exactly one image forever. Update the digest deliberately, on a schedule, as a reviewed commit, so that a base image change is a change you can see in a diff and bisect later. This is the control that turns "the build started failing and nobody touched anything" into a two-minute investigation.

Reproducible enough to be useful

Bit-for-bit reproducible container images are achievable and are usually not worth the effort for an application team. Timestamps, file ordering, and compiled bytecode all fight you, and the payoff is small compared to the next control down.

The property you actually need is weaker and far more valuable: given a commit, you can rebuild an image that behaves identically, and you can prove which image is running. Get there with four cheap moves. Record the resolved image digest at deploy time, not the tag. Expose a version endpoint that returns the git commit, the image digest, and a hash of the lock file. Set PYTHONDONTWRITEBYTECODE=1 and PYTHONUNBUFFERED=1 so the runtime does not scatter state and logs are not held in a buffer during a crash. Precompile bytecode once during the build with python -m compileall so first-request latency does not include compilation.

If you do want closer determinism, SOURCE_DATE_EPOCH normalizes timestamps for build backends that honor it, and BuildKit can produce more stable layers. Treat that as a refinement after the four moves above, not before.

The process model is where the outages live

Packaging gets the attention. Process supervision causes the incidents. A typical WSGI or ASGI service in production runs a master process and a pool of workers, and the interesting behavior is at the edges.

Worker count. The classic starting point of two times the core count plus one is a reasonable default for CPU-bound synchronous work and a poor one for anything else. An I/O-bound async service wants fewer processes and more concurrency inside each. A service holding database connections has a hard ceiling: workers multiplied by pool size must stay under the database connection limit, and this is the arithmetic that most commonly takes a service down under load, well before CPU is the constraint.

Signals and PID 1. Use the exec form of CMD so your supervisor is PID 1 and receives SIGTERM directly. Shell form wraps the command in /bin/sh -c, which does not forward signals, so the orchestrator's graceful stop does nothing and every deploy ends in a kill after the grace period. In-flight requests die, and the dashboard shows a small error spike on every release that everyone learns to ignore.

Timeout ordering. Three timeouts must be ordered deliberately: the application request timeout, the worker timeout, and the platform's grace period. Kubernetes defaults terminationGracePeriodSeconds to 30, and plain Docker gives 10 seconds before SIGKILL. If your longest legitimate request takes 45 seconds, a graceful drain cannot finish and you lose those requests on every deploy. Either shorten the request or lengthen the grace period, but decide it rather than discovering it.

Preloading. Loading the application in the master before forking saves memory through copy-on-write and speeds worker startup. It also means anything created at import time is inherited by every worker. Database connections and client objects created at import and then forked produce corruption that looks random and is not. If you preload, create connections in a post-fork hook.

Configuration, and the secrets that live forever in layer history

Configuration comes from the environment, never from a file baked into the image, because the image should be identical across staging and production. The variable part is the environment, and that is the whole point of building once and promoting the same artifact.

Validate configuration at startup and fail immediately. A typed settings object, with pydantic-settings or an equivalent, that parses every variable and raises on a missing or malformed value turns a 3 a.m. KeyError in a request handler into a container that refuses to start and says why. The orchestrator then holds the previous version, which is exactly the behavior you want.

Never put a secret in ENV or ARG in a Dockerfile. Both persist in image metadata and layer history, readable by anyone who can pull the image, and a later unset does not remove them from the earlier layer. If a build genuinely needs a credential, use BuildKit secret mounts, which expose the value to a single build step without writing it into a layer. If a credential has already shipped this way, rotate it. Editing the Dockerfile does not retract the image.

Migrations decide whether you can roll back

Deployment is reversible. Schema change is not, and coupling them makes both irreversible. The rollback that fails is almost always old code meeting a new schema.

Run migrations as a separate step from the application rollout, not from an application entrypoint. Entrypoint migrations run once per replica, race each other during a rolling update, and make the migration's success dependent on which container happened to start first.

Use expand and contract for anything destructive. To rename a column, add the new one, write to both, backfill, move reads to the new column, then drop the old one in a later release. Each step is independently deployable and independently reversible, and at every point in the sequence both the current and previous application version can run against the current schema. That last property is the whole test. If it does not hold, the deploy has no rollback regardless of what your pipeline claims.

Two operational details that cost real downtime. Long-running migrations that take table locks will block traffic on a busy table, so run them with a lock timeout and a retry rather than letting one statement queue every subsequent query behind it. And index creation should be concurrent where the database supports it, since the non-concurrent form holds a write lock for the duration.

Health checks that do not take down the fleet

Liveness and readiness answer different questions and conflating them creates outages that look like infrastructure failures.

Liveness asks whether the process is wedged and needs restarting. It should be trivial and depend on nothing external. A handler that returns 200 is correct. If liveness checks the database, a brief database blip restarts every replica at once, which converts a recoverable dependency problem into a total outage plus a thundering herd of reconnections.

Readiness asks whether this replica should receive traffic right now. Checking the dependencies it needs to serve a request is appropriate here, because failing readiness removes one replica from rotation instead of killing it. Cache the result for a second or two so a probe every few seconds does not add measurable load.

Startup covers slow initialization. A service importing a large machine-learning stack or loading model weights can take a minute to become useful, and without a startup probe the liveness probe kills it before it finishes, forever, in a loop that looks like a crash.

The supply chain: hashes, indexes, and one flag that opens a hole

Hash pinning is a supply-chain control before it is a reproducibility control. With --require-hashes, an index that serves a different artifact under a name and version you already resolved cannot install. Without it, the index is trusted on every build, forever.

The specific configuration mistake worth naming is --extra-index-url. Many teams host internal packages on a private index and add it alongside PyPI with that flag. It does not mean "look here first." pip considers every configured index together and selects by version, so a public package published under your internal package's name at a higher version number wins. That is the dependency confusion attack class, and it is a configuration issue rather than a vulnerability in pip. The fix is to point --index-url at a single repository that proxies public packages and hosts internal ones, so there is one namespace and one authority. Artifactory, Nexus, and the managed artifact registries all do this.

Alongside that, run pip-audit against the lock in CI and generate an SBOM in CycloneDX or SPDX on every build, stored with the image. The audit tells you what is known-vulnerable today. The SBOM is what lets you answer "are we affected" in ten minutes rather than two days the next time a widely used package has a bad week. Both are a few lines of pipeline configuration, and the SBOM in particular is the kind of thing enterprise customers and SOC 2 auditors now ask for by name.

Native dependencies and the machine-learning tax

Services with a scientific or deep-learning stack change the calculus, mostly through size and build time. A default install of a major deep-learning framework pulls CUDA runtime libraries and produces a multi-gigabyte image even when the service will only ever run inference on CPU. Frameworks publish CPU-only builds on separate index URLs for exactly this reason, and selecting one is usually the single largest image reduction available.

Model weights are the second decision. Baking them into the image makes the image self-contained and immutable, at the cost of size and a full rebuild for every model update. Loading them at startup from object storage keeps the image small and decouples model releases from code releases, at the cost of a startup dependency and a slower cold start. Both are defensible. What is not defensible is downloading weights from a public hub at container start with no pinned revision, because the running model then changes without a deploy and nothing in your logs will say so. If weights are fetched at runtime, pin the revision and verify the digest.

Five failures, and what they look like from the outside

Symptom	Actual cause	Control that prevents it
Build broke, nobody changed anything	A transitive dependency or the base image tag moved	Hashed lock plus digest-pinned base image
Works locally, fails in the cluster	No matching wheel for the target platform, or a missing shared library in a slimmer base	Lock generated in a container matching production; --only-binary=:all:
Small error spike on every deploy	SIGTERM not reaching workers, or grace period shorter than the longest request	Exec-form CMD, ordered timeouts, drain before shutdown
Rollback made it worse	Old code against a migrated schema	Expand and contract, migrations decoupled from rollout
Brief database blip became a full outage	Liveness probe checking an external dependency	Trivial liveness, dependency checks in readiness only
Connection pool exhausted well below CPU limits	Workers multiplied by pool size exceeds the database limit	Size the pool against worker count before load testing

What we see most often on a first pass

• A lock file that CI regenerates instead of verifying, which means the pin is decorative
• Layer order inverted: the source tree copied before dependencies are installed, so every commit rebuilds the full dependency layer
• Base image referenced by a floating tag, so the runtime changes without a commit
• Migrations run from the container entrypoint, racing every other replica during a rolling update
• Shell-form CMD, so graceful shutdown has never once worked and nobody noticed
• An internal index added with --extra-index-url, leaving package names resolvable from two authorities

A two-week hardening sequence

Two weeks is realistic because none of it requires an application rewrite. Every item is release plumbing, testable in a staging environment, and independently shippable. Do them in this order because each one makes the next easier to verify: once the lock is enforced, the Dockerfile change is provably behavior-neutral; once the image is deterministic, a probe change can be attributed to the probe.

The pre-deploy checklist

• CI installs from the lock with --require-hashes and fails when the lock is stale
• The lock was generated on the production platform and interpreter version
• Base image pinned by digest; digest bumps are reviewed commits
• Runtime image contains no compiler, no build tools, and runs as a non-root user
• A version endpoint returns commit, image digest, and lock hash
• Graceful shutdown verified by watching in-flight requests during a rolling restart
• Liveness depends on nothing external; readiness carries the dependency checks
• The previous application version can run against the current schema
• One index URL, an SBOM stored with the image, and an audit step that can fail the build

Bottom line

Python packaging has a reputation for chaos that is mostly out of date. The standards landed, the tools converged on hashed transitive locks, and the remaining difficulty is not knowing what to pin but deciding to enforce it. A service becomes durable at the point where a rebuild is a lookup instead of a derivation: the lock says which artifacts, the digest says which base, the version endpoint says which image is running, and the migration sequence guarantees the previous version still works.

None of that is expensive. It is a week or two of release plumbing on a service that already exists, and it is the difference between a deploy that is boring and one that requires the person who wrote it to be awake.

Frequently asked questions