Air-gapped deployment of machine learning

The constraint is the transfer cadence, not the missing network

Teams preparing their first disconnected delivery spend their planning energy on the wrong thing. They ask whether the model can run without a network, which is a software question with a known answer and a known test method. The question that decides whether the program lands on schedule is different and it is administrative: how often can a file cross the boundary, who approves it, and what does one crossing cost? Every engineering decision downstream — how you package, how you sign, how you version, how you update, how you find out whether it is working — is a consequence of that number.

The arithmetic is unforgiving. On a connected system, a defect found in staging is fixed in an hour and redeployed in ten minutes. Inside a boundary where transfers happen on a weekly review cycle, that same defect costs a week, and the second defect found after the first fix costs another. Three sequential mistakes is a month, and the month is visible to a program office that has already scheduled an operational test. The discipline of this work is front-loading: the goal is to need one transfer where an unprepared team needs five.

This piece is about the delivery and the sustainment. The narrower runtime question — which libraries quietly reach out to the internet, and how to enumerate them before you travel — is covered separately in what actually runs with no network. Assume that work is done and the software genuinely runs offline. The program can still go badly.

Three environments, all called "air-gapped"

The word covers at least three arrangements with different costs. Establishing which one you are entering, in writing, before you scope anything, prevents most of the surprises.

The egress-restricted enclave. There is a real internal network. There are internal DNS resolvers, an internal package mirror, an internal container registry, internal identity. What is missing is a route to the public internet. This is the friendliest of the three and the most common in civilian agencies and in commercial regulated environments. Delivery means getting artifacts into the internal registry once; after that, the enclave behaves almost normally, and the work looks like ordinary platform engineering with a strict allow-list.

The physically separated network with a controlled transfer path. Two networks that do not touch, plus an approved mechanism for moving data between them: removable media under an escorted procedure, a one-way transfer device, or an accredited cross domain solution. This is the classified-network case and the one most people mean. Transfers are events, not operations. They have owners, forms and a queue.

The standalone system. One host, or a small cluster, with no network beyond its own switch. Common in test ranges, mobile shelters and lab instruments. Here the constraint is not only transfer difficulty but the absence of any platform to build on — no registry, no mirror, no identity provider, no log aggregation. Everything the system needs is on the disk you carry in.

Where the requirements actually come from

Buyers do not invent these constraints. They inherit them, and the specific documents matter because they tell you which arguments are winnable and which are not. A few of these moved recently, and one of them is genuinely in flux.

One item is moving under everyone's feet. On September 24, 2025 the Department announced the Cybersecurity Risk Management Construct, a replacement for the Risk Management Framework that organizes security work across five phases — design, build, test, onboard and operations — with continuous automated assessment rather than periodic checkpoints. It is early, implementation guidance is still arriving, and programs are at different points in adopting it. The honest posture with a customer right now is to ask which construct their authorizing official is operating under this fiscal year rather than to assume.

The bundle is the deliverable

On a connected system the deliverable is a repository and a pipeline. Here it is a single reviewed, signed, hashed archive, and its contents are worth being specific about, because the items teams forget are always the same items.

• Container images as archives — exported OCI or Docker archives with their digests recorded, not a registry reference. A tag is a promise about a network you will not have.
• A complete dependency set — every wheel, every transitive dependency, every platform variant for the target architecture. Installing a local file still consults an index unless you explicitly install with no index against a local directory.
• Weights, tokenizer, and config together — with hashes, and with the caches pre-seeded at the paths the runtime actually searches. Weights alone are a partial delivery.
• The build toolchain, not only the build output — if any artifact is compiled or optimized for specific hardware, the thing that produced it travels too, at the same versions.
• A held-out evaluation set and its harness — so acceptance can be demonstrated on-site rather than asserted from a slide about work done elsewhere.
• A rollback artifact — the previous known-good version, in the same bundle. Restoring from backup is not the same thing as reverting to a version you shipped.
• Signatures, trust roots, and the SBOM — everything needed to verify the bundle without reaching a certificate authority, a transparency log or a revocation responder.
• The runbook and the license files — install, verify, start, stop, roll back, and the redistribution terms for every model and library in the archive.

Signatures that verify with nothing to call

CM-14 says components are not installed without verifying a digital signature the organization recognizes. That sentence is easy to satisfy on a connected network and quietly hard here, because most modern signing infrastructure assumes reachability at verification time as well as at signing time.

Sigstore is the common case. Keyless signing with cosign obtains a short-lived certificate from Fulcio bound to an OIDC identity and records the signature in the Rekor transparency log. All of that is fine on the outside, where you sign. Inside, the verifier has no route to either service. The documented path is to bundle the Rekor inclusion proof with the signature so verification can run offline against material already in the archive. Where the signing environment itself cannot reach the log, cosign can sign without uploading to the transparency log at all — which works, and costs you the transparency guarantee. That is a real trade and it should be a decision someone makes deliberately, not a flag someone adds to make an error go away.

The provenance layer sits above the signature. The SLSA build track describes increasing assurance about how an artifact was produced; its top build level requires a hardened, isolated build environment such that a compromised build script cannot forge the provenance. For a bundle that will be inspected by a security team who has never met you, provenance is the part that answers "where did this actually come from," and it travels as a file like everything else.

Then the bill of materials. The 2021 NTIA minimum elements were the long-standing baseline; CISA ran a public comment period in 2025, closing October 3, on an updated set that added component hash, license, tool name and generation context to the required fields, and a 2026 joint update has since been published that updates and replaces the NTIA document. The field list is what moved, so ask which set the receiving organization expects rather than shipping what your tooling emits by default. Our fuller treatment is in SBOM and supply-chain risk for federal AI.

A note on hardened base images, because it catches people. Iron Bank, the Platform One container repository, is a useful source of pre-hardened images, and its pipeline rebuilds them every 24 hours with updated OS packages. It currently operates at IL2, with IL5 and IL6 described as future work, and pulling from the registry requires an account rather than anonymous access. Two consequences follow: the images still travel into a higher-side enclave through your transfer procedure like anything else, and a daily rebuild cadence collides with a monthly transfer window. Pick a digest, record it, and ship that — do not track a moving tag you cannot reach.

Pin the versions you do not control

The second category of delivery failure is a version mismatch on something the customer owns. GPU drivers are the classic. CUDA minor version compatibility, available from CUDA 11 onward, lets an application built within a major release family run on a sufficiently new driver, and forward compatibility packages allow a newer-toolkit build to run on an older driver from a different major family, subject to platform and GPU support. Both mechanisms work. Neither works if nobody checked which driver is actually installed on the host inside the boundary, and "we will update the driver" is a change request against a system you do not administer.

The same reasoning applies to hardware-specific inference artifacts. An optimized engine built on one GPU model is not guaranteed to load on another; that is why the build toolchain belongs in the bundle. The specifics of engine portability are covered in the runtime piece. Container runtime version, kernel version, OS baseline, cgroup version and available storage driver all belong on the same list. Ask for the versions in writing, from the person who runs the host, before packaging anything.

Model licenses belong on this list too, because you are physically redistributing weights into someone else's facility. Terms vary more than people assume. The Llama community licenses grant broad rights to reproduce and distribute, and they attach conditions: a specific attribution notice must be retained in a NOTICE file distributed with copies you redistribute, and a monthly-active-user threshold above which a separate commercial license must be requested from Meta. These are open-weight terms, not OSI open-source terms, and the distinction matters when a government customer asks what they are allowed to do with the artifact after you leave. Read the license of the exact model you are carrying and put a copy in the bundle.

Updating a model that lives behind a boundary

This is where air-gapped programs succeed or slowly fail, and it deserves a decision rather than a default. Four patterns are viable, and they differ mainly in what has to cross the boundary and in which direction.

The direction of flow is the thing to understand early. Moving data toward a higher classification is a comparatively routine transfer. Moving anything outward — metrics, logs, a model that trained on enclave data — is the reviewed direction, because the review exists to prevent leakage, and it is slower, more constrained, and sometimes not available to you at all. A plan that assumes weights can come back out for analysis is a plan that may not survive its first security review. Where the data itself cannot move, the architecture question is treated in running a model where the data cannot leave.

Monitoring when nothing phones home

A fielded model degrades, and the usual apparatus for noticing — telemetry to a collector, dashboards, alerting — is exactly what the boundary removes. Dropping monitoring is the common response and the wrong one. What changes is not whether you monitor but who reads it and how the reading gets to you.

Three things make this work. First, everything computes locally and persists locally: input distribution summaries, confidence distributions, refusal and fallback rates, latency, throughput, error taxonomy. Retention is a sizing decision made in advance, because a disk filling up inside a boundary is an outage you cannot log into. Second, the operator inside the boundary is the primary audience, not you. That means a plain readable report rather than a metrics stream — a page a duty operator can look at and act on, with thresholds written down and a documented response for each. Third, whatever comes out to you comes out on the transfer cadence, in the reviewed direction, in a form that has already been checked for anything that should not leave. Aggregates and counts survive that review. Raw inputs generally do not.

Design the summary with the reviewer in mind and it will move; design it as a database export and it will sit in a queue. The measurement techniques themselves do not change much, and are covered in model monitoring in practice — what changes is the reporting path.

Acceptance happens inside, or it did not happen

The strongest thing you can carry into a disconnected install is an evaluation that runs on their hardware, on data they control, and prints a result while their people watch. It removes the entire category of argument that begins "it worked in our lab."

That means the held-out set travels in the bundle, the harness runs from the same container as the service, and the output is a signed artifact with the model digest, the container digest, the dataset hash, the host identifiers and the metric values. Somebody on their side countersigns it. That artifact is what your contract's acceptance criteria should point at — not a demonstration, not a briefing, and not a benchmark score produced somewhere else on hardware nobody in the room can inspect. Writing acceptance this way protects both parties: it gives the customer a real test and it gives you a defined finish line that does not move after the install.

What the schedule actually looks like

Durations below are planning ranges for a first disconnected delivery of a single ML service, and the transfer window is the term that dominates. Substitute the customer's real cadence as soon as you know it; if their transfer review runs monthly, the sequence stretches accordingly and no amount of engineering speed compresses it.

The contract terms that decide how this goes

Most of the pain in these programs is contractual pain wearing engineering clothes. Six things belong in the agreement before the date is set.

Named target hardware, at version. GPU model and count, driver version, OS baseline, container runtime. If the target changes after signature, that is a change, and it should be priced as one rather than absorbed.

Transfer cadence and ownership. How often a bundle can cross, who approves, and what the customer commits to. This is the single largest schedule risk and it is almost never the vendor's to control, which is exactly why it belongs in writing.

Escort and access. Who accompanies engineers, what tooling may enter the space, and what happens to the media afterward under the customer's sanitization procedure. A drive that does not come back is a cost, and it is a cheap one to plan for and an annoying one to discover.

Acceptance defined on the enclave-side artifact. The countersigned evaluation output, produced on their hardware, is the criterion. Say so explicitly.

Data rights on anything trained inside. If a model is fine-tuned on enclave data, the resulting weights are a new artifact and their rights need an owner named in the contract. This is easier to settle before the weights exist than after. Background in data rights in federal AI contracts.

Who holds the evaluation set. If it contains customer data it stays inside and you never see the raw records. That is workable, and it changes how you write the harness — the harness must be self-explanatory to someone running it without you.

Bottom line

Air-gapped machine learning is not a harder version of ordinary deployment. It is deployment with the feedback loop removed, and the engineering response is to move everything you would normally learn later to before the transfer: the versions, the toolchain, the rollback, the evaluation, the diagnostics, the license terms. Teams that do that ship once and hand over a working system. Teams that treat the boundary as an inconvenience at the end discover it in the worst room, on the worst day, with an audience.

Frequently asked questions