Running a model where the data cannot leave

The requirement arrives as a sentence about someone else's network

It rarely arrives as a specification. It arrives mid-meeting, as an aside: the data cannot leave our environment. For a team built on hosted model APIs, the instinct is to treat that as a hosting change — same product, different address. It is not. Egress is load-bearing in almost every modern AI stack, and removing it changes model selection, evaluation, release engineering, and support all at once. The projects that go badly are the ones that discover this one surprise at a time, after a date has been promised.

Pin the requirement down first, because "cannot leave" means at least four different things and each costs differently. It can mean a customer-controlled cloud tenancy with egress blocked by network policy, where instances still get patched through an approved path. It can mean their own datacenter behind an outbound proxy and a narrow allow-list. It can mean genuinely disconnected — no route outward at all, updated only by physical media through a documented transfer procedure. Or it can mean an accredited enclave at a stated impact level, where the binding constraint is not network topology but the authorization package the system has to fit into.

These are not points on a spectrum you slide along later; they imply different architectures. Ask which one, in writing, before quoting anything — what air-gapping actually costs walks the commercial consequences, and what impact levels change for an AI workload covers the federal side.

You will not be picking the hardware

In a hosted deployment you choose the instance type. Inside someone else's boundary the compute already exists, is already authorized, and is already inventoried in a system security plan. That inverts the usual order of decisions, and the inversion is the most useful thing to internalize early.

Adding a card is not a purchase, it is a configuration change. NIST SP 800-53 Rev. 5 routes changes through a configuration change control process (CM-3), and a large enough change becomes a "significant change" under NIST SP 800-37 Rev. 2 — defined there as one likely to substantively affect the security or privacy posture of a system — which can trigger reauthorization. "We will just add a GPU" is a schedule item measured in months, not a line item measured in dollars.

The installed hardware is usually a generation or two behind current, and the number that matters is memory, not throughput. From NVIDIA's published datasheets: the A100 ships in 40 GB and 80 GB configurations, the H100 SXM at 80 GB of HBM3, the inference-oriented L40S at 48 GB of GDDR6. Capacity decides whether a model runs at all; bandwidth only decides how fast once it fits. At the most sensitive tiers the hosting question is settled before you arrive — DISA's Cloud Computing Security Requirements Guide, which defines the DoD impact levels, places controlled unclassified information at IL4 and IL5, with IL5 also covering unclassified National Security Systems, and reserves IL6 for information classified up to Secret on infrastructure connected to a classified network. The practical translation: the reachback a commercial design assumes is not merely discouraged. There is nothing on the other end of it.

The memory arithmetic almost nobody does up front

Most model-selection conversations start with quality and end in a surprise. Reverse the order. The memory ceiling produces a short list, and you choose the best model on that list — not the other way around.

Three things consume GPU memory. Weights are the obvious one: parameter count multiplied by bytes per parameter, plus the per-group scales and zero points that quantized formats carry. The KV cache is the one that surprises people, because it grows with context length and with the number of requests in flight simultaneously. Per token it costs two tensors (keys and values) for every layer, for every key/value head, at the head dimension, in whatever precision the cache is held. Grouped-query attention — introduced in "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints" (Ainslie et al., EMNLP 2023) and now standard in open-weight models — shrinks that by letting several query heads share one key/value head. Third is runtime overhead: activations, framework allocations, and fragmentation. The vLLM paper, "Efficient Memory Management for Large Language Model Serving with PagedAttention" (Kwon et al., SOSP 2023), measured that prior serving systems held actual token state in only 20.4%–38.2% of the KV-cache memory they allocated, the rest lost to fragmentation and over-reservation; the paged allocator that fixes it works by reserving a pool up front, which is memory you must budget rather than memory you get back.

Illustrative arithmetic, not a measurement of any deployed system. The per-token expression is the standard one for a transformer with grouped-query attention; substitute the real layer count, KV-head count, head dimension, and cache precision for the model under consideration. Sources: Ainslie et al., EMNLP 2023 (GQA); Kwon et al., SOSP 2023 (PagedAttention).

The row that decides the design is concurrency. A model that fits comfortably for one analyst at an 8,000-token context may not fit for sixteen analysts at once, and concurrency is almost always what the customer actually needs. Two numbers — peak simultaneous users and maximum context length — constrain the candidate list more than any benchmark score does. Get both in the first conversation.

What quantization buys, and what it costs

Quantization is the standard lever for fitting under the ceiling. The two post-training methods most widely implemented in open tooling are GPTQ, from "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers" (Frantar et al., ICLR 2023), and AWQ, from "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration" (Lin et al., MLSys 2024). Four-bit weight quantization cuts the weight footprint to roughly a quarter of 16-bit, which is frequently the difference between fitting on the installed card and not.

It is not free, and the honest version of the cost is that it is task-dependent and not predictable from published averages. Those averages are measured on public benchmarks. Your customer's corpus — open-source intelligence reporting, claims files, maintenance narratives, sensor logs — is not those benchmarks, and degradation on a specialized corpus can differ sharply from the aggregate. The only number that means anything is measured on their data, inside their boundary; anyone quoting a quality delta before that measurement exists is quoting somebody else's result. Quantizing the KV cache is a separate lever with a separate cost, and often the one that actually buys the concurrency, so evaluate it on its own rather than folding it into a single "we quantized it" claim. For the smaller end of the range, small language models for federal edge deployments covers what that class does well and where it breaks.

The weights have to get through the door too

Model files are software, and they enter the boundary the way software enters it: on approved media, hash-verified against a value obtained through a separate channel, scanned, and recorded. Two details matter more than teams expect.

The first is file format. Checkpoints serialized with Python's pickle protocol reconstruct arbitrary objects on load, so loading an untrusted checkpoint can execute arbitrary code — a property acknowledged in the PyTorch documentation and the reason Hugging Face introduced safetensors, a container holding raw tensor data with no embedded executable logic, subsequently put through a published security audit. Where you cannot simply re-download and try again, prefer safetensors and record the format in the package rather than leaving it implicit.

The second is that nothing installs itself. No package resolution at runtime, no weight download at first boot, no license or telemetry handshake. Every dependency is mirrored and carried in as a fixed set — an offline install against a local wheel directory with the public index disabled, and a container image built outside and transferred whole. That is the shape of the build we describe on our classified AI capability page: open-weight models imported through the accredited transfer path with SHA-256 verification and registration in a model registry, served on vLLM, TensorRT-LLM, llama.cpp, or Triton with nothing reaching outward, retrieval self-hosted on Qdrant, Milvus, Weaviate, or pgvector with classification labels carried at the chunk level.

Evaluating when nothing can be sent out

The standard evaluation loop assumes egress in three places at once: a hosted judge model, a public leaderboard for comparison, and telemetry reporting how the deployed system behaves. All three are gone, and what replaces them has to be built before the install — because afterward, the people who can see the data are not you.

The golden set is the center of it. It is assembled inside the boundary by people authorized to see the data, and it never leaves, which makes the customer's staff time a line item in your plan rather than an assumption. Judging can still be automated — the LLM-as-judge method from "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (Zheng et al., NeurIPS 2023) works offline — but the judge is now a model you carried in, and a judge from the same family as the system under test will flatter it. Report per-item results the customer can inspect rather than one average, because averages hide exactly the failures that matter in a mission workflow. Our piece on red flags in an AI benchmark goes further into how these numbers get inflated.

• Real documents — drawn from the corpus the system will actually serve, not synthetic stand-ins written to be easy.
• The failures they already know about — every organization has a list of cases their current process gets wrong. That list is the most valuable test data in the building.
• Abstention items — questions the correct answer to which is "not in the corpus." A model that never declines is a liability in an analytic workflow.
• Long-context items at the real ceiling — quality at 2,000 tokens tells you very little about quality at 100,000.
• Latency under concurrency — measured at peak simultaneous users on the actual hardware, not single-request timings.
• A frozen version and a changelog — an evaluation set that quietly changes between runs cannot support a claim about a release.

Determinism is what they will actually test

Assessors and customer engineers reach for reproducibility early, because it is easy to check and it reveals whether a system is understood. Be precise about what can honestly be promised. PyTorch's own reproducibility documentation states that "completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms," and that results "may not be reproducible between CPU and GPU executions, even when using identical seeds." The torch.use_deterministic_algorithms() switch forces deterministic kernels where they exist and raises an error where they do not, at a performance cost the documentation is explicit about.

There is a second effect specific to serving. Continuous batching changes the shape of the reductions inside a forward pass, and floating-point summation is not associative, so identical prompts can produce slightly different logits depending on what else happened to be in the batch. If a claim of determinism goes into an authorization package, scope it to a stated configuration — greedy decoding, fixed batch composition, pinned framework and driver versions — and say so in that language. A promise that quietly depends on an idle server is a promise that fails during the customer's own testing.

The update path is the architecture

In a connected product, release cadence is yours to choose. Inside a boundary it belongs to the transfer window. Media transfers are scheduled, reviewed, and often mediated by a cross-domain solution from the certified baseline maintained by the National Cross Domain Strategy and Management Office at NSA. If transfers happen monthly, the patch clock is monthly, regardless of what upstream projects do.

Three design consequences follow. Ship a bundle rather than a stream: one signed artifact carrying weights, container images, dependency wheels, the evaluation set, release notes, and a bill of materials. That bill of materials is not decorative — NTIA's Minimum Elements for a Software Bill of Materials, published 12 July 2021 under Executive Order 14028, defines the baseline data fields and names SPDX, CycloneDX, and SWID as acceptable formats, and it is how the receiving security team decides whether your update contains anything they must act on. Second, treat patching as a control rather than a courtesy: SI-2 (flaw remediation) and SA-22 (unsupported system components) in NIST SP 800-53 Rev. 5 are the ones that bite, and SA-22 requires replacing components once the developer or vendor stops supporting them, or arranging alternative support. A dependency you pinned and forgot becomes a finding when it goes end-of-life inside an enclave nobody can reach. Third, keep expensive changes rare — swapping model families or inference runtimes can meet the significant-change threshold in NIST SP 800-37 Rev. 2 and pull the system back through reauthorization, so design an interface and container that stay stable while the weights behind them change.

• Which of the four meanings of "cannot leave" applies — in writing
• The exact installed hardware, including memory per card and card count
• Maximum context length and peak concurrent users
• Who builds the evaluation set, when, and who is cleared to see the data
• Transfer window cadence and who approves a transfer
• What the customer's logging platform ingests, and in what format
• Who signs the authorization, and what evidence they expect from you

Support with the telemetry switched off

No crash reports arrive, no usage analytics accumulate, no remote session is possible. Logs go to the customer's own monitoring stack and stay there, so the system has to explain itself locally: visible version stamps, health endpoints stating what is loaded, and a diagnostic bundle the customer can generate, review, and choose whether to release. The first responder to any incident is their engineer, not yours — so documentation and error messages carry weight they never carry in a hosted product, and the support contract has to price that rather than assume it away. The companion piece on headless deployment into a customer-controlled sandbox covers the interface contract in more depth.

Where our scope ends

Being specific about refusals is more useful than claiming breadth, so here is ours on this subject.

How an engagement starts

The first conversation is short and mostly diagnostic. Which of the four constraints applies. What hardware is already installed and authorized. What peak concurrency and context length really are. Who can build an evaluation set, and when. What the transfer cadence is. Those answers decide whether the honest recommendation is a bespoke small model, an off-the-shelf open-weight model with retrieval, or a different architecture entirely — and they decide it before anyone has spent money on the wrong one. Reach us through the contact form; a paragraph describing the situation is enough to start, and if the work belongs somewhere else we will say so. Our standing scope and refusals are written out at what we build and what we decline.

Where to check this yourself

Every external claim above traces to a primary document. These are the documents, not summaries of them.

• NIST SP 800-37 Rev. 2 — the Risk Management Framework, including significant change, ongoing authorization, and reauthorization. csrc.nist.gov
• NIST SP 800-53 Rev. 5 — CM-3 configuration change control, SI-2 flaw remediation, SA-22 unsupported system components. csrc.nist.gov
• DoD Cloud Computing Security Requirements Guide — the definition of impact levels IL2 through IL6, published by DISA. cyber.mil
• NTIA, The Minimum Elements For a Software Bill of Materials, 12 July 2021, issued under EO 14028. ntia.gov
• National Cross Domain Strategy and Management Office — the NSA office maintaining the certified cross domain solution baseline. nsa.gov
• Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention," SOSP 2023 — KV-cache fragmentation and paged allocation. arXiv:2309.06180
• Ainslie et al., "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints," EMNLP 2023. arXiv:2305.13245
• Frantar et al., "GPTQ," ICLR 2023 and Lin et al., "AWQ," MLSys 2024 — the two widely implemented post-training quantization methods. arXiv:2210.17323 · arXiv:2306.00978
• Hugging Face safetensors security audit — the format and what it removes relative to pickle-based checkpoints. huggingface.co
• PyTorch reproducibility notes — determinism guarantees, and their limits. docs.pytorch.org

Related reading on this site: air-gapped inference and what actually runs with no network, GPU capacity planning for federal AI workloads, and SBOM and software supply chain for federal AI.

Frequently asked questions