A model that works in the cloud is not a capability. It is a promise about a capability. The thing that gets fielded runs on a processor chosen three years ago against a thermal budget, inside a power envelope shared with everything else on the platform, with a network that may be intermittent, contested, or absent for the whole mission. The gap between those two situations is where integration schedules go, and it is almost always discovered at the wrong time: after the model team has demonstrated something impressive and the platform team is asked to make room for it.
This is written for the platform chief engineer who owns that room. The argument here is that edge deployment is not a compression problem to be solved at the end. It is an interface between two teams that has to be written down at the start, and most of the pain comes from that interface being implicit.
What the platform actually constrains
Model teams tend to hear "the edge" as a statement about memory. It is a statement about at least seven things, and any one of them can be the binding constraint.
Compute, in the form the hardware provides. Not floating point operations in the abstract, but which operations the accelerator supports at which precision, which layer types the runtime's compiler can actually fuse, and what happens to the ones it cannot: a silent fallback to the general-purpose processor that costs an order of magnitude. A model that looks fast on paper and contains one unsupported operation in its middle is slower than a plainer model that maps entirely onto the accelerator.
Memory, in two kinds. Space for the weights, and space for the activations at the batch size and input resolution the mission requires. The second is what surprises teams, because it scales with input size and is invisible in a parameter count.
Power and heat. A processor that sustains its peak rate for ninety seconds and then throttles is not a processor that sustains its peak rate. Sustained thermal performance under the enclosure's real airflow is the number that matters, and it is measured on the platform, not in a datasheet.
Latency, including everything around the model. The mission budget covers sensor read, preprocessing, inference, post-processing, and delivery to whatever consumes the output. Inference is often the smallest term. Teams optimize it because it is the one they own.
Determinism and scheduling. On a platform with real-time components, an inference task that occasionally takes four times its median is a scheduling hazard even if the median is fine. The relevant number is the worst case at a stated percentile under load, not the average.
Connectivity. How often, at what bandwidth, with what security, and whether the platform can be assumed to connect at all within an update cycle. This determines the entire update and monitoring design.
The software environment. Operating system, runtime version, driver stack, container support or its absence, and the configuration control regime governing all of it. On many platforms the runtime version is effectively frozen between accreditation cycles, which caps what model formats can be deployed at all.
What most often decides whether a model fits the platform
Editorial weighting, illustrative rather than measured. The last row is deliberately low: parameter count predicts almost nothing about on-platform behavior.
The integration contract between the platform team and the model team
The single change that most improves an edge AI integration is writing this contract down in the first two weeks. It has two halves, and each half is a set of numbers rather than intentions.
What the platform team states. The exact target: processor and accelerator part, memory available to the inference process, sustained power budget, operating system and runtime versions with the configuration control regime governing them, and whether containers are available. The timing budget for the whole inference path with the percentile it is stated at. The input contract: format, resolution, rate, color space, calibration, and what happens when a frame is dropped. The output contract: schema, units, coordinate frame, timestamp semantics, and how uncertainty is expressed. The failure contract: what the platform does when inference is slow, unavailable, or reports low confidence. And the deployment mechanism: how an artifact gets onto the platform, who signs it, and how long that takes.
What the model team states. The measured accuracy of the deployed artifact, on the platform, at the deployed precision, against the same test data used for the cloud baseline, broken out by operating condition. The measured timing distribution on the target, not an estimate. The memory footprint at the mission input size. The behavior outside the operating envelope and how the model signals it. The exact provenance of the artifact: what data, which training run, which conversion steps, with hashes. And the retraining and update expectations, so the platform team can plan the mechanism rather than be handed one.
Every argument we have seen between a platform team and a model team was an argument about a line item on this list that nobody had written down. The document is short. Writing it costs a week and saves a quarter.
Making the model fit, with the accuracy cost measured
Shrinking a model is a well-understood engineering activity, and the discipline that matters is not the technique. It is that every step is measured against the same evaluation the program will use to accept the system, on the target hardware, per operating condition.
Quantization reduces the numeric precision of weights and activations, which shrinks the artifact and, on hardware with the matching integer or reduced-precision paths, speeds it up substantially. Post-training quantization is fast and often sufficient, provided the calibration data is drawn from the deployed distribution rather than from a convenient sample. Where post-training conversion costs too much accuracy, quantization-aware training recovers most of it at the cost of a training cycle. The number to watch is not the aggregate accuracy delta but the per-slice delta, because quantization tends to cost most on exactly the rare and hard cases the mission cares about.
Pruning removes structure from the model. Unstructured pruning produces sparse weights that most edge runtimes cannot exploit, so the artifact is smaller and no faster. Structured pruning, which removes whole channels or blocks, produces a smaller dense model that any runtime speeds up. On constrained hardware, structured is almost always the right choice, and the honest comparison is against simply training a smaller model from scratch, which sometimes wins.
Distillation trains a compact model to match a larger one's behavior. It is the technique that most often produces a genuinely good small model, and it has a specific operational advantage: the large model can keep running in the cloud as a reference, giving the program a way to measure how far the fielded artifact has drifted from the intended behavior.
Architecture choice beats all three when it is still available. A model designed for the target's operator set and memory pattern outperforms an aggressively compressed model designed for a different world. This is why the platform target belongs in the first conversation and not the last.
Whatever the mix, the artifact that gets accepted is the converted, quantized, packaged one measured on the target. A cloud accuracy figure for a model that will not be deployed is not evidence about the system, and presenting it as such is how programs end up re-doing an evaluation campaign after conversion.
Two update models, compared
How a fielded model gets replaced is a system design decision with security, accreditation and sustainment consequences. Most programs pick implicitly and discover the consequences later.
| Dimension | Model bundled with the software build | Model as a separately signed data artifact |
|---|---|---|
| Update cadence | Tied to the platform's software release cycle | Independent, within a pre-approved envelope |
| Accreditation effect | Each model change is a software change | Model updates handled as data under a defined process, if agreed in advance |
| Rollback | Whole-build rollback, coarse and slow | Swap the artifact, keep the previous one resident |
| Integrity control | Inherited from the software signing chain | Requires its own signing, verification and version pinning |
| Best fit | Rarely updated models, tightly frozen platforms | Models expected to be retrained during the fielded life |
| Failure mode | A needed model fix waits months for a build slot | Version skew between model, preprocessing and interface schema |
The version skew risk in the right-hand column is the one to engineer against, and the fix is straightforward: the artifact carries its required preprocessing configuration, its input and output schema versions and its expected runtime version inside the signed package, and the platform refuses to load an artifact whose declarations do not match what it provides. That single check eliminates the most common field failure in separately-updated model deployments, which is a new model quietly receiving inputs prepared for the old one.
Monitoring a system that may never call home
Everything written about model monitoring assumes a network. On a disconnected or intermittently connected platform, monitoring has to be designed as an on-board function with an eventual reporting path.
The design that works has three layers. On-board detection computes the monitoring statistics locally: input distribution summaries against the reference recorded at test, output score distributions, abstention and low-confidence rates, timing percentiles, and resource use. These are small, bounded computations that fit in the leftover budget, and they produce a compact record rather than raw data.
Bounded local retention. A fixed-size store holds the summaries plus a sampling of interesting cases, chosen by rule: the highest-uncertainty inputs, the ones nearest the decision threshold, any input flagged as outside the operating envelope, and a random sample for unbiased reference. Sizing this store is a real design decision, and the sampling rules are a place where a little care pays for itself, because these records are the training and evaluation data for the next model version.
Opportunistic reporting. When a connection exists, the summaries go first, ordered by value, in a format that tolerates partial transfer. Bulk case data follows if the link permits. The reporting path carries the same handling and security controls as any other mission data, and that classification question should be settled before the first collection rather than after.
On-board behavior also needs a defined response to its own alarms. A platform that detects it is outside the operating envelope should change what it does: widen a threshold, mark its output as low confidence, hand back to the operator, or fall back to a simpler method. The point of on-board detection is not telemetry. It is that a disconnected system is the only one that can react to its own degradation in time.
Where an edge integration schedule is usually lost
Editorial weighting, illustrative rather than measured. The last row is deliberately low: the family is rarely the problem, the interface usually is.
The preprocessing trap
One failure deserves its own section because it is common, quiet, and expensive. Preprocessing gets implemented twice: once in the training pipeline, in one language with one set of libraries, and once on the platform, in another. The two implementations differ in ways nobody notices, because both produce plausible-looking inputs. Resize interpolation, color space conversion, normalization constants, channel order, padding, timestamp alignment, and the handling of missing or clipped values are all places where two reasonable engineers make different choices.
The effect is an accuracy loss that appears at integration, has no obvious cause, and gets attributed to the hardware or the quantization. Teams have spent weeks on it.
The fix is structural. Define preprocessing once, as a specification with test vectors: a set of recorded inputs with their exact expected preprocessed outputs, at a stated numerical tolerance. Both implementations are tested against the vectors in their own build pipelines. Any divergence fails a build rather than surfacing as a mysterious accuracy drop, and the vectors ship with the model artifact so a future maintainer inherits the check. Where the runtime allows, fold preprocessing into the model graph itself so there is only one implementation to begin with.
How we work inside a prime's program
Precision Federal builds AI, data platforms and software and delivers them into production inside federal agencies. On a platform integration we come in as the specialist subcontractor on the model, its conversion and packaging, and the on-board monitoring, while the prime holds the platform, the system integration, the accreditation and the customer relationship.
The first four weeks. We take the target specification and the existing model, if there is one, and produce a fit assessment on real hardware: operator coverage in the target runtime with every fallback identified, measured timing distribution at the mission input size, memory footprint including activations, sustained-load behavior, and the accuracy of the converted artifact per operating condition against the same evaluation the program will accept. Alongside it we draft the integration contract with both halves filled in as far as the facts allow. That pair is a decision document, and it usually changes the plan.
Weeks five through sixteen. The deliverable model: architecture chosen or adapted for the target, trained, converted, quantized with per-slice accuracy accounting, packaged for the platform's runtime with its preprocessing specification and test vectors, and signed into whatever deployment mechanism the program uses. Plus the on-board monitoring, the local retention design, and the reporting path. All in the prime's environment, repositories and pipeline, written to the program's standards and CDRL formats.
What the prime keeps. All of it. Model source, training and conversion pipelines, evaluation code, preprocessing specification and vectors, monitoring code, packaging scripts, and the customer relationship. Intellectual property in the delivered work is assigned by a present written assignment; any pre-existing tooling we bring is named, carved out and licensed back perpetually so the platform team is never blocked. The exit package is a condition of final payment: repository, build and conversion pipeline, infrastructure as code, configuration, and a runbook, with a handover rehearsal in which the receiving team rebuilds and re-deploys the artifact while our engineers watch. We are named on a bid with a defined technical scope and committed key personnel where the prime wants that, which on a scored proposal generally reads better to an evaluator than an unnamed pool.
How it is priced. The fit assessment and integration contract works as a firm-fixed-price first milestone, because its output is defined and short. The delivered model and monitoring work as fixed-price milestones with acceptance criteria written as measured thresholds on the target hardware, which is the form of acceptance criteria that suits this work exactly. For sustainment across option years, a committed team at a fixed monthly rate fits better. A small fixed-price first increment is how we prefer a new teaming relationship to start.
Bottom line
Edge deployment is an interface problem before it is a compression problem. Name the target hardware, the runtime, the timing budget at a stated percentile, the input and output contracts and the deployment mechanism before the first model is trained, and write the model team's obligations as measured numbers on that target. Then compress with per-slice accuracy accounting rather than a headline figure, define preprocessing once with test vectors that fail a build when the two implementations diverge, decide deliberately whether the model ships inside the build or as a separately signed artifact, and design monitoring that works with no network and reacts on board. Programs that do this integrate in weeks. Programs that discover the target late spend a quarter finding out. The first step is one email with a one-page brief naming the platform, the runtime, the timing budget and the task, and we return a scoped, priced statement of work.
Frequently asked questions
Start from the target rather than the model. Check which operations the runtime supports at which precision and find every silent fallback to the general-purpose processor, since one unsupported layer can cost more than the whole compression effort saves. Then choose or adapt the architecture for that operator set, quantize with calibration data drawn from the deployed distribution, and measure accuracy per operating condition on the target hardware using the same evaluation the program will accept.
It depends on the model, the precision and the calibration data, and the aggregate figure is the wrong thing to watch. Reduced precision usually costs most on rare and hard cases, which are often the ones the mission cares about, so track the per-condition delta rather than the pooled number. Post-training quantization is fast and frequently sufficient. Where it costs too much, quantization-aware training recovers most of the loss at the price of a training cycle.
If the model is expected to be retrained during the fielded life, yes, and the process should be agreed in advance so model updates are handled as data within a pre-approved envelope. Guard against version skew by putting the required preprocessing configuration, the input and output schema versions and the expected runtime version inside the signed artifact, and having the platform refuse to load anything whose declarations do not match. Otherwise a new model quietly receives inputs prepared for the old one.
Compute the monitoring statistics on board: input distribution summaries against the reference recorded at test, output score distributions, abstention and low-confidence rates, timing percentiles and resource use. Hold them in a fixed-size local store with a rule-based sample of interesting cases, including high-uncertainty inputs and anything flagged outside the operating envelope. Report opportunistically when a link exists, summaries first. Most importantly, give the platform a defined on-board response to its own alarms.
Often it is not the hardware or the quantization. It is preprocessing implemented twice, once in the training pipeline and once on the platform, differing in resize interpolation, color space, normalization constants, channel order, padding or the handling of missing values. Fix it structurally: define preprocessing once as a specification with recorded test vectors at a stated numerical tolerance, test both implementations against those vectors in their build pipelines, and fold preprocessing into the model graph where the runtime allows.