Spot and preemptible instances for ML workloads

What you are actually buying

A spot instance is the same machine as an on-demand instance, in the same rack, on the same network, with one clause changed: the provider may take it back at short notice and owes you nothing when it does. That clause is worth somewhere between half and ninety percent off list price. Every other question about spot capacity is downstream of one piece of arithmetic, which is whether the cost of losing a machine mid-run is smaller than the discount. Most teams never run that arithmetic. They either avoid reclaimable capacity because someone once called it unreliable, and pay three or four times too much for work that does not care, or they move a multi-day training job onto it, lose eleven hours of progress on the first Friday night, and write it off for a year.

The reason this is worth getting right is that machine learning work has an unusual cost shape. A large share of the compute bill sits in jobs that are batch, restartable in principle, and tolerant of a few minutes of lost progress: feature builds, embedding backfills, hyperparameter sweeps, offline scoring, evaluation runs, nightly retraining. That is exactly the profile reclaimable capacity is priced for. The gap between what a team pays and what it could pay is usually not a negotiation problem or a rightsizing problem. It is an engineering problem that nobody has been given two weeks to solve.

Three providers, three notice windows

The mechanics differ enough that a design assumption carried from one cloud to another will fail quietly. The important differences are the length of the notice, how you receive it, and what happens to the machine and its disks afterward.

Provider	Notice	How you receive it	What to know
AWS EC2 Spot	Two minutes, best effort	Instance metadata endpoint, plus an earlier and advisory rebalance recommendation	Prices have been set by AWS and adjusted gradually since 2017; there is no bidding war to win. Your max price defaults to the on-demand rate. Instances can be set to terminate, stop, or hibernate.
Google Cloud Spot VMs	About thirty seconds	ACPI soft-off signal to the guest; a shutdown script gets the window	Spot VMs replaced preemptible VMs and dropped the hard 24-hour lifetime the older product had. GPUs and local SSDs are billed at spot rates too when attached.
Azure Spot VMs	Thirty seconds	Scheduled Events endpoint, which you poll	Eviction policy is a choice: deallocate keeps the VM object and its disks, delete removes it. Eviction can be triggered by capacity or by your max price.

Design for thirty seconds everywhere. The two-minute window on AWS is a ceiling and not a promise, the rebalance recommendation is advisory and sometimes arrives with the termination notice rather than well ahead of it, and any handler that only works with a full two minutes will fail the first time it runs on a different cloud or a different day. Thirty seconds is enough to stop accepting work, flush a small buffer, release a lease, and write a pointer. It is not enough to serialize forty gigabytes of optimizer state, and a design that assumes otherwise is not a design.

The discount on the pricing page is not the discount you get

Providers advertise up to ninety percent. That ceiling describes the deepest, least wanted pools, usually an older instance family in a region nobody is fighting over. The realized number depends entirely on which pool you land in, and it moves. Three things shape it in practice.

The newest accelerators are barely on the menu. When a GPU generation is capacity constrained, the spot pool for it is thin, expensive relative to list, and reclaimed aggressively. The deep discounts live one or two generations back. If your training code only runs on the newest part, spot is not a cost strategy, it is a lottery ticket.

Spot does not stack with commitments. Savings Plans, Reserved Instances, and committed use discounts do not apply to spot capacity. If your steady-state baseline is already covered by a commitment, spot only helps for the work above that line, and buying more commitment and running more spot are competing uses of the same budget. Decide the baseline first, then spot the burst.

Your realized rate is the blended rate. The number that belongs in a budget is not the spot price. It is total dollars divided by useful work completed, which includes hours spent on compute that was later thrown away, hours spent re-reading data after a restart, and hours the job sat in a queue because the pool had no capacity. That last one never appears on the invoice, which is why it is the one that surprises people.

The only question that matters: what does one interruption destroy

Sort every job by the cost of losing a machine halfway through. That sort, not the price sheet, tells you what to move.

Stateless batch transform. Scoring a partition, resizing images, generating embeddings for a shard. An interruption costs one partition of work, the partition gets requeued, nothing else notices. This is the ideal case and it is usually the largest line on the bill.

Hyperparameter search. Each trial is independent. Interruption costs one trial, or less if trials checkpoint. Search frameworks already handle trial failure because trials fail for ordinary reasons all the time.

Single-node fine-tuning. Cost is bounded by the checkpoint interval. Manageable, and the arithmetic below tells you exactly what to set the interval to.

Multi-node distributed training. The hard case. Losing one worker in a gang-scheduled job stalls all of them, and the loss is the whole cluster's progress since the last checkpoint, not one node's.

Online inference. The risk is not lost work, it is lost capacity. A reclaim removes throughput at whatever moment the provider chooses, which may be the moment your traffic peaks, because both are correlated with regional demand.

Anything holding the only copy of something. Databases, queue brokers, coordination services, the rendezvous store for your own training job. Never. The savings are trivial relative to the failure.

Checkpoint arithmetic, and it has a closed form

Teams argue about checkpoint frequency by instinct. There is no need. The optimal interval for a periodically checkpointed job on unreliable hardware has been known since Young published it in 1974 and Daly refined it in 2006, and the first-order result is short enough to write on a whiteboard.

Let C be the time it takes to write one checkpoint, and M the mean time between interruptions for the machine or cluster running the job. The interval that minimizes total wasted time is approximately the square root of two times C times M. The fraction of your compute lost to checkpointing and rework at that interval is approximately the square root of two C divided by M.

Work an example. Checkpoint writes take 90 seconds, and the pool you are in gives you a mean of eight hours between reclaims. Two times 90 times 28,800 is 5,184,000, and the square root is roughly 2,280 seconds. Checkpoint every 38 minutes. Overhead is the square root of 180 over 28,800, which is about 7.9 percent. A nominal 65 percent discount survives that easily.

Now change one input. Checkpoint writes take five minutes because the job serializes full optimizer state synchronously to a single object-store key, and the pool is hot enough to give you 90 minutes between reclaims. The optimal interval drops to about 30 minutes and overhead climbs to 33 percent. A third of your compute is now spent writing checkpoints and redoing work, and the discount has been eaten. Notice which input moved the answer: not the interruption rate, the checkpoint cost.

That is the practical lesson. Make checkpoints cheap before you make them frequent. Write shards in parallel from every rank instead of gathering to rank zero. Use asynchronous checkpointing so the copy to host memory blocks training briefly and the upload happens in the background. Save only what resume requires: model weights, optimizer state, learning-rate schedule position, RNG state, and the data loader position. Compress where the GPU is idle anyway. Keep a rolling window of two or three checkpoints rather than one, because the checkpoint written during a reclaim is the one most likely to be truncated.

The data loader position is the field teams forget, and it fails silently. Resume without it and the job restarts the epoch from sample zero, retraining on data it has already seen while the loss curve looks entirely normal. Nothing errors. You find out months later, if at all.

Put restore in continuous integration. A small job trains for two hundred steps, checkpoints, gets killed, restores, trains two hundred more, and asserts that the loss trajectory matches an uninterrupted run within tolerance. It runs on every merge to the training code. This one test catches version skew in the optimizer state, missing RNG seeding, silently dropped scheduler state, and the data loader problem above, and it costs a few minutes of CI time per merge.

Why a 64-GPU job is not eight 8-GPU jobs

Gang-scheduled training is all-or-nothing. If any worker disappears, the collective stalls and the job either dies or has to re-form. So the interruption rate that matters is not the per-node rate, it is the rate for the union of nodes, and it compounds fast.

Take a per-node hourly reclaim hazard of five percent, which is a plausible figure for a moderately contended pool. The chance a one-hour window completes with no node reclaimed is 0.95 raised to the number of nodes.

At 32 nodes the effective mean time between interruptions is about 37 minutes. Feed that back into the checkpoint formula with a 90-second write and you get an optimal interval near nine minutes and overhead above 15 percent, before counting the time to re-acquire replacement capacity and re-form the process group. At 64 nodes a synchronous job is essentially never running for an hour uninterrupted.

There are three honest responses. Run the synchronous core on capacity you control and burst the rest. Make the job elastic, so the world size can shrink and grow and training continues at reduced throughput rather than stopping, which PyTorch elastic launch, Ray Train, and similar frameworks support if your code tolerates a changing world size. Or split the work so that gang scheduling is not required, which is often possible for search and evaluation and rarely possible for a single large training run. What does not work is running a 64-way synchronous job on a single spot pool and hoping.

Diversification is the largest single lever

A capacity pool is the combination of instance type, size, availability zone, and region. Reclaims happen because a pool runs short. If your fleet can only live in one pool, you have taken a single point of failure and attached your entire compute budget to it.

Six to ten interchangeable pools changes the character of the problem. When one tightens, the fleet drifts into the others and the job keeps running. Every provider gives you the controls: AWS has allocation strategies on EC2 Fleet and Auto Scaling groups, Google Cloud lets a managed instance group or GKE node pool span machine families, Azure has a priority mix in scale sets that blends spot and on-demand under one autoscaler.

Diversification has a code consequence people underestimate. Your job must not care which GPU it lands on. That means no hard-coded device names, batch sizes that adapt to available memory, no assumptions about interconnect topology, and a container image that carries drivers and kernels covering every family in the set. Teams that skip this end up with a beautifully diversified fleet and a job that only runs on one of the instance types in it.

What to do inside the notice window

Thirty seconds is a budget. Spend it on releasing responsibility, not on saving state, because state should already be saved. A handler that fits comfortably does these things in order: stop pulling new work, mark the node unschedulable, drain from the load balancer, negative-acknowledge or extend the visibility timeout on in-flight messages so another worker picks them up immediately rather than after a timeout expires, release any distributed lease, flush the small local buffer that has accumulated since the last checkpoint, and emit a metric so you can count reclaims later.

On Kubernetes, the plumbing has specific traps. Karpenter watches an interruption queue and starts a replacement node when the notice arrives; on managed node groups the AWS Node Termination Handler does the equivalent; GKE graceful node shutdown gives pods a window on Spot VMs; Azure requires something to poll Scheduled Events. Whichever you use, terminationGracePeriodSeconds must be shorter than the notice window, not longer. Setting it to 120 on a cloud that gives you 30 does not buy you 120 seconds. It means the kubelet is still politely waiting when the machine vanishes, and your preStop hook never finishes.

The related misconception is worth naming plainly. PodDisruptionBudget constrains voluntary evictions, which is drain and upgrade and rebalance. A capacity reclaim is involuntary. The budget will not slow it, and a team that believes otherwise has an availability plan resting on a mechanism that does not apply. Availability on reclaimable capacity comes from replicas spread across pools and zones, from fast replacement, and from an on-demand floor. It does not come from a policy object.

Keep the coordination layer off spot. The rendezvous store for elastic training, the metadata database, the scheduler, the queue broker: these are cheap relative to the fleet and expensive to lose. Running the control plane on the same reclaimable capacity as the workers is how a routine reclaim turns into an outage that takes an afternoon to understand.

Serving traffic on reclaimable capacity

Online inference can use spot, with two conditions. The first is a floor of on-demand or committed capacity sized to the traffic you must never drop. The second is that replacement capacity comes online fast enough to matter, and for model serving that is almost always a cold-start problem rather than a scheduling problem. If a new replica needs twelve minutes to pull a container image and load weights from object storage, a reclaim is an outage regardless of how good your autoscaler is. If it needs ninety seconds because the image is pre-pulled onto the node and the weights sit on a shared read-only volume or in a local cache, spot becomes usable.

The failure mode to design against is correlation. Reclaims within a pool are not independent events. The provider needs capacity back in a zone and takes it from many instances at once, so a fleet that is one hundred percent spot in one pool does not degrade gracefully. It goes to zero. Spread across pools and zones, keep the floor on committed capacity, and treat spot replicas as capacity above the line rather than capacity you are counting on.

The easier and larger win is asynchronous inference. Put requests on a queue, let workers on spot drain it, and accept a response-time budget measured in minutes. Interruption then costs one in-flight message, which the queue redelivers. A great deal of what gets built as synchronous serving does not need to be, and moving it behind a queue both cuts the bill and removes the reclaim risk in one change.

Workload	Spot share we would run	What makes it safe	What breaks it
Batch scoring and backfills	90 to 100 percent	Partitioned work, idempotent writes, requeue on failure	Non-idempotent side effects, partitions too large to redo
Hyperparameter search	80 to 100 percent	Trial-level checkpointing, scheduler that reissues failed trials	Trials that only report at the end
Single-node training	70 to 90 percent	Checkpoint at the Young interval, tested restore, six or more pools	Slow synchronous checkpoint writes
Multi-node training	0 to 50 percent	Elastic world size, sharded async checkpoints, on-demand core	Fixed world size, one pool, gang scheduling
Async inference	80 to 100 percent	Queue with redelivery, visibility timeouts tuned to task length	Timeouts longer than the notice window
Online inference	0 to 60 percent above the floor	Committed floor, sub-two-minute replica start, spread across pools	Cold start measured in minutes, single pool

The break-even model

Write it down before you build anything. Annual savings equal compute hours times the on-demand rate times the realized discount fraction. Against that, subtract the compute you throw away, which the checkpoint formula gives you as a percentage, and subtract the engineering. Making a pipeline properly preemption-safe is two to four engineer-weeks for a team that already has containerized jobs and object-store checkpoints, and more if state currently lives on instance disks. Then there is ongoing cost: someone maintains the handlers, watches the reclaim metrics, and refreshes the pool list as instance families change.

The rough shape that falls out is this. Below a few hundred accelerator-hours a month, the engineering does not pay unless your framework gives you most of it for free, in which case take it. In the low thousands of hours a month it pays clearly, and the two to four weeks return inside the first quarter. Above that it is not optional, because a team spending six figures a year on interruptible batch work at on-demand rates is buying insurance against a risk it could engineer away in a month.

One cost never shows on the invoice: schedule. Spot capacity you cannot get is a delay, not a charge. If a model has to be retrained before a release, the run belongs on capacity you control, and the savings you gave up are the price of a date you can commit to. Reserve the discount for work whose deadline is soft.

A two-week path to running on it

Step five is the one that gets skipped and the one that finds the bugs. Terminate a worker in the middle of a real run, on purpose, during business hours, and watch what happens. If the job resumes cleanly, requeues the lost partition, and reaches the same result, the design holds. If it hangs waiting on a collective, writes a truncated checkpoint, or silently restarts the epoch, you learned that on a Tuesday afternoon instead of at 3 a.m. on a Sunday.

Mistakes we see repeatedly

• Choosing capacity by lowest price. The cheapest pool is cheapest because nobody wants it, which is also why it is the first one drained.
• Grace period longer than the notice window. A 120-second termination grace on a 30-second notice means the shutdown hook never completes.
• Checkpoints on the instance's local disk. The disk goes away with the instance. Local disk is a staging area, not a destination.
• Resuming without the data loader position. Silent re-training on the same samples, no error, no alert, wrong model.
• One instance type in one zone. Every other lever is secondary to this one, and it is the easiest to fix.
• Treating reclaims as independent. They arrive in bursts, because the provider needs capacity back all at once.
• The coordination service on spot. The rendezvous store, the scheduler, and the queue broker are cheap. Losing them is not.
• Reporting savings against the spot price. Report cost per unit of completed work, or the number is fiction.

Checklist before a job moves to reclaimable capacity

• The job checkpoints to object storage, and restore is tested in CI
• Checkpoint write time is measured, and the interval is set from it
• Six or more interchangeable instance pools are configured and the image runs on all of them
• Allocation strategy weighs capacity, not price alone
• Termination grace period is shorter than the shortest notice window in use
• In-flight work is leased, acknowledged, or made idempotent so a reclaim redelivers it
• Reclaims and restarts are counted as metrics, with an alert on a rate change
• A deliberate node kill has been run against a live job and the result verified
• Coordination and state services run on capacity that cannot be reclaimed

Bottom line

Reclaimable capacity is a good deal for work that can be interrupted and a bad deal for everything else, and the boundary between the two is set by engineering you control. Measure what one interruption destroys. Make the checkpoint cheap, then set the interval from the formula instead of from instinct. Spread across pools until no single one can stop you. Keep an on-demand floor under anything serving users, and keep the control plane off spot entirely. Do that and the discount is real and durable. Skip it and you will pay for the compute twice, once when it runs and once when you redo it.

Frequently asked questions