The four assumptions GPUs break
Kubernetes was designed around resources that divide cleanly. A CPU is a number you can cut into thousandths, overcommit two to one, and claw back under pressure. Memory is a byte count the kernel enforces for you. Nearly every scheduling, autoscaling and cost behavior in the platform leans on those two properties. A GPU has neither. It arrives as an opaque whole-device count with no overcommit, no fractional request, no kernel-enforced limit on the thing that actually runs out, and a failure surface the control plane cannot observe. Everything that feels wrong about GPUs on Kubernetes comes back to that mismatch, and most of the work is putting those properties back by hand.
None of this is an argument against Kubernetes. If you have several teams sharing expensive hardware, need quota and preemption between them, and want one control plane in front of both your CPU services and your training jobs, Kubernetes is the right substrate and there is not a close second. It is an argument against assuming the cluster you already run will absorb GPUs the way it absorbed your last microservice. It will not. Below is the list of what changes, in the order teams hit it.
You are probably here because
- Multi-GPU jobs sit pending for hours while the cluster reports plenty of free GPUs.
- A node is Ready, nvidia-smi shows every card when you exec onto it, and it advertises zero allocatable GPUs.
- The dashboard reads ninety-five percent utilized and nobody on the team believes the fleet is that busy.
- A distributed job finishes correctly and takes far longer than it should, with no error anywhere to point at.
Changes one, four, five and ten below take these one at a time, and they share a root cause: the scheduler is counting whole opaque devices it cannot subdivide, cannot enforce, and cannot see inside.
Change one: the resource model has no dial
GPUs enter the scheduler as an extended resource. A device plugin runs as a DaemonSet on each node, enumerates the hardware, and reports capacity to the kubelet, which advertises something like nvidia.com/gpu: 8. Pods then ask for whole units of that resource. Three constraints follow immediately, and they are not configurable.
Integers only. There is no 0.5 GPU. A notebook that needs 4 GB of device memory occupies the same schedulable unit as a job saturating 80 GB of HBM. Every sharing scheme later in this article exists to work around that one sentence.
Requests must equal limits. Kubernetes does not allow an extended resource to be requested at one value and limited at another. Every GPU pod is therefore Guaranteed QoS, which is fine, but it also means there is no burst behavior and no soft allocation to reclaim when the node gets busy.
No overcommit and no throttling. A CPU-bound pod that exceeds its share gets throttled by cfs quota and keeps running slowly. There is no equivalent for a GPU. The device is allocated or it is not, and if two processes end up on one device by other means, neither is bounded.
The practical consequence is fragmentation, and it is the largest source of quiet waste in a GPU cluster. On eight-GPU nodes, a fleet that has drifted into an odd allocation pattern will show plenty of free capacity in aggregate while no single node can host a four-GPU job. The scheduler is not broken. It is doing exactly what a bin packer does with indivisible items, and the fix is scheduling policy rather than more hardware.
The second consequence is subtler. Device memory is the resource that actually runs out, and Kubernetes has no idea it exists. A pod's memory limit governs host RAM. Nothing in the cgroup hierarchy bounds HBM. Two containers placed on the same physical device will happily allocate until one of them takes a CUDA out-of-memory error, and the one that dies is usually not the one that misbehaved.
Change two: there is a stack under the scheduler that has to be exactly right
Before a single pod schedules, six things have to line up on every GPU node: a kernel driver, the container toolkit that injects device nodes and libraries into containers, a container runtime class wired to that toolkit, the device plugin that advertises capacity, a labeler that describes what the hardware is, and an exporter that reports telemetry. The NVIDIA GPU Operator packages all of it, along with a MIG manager and a validator pod that refuses to mark the node good until the chain works end to end.
Use the operator. Hand-rolling this is a week of work followed by a year of version skew, and version skew is the single most common cause of the failure that looks like nothing: a node that is Ready, shows GPUs in nvidia-smi when you exec onto it, and advertises zero allocatable nvidia.com/gpu. The scheduler is not hiding anything from you. The plugin never registered, so from the control plane's point of view the node has no GPUs at all.
Two node-level settings are worth deciding deliberately rather than inheriting. The first is labeling. GPU Feature Discovery writes labels such as nvidia.com/gpu.product, nvidia.com/gpu.memory and nvidia.com/gpu.count. Use them in node affinity from day one, even if the fleet is homogeneous today. It will not stay homogeneous, and a job tuned for one architecture that lands on another either crashes on a missing capability or runs at a fraction of the speed with no error.
The second is the kubelet Topology Manager. On a multi-socket node, a GPU, the NIC that feeds it, and the CPU cores driving the data loader all hang off specific NUMA domains. With the default policy the kubelet will cheerfully hand you a GPU on one socket and pinned CPUs on the other, and every byte crosses the interconnect. Setting the policy to single-numa-node forces aligned allocation or an admission failure, which is the outcome you want. An admission failure is visible. A silent thirty percent throughput loss is not.
Where the engineering effort goes — first shared GPU cluster
Our planning split, summing to 100. The install everyone budgets for is the smallest line.
Change three: sharing a device means picking an isolation contract
Because the scheduler only counts whole devices, sharing has to happen below it. There are three mechanisms and they are not interchangeable. They differ on the only question that matters in a shared cluster: what happens to tenant B when tenant A misbehaves.
Time-slicing is a device plugin setting. You declare a replica count and one physical GPU advertises as several schedulable units. The GPU context-switches between them. There is no memory partitioning, no performance isolation, and no fault isolation. It is close to free to turn on and it is the right answer for notebooks, CI, small batch inference and anything where an occupant can tolerate being slowed down or killed by a neighbor.
Multi-Process Service lets kernels from several processes execute concurrently on one device rather than interleaving. Throughput on small kernels improves substantially compared with time-slicing. You can cap each client's share of streaming multiprocessors and pin a device memory limit per client, which gives you a real memory bound, but the processes still share an address space and a fault domain.
Multi-Instance GPU partitions the hardware. On data-center parts from the Ampere generation forward, a device splits into as many as seven instances, each with its own slice of streaming multiprocessors, its own L2 and its own memory with dedicated bandwidth. A crash in one instance does not touch the others. The cost is rigidity: the profiles are fixed sizes, the device has to be drained and reset to change layout, and the MIG manager needs the node cordoned while it does that. You are trading elasticity for a hard boundary.
| Mechanism | Memory bound | Fault isolation | Reconfigure cost | Use it for |
|---|---|---|---|---|
| Whole device | Entire device | Complete | None | Training, large-model inference, anything with a latency target |
| Time-slicing | None enforced | None | Config change and plugin restart | Notebooks, CI, dev, bursty low-stakes inference |
| MPS | Per-client cap available | Weak, shared address space | Daemon restart | Many small concurrent processes owned by one team |
| MIG | Hardware-partitioned | Strong | Cordon, drain, GPU reset | Multi-tenant inference where a neighbor must not be able to hurt you |
| Dynamic Resource Allocation | Driver-defined | Driver-defined | Claim template change | Fleets with mixed devices and structured selection needs |
Dynamic Resource Allocation is the direction the platform is moving. It replaces the count-of-opaque-things model with claims and device classes, so a workload can ask for a device with specific attributes, share a claim between pods, and express constraints the integer counter never could. It went beta in 1.32 and graduated in 1.34. If you are designing a fleet now, know it exists and keep your allocation logic in one place so you can move to it, but do not rebuild a working cluster around it before your drivers and your framework tooling have caught up.
Change four: one pod at a time is the wrong unit for distributed training
The default scheduler places pods independently. That is correct for services and actively harmful for a job whose ranks must all exist before any of them can make progress. Ask for eight pods across two nodes, get six of them placed, and those six sit holding six GPUs, initializing a collective that will never complete, until something kills them. Meanwhile another job holds the complement and is doing the same thing. Both are stuck, both are billing, and neither will move without intervention. This is the standard first outage of a shared training cluster and it usually arrives in week three, when the second team starts using it.
The fix is all-or-nothing admission. Kueue is the option we reach for first: it sits in front of the scheduler with a Workload abstraction, admits a job only when the full shape can be placed, and gives you cluster queues with quota, cohorts that can borrow unused capacity from each other, and preemption rules you write down rather than discover. Volcano is the heavier alternative, a full replacement scheduler with gang plugins, queue fairness and job-level lifecycle. Either is a real improvement over hoping. Neither is optional once more than one team shares the hardware.
Two policies matter as much as the mechanism. The first is queue-based fairness with borrowing: idle capacity should flow to whoever can use it, and come back when the owner returns, which is the only way to run high utilization without making one team's work hostage to another's. The second is topology awareness. On an eight-GPU node the devices are not equidistant. A four-GPU request that straddles two NVLink groups runs measurably slower than one that does not, and across nodes the network fabric has structure too. Placement that respects that structure is worth more than the last few percent of packing efficiency.
Change five: the network decides whether "it works" and "it is fast" are the same thing
Multi-node training moves gradients through a collective library, and that library will pick whatever transport it can find. If the fast path is not configured, or is configured and not reachable, it falls back to TCP over the pod network. The job still runs. It still produces correct results. It runs at a small fraction of the speed, and there is no error anywhere to tell you.
Getting the fast path requires a second network. That means a meta-plugin such as Multus to attach an additional interface, a device plugin that exposes the RDMA-capable hardware, and pod specs that request it. The NVIDIA Network Operator assembles that side of the stack the way the GPU Operator assembles the compute side. Then GPUDirect moves data between device memory and the NIC without staging through host RAM, which is where most of the win comes from.
The verification step is the part teams skip. Run one job with NCCL_DEBUG=INFO and read the initialization output. It tells you which transport was selected and which interfaces were considered. Pin NCCL_SOCKET_IFNAME and NCCL_IB_HCA explicitly rather than letting autodetection choose, because autodetection on a pod with three interfaces frequently picks the management one. Then run a bandwidth test between two pods on different nodes and compare the number to the hardware's rated line rate. If those two numbers disagree by an order of magnitude, you have found the problem before it costs you a month of training time.
Failure modes that arrive disguised as something else
- A container sets NVIDIA_VISIBLE_DEVICES to all and sees every GPU on the node, bypassing the allocation the scheduler made. It looks like a scheduler bug and it is a container configuration hole.
- A job dies with CUDA out-of-memory on a node with free host RAM. Pod memory limits do not bound device memory, so the neighbor that overallocated is not the process that fails.
- Training throughput drops by half with no failed pod. One GPU is thermally throttling or has fallen back to a degraded link, and a synchronous all-reduce runs at the speed of the slowest rank.
- The cluster shows plenty of free GPUs and multi-GPU jobs will not start. Fragmentation, not capacity. Aggregate free count is the wrong metric to alert on.
- A node reports Ready and never receives GPU pods. The device plugin failed to register, usually driver and toolkit version skew after an image update.
- Inference latency spikes every time the autoscaler adds a replica. The new pod passed its readiness probe before the model weights finished loading and started taking traffic cold.
- A dashboard shows ninety-five percent GPU utilization on a cluster nobody believes is busy. The metric counts whether any kernel was resident, not whether the silicon did work.
Change six: checkpoints and datasets are an infrastructure problem, not a training one
Checkpoint size is arithmetic and it surprises people. Mixed-precision training with an Adam-family optimizer carries roughly sixteen bytes per parameter once you count half-precision weights, the two optimizer moments in single precision, and a single-precision master copy. A seven-billion-parameter model is therefore around 110 GB of state, and a seventy-billion-parameter model is around 1.1 TB. Write that to object storage every thirty minutes over a shared node interface and you have built a job that spends a meaningful share of its life not training.
Three changes fix most of it. Write sharded checkpoints, so each rank writes its own slice in parallel instead of gathering everything to rank zero. Write to node-local NVMe first and upload asynchronously, so the training loop blocks on a local write measured in seconds rather than a remote write measured in minutes. And set the interval from the failure rate rather than from habit: on preemptible capacity with a mean uptime of a few hours, checkpointing every ten minutes is cheap insurance, and on stable capacity every thirty to sixty minutes wastes less.
The read path deserves the same attention. A data loader starved by a shared filesystem shows up as low GPU utilization that everyone blames on the model. Measure it directly: run the training loop against synthetic tensors generated in memory and compare the step time to the real one. If synthetic is much faster, the problem is I/O and no amount of GPU tuning will touch it.
Change seven: node lifecycle is measured in minutes, not seconds
A GPU node is slow to become useful. The driver stack installs or verifies, the plugin registers, and then the pod pulls an image that, with a CUDA base and a deep learning framework inside it, commonly runs between five and twenty gigabytes. Five to fifteen minutes from scale-up request to first training step is normal, and every autoscaling assumption carried over from stateless services is wrong by two orders of magnitude.
What we do about it: keep a small warm pool rather than scaling from zero for interactive work, pre-pull the standard images onto the node group so the first pod does not pay for them, keep the training image lean by moving datasets and large assets out of it, and taint GPU nodes so ordinary workloads cannot drift onto expensive hardware. The extended-resource toleration admission controller handles the tolerations automatically for pods that actually request GPUs, which keeps the taint from becoming boilerplate in every manifest.
Separate node groups per device type, always. Mixing device generations inside one autoscaling group means the autoscaler cannot reason about what it is adding, and a pending pod that requires a specific architecture will trigger scale-up of a group that cannot satisfy it, repeatedly.
Decision weights — does this workload belong on Kubernetes
Score each 0 to 10, multiply, sum, divide by 10. Above 60, Kubernetes earns its overhead.
Change eight: the hardware fails in ways the control plane cannot see
A CPU that fails takes the node down and Kubernetes handles it. A GPU has an entire category of partial failures with no equivalent in the platform's model. Uncorrectable memory errors retire pages and shrink usable capacity. Driver-level Xid events record faults that range from a bad user kernel to hardware that needs replacement. Devices drop off the bus and stop appearing in enumeration while the node stays perfectly Ready. Thermal and power limits reduce clocks without producing any event at all.
None of that reaches the scheduler on its own. You have to build the bridge: run DCGM health checks on a schedule, feed the results into Node Problem Detector so failures become node conditions, and run a small controller that cordons and drains on a condition, opens a ticket, and reboots or replaces on the classes of fault that warrant it. Without that loop, a bad device silently accumulates failed jobs and the pattern only becomes visible when someone notices the same node in three different postmortems.
Straggler detection deserves its own mention because it is the failure nobody instruments. In a synchronous distributed job, one slow rank sets the pace for all of them, and the job completes successfully at reduced speed. Emit per-rank step time, alert on the spread rather than the mean, and you catch the degraded device the same day instead of at the end of the run.
Send it over and we will tell you what we would change.
Email your GPU node shapes, one kubectl describe node from a GPU node, and the NCCL_DEBUG=INFO initialization log from a job that felt slow to contact@precisionfederal.com. You get back a short written note naming the three things we would change and why. One business day. No charge, no meeting, no deck.
contact@precisionfederal.comChange nine: inference is a different tenant wearing the same hardware
Training and serving have opposite characteristics and they end up in the same cluster. Training is throughput-bound, tolerant of preemption if it checkpoints, and happy to queue. Serving is latency-bound, intolerant of cold starts, and needs headroom that looks like waste on a utilization dashboard. Running both without separating them produces the worst of each.
Three specifics. First, autoscale on the right signal. Requests per second and CPU utilization are both nearly meaningless for a GPU inference service. Queue depth, queue wait time, and batch fullness track the thing you care about. Second, account for weight load in the scaling math. A 140 GB set of half-precision weights read from local NVMe at a few gigabytes per second takes tens of seconds; the same read from object storage over a shared interface takes minutes. Scale ahead of demand, cache weights on the node, and make the readiness probe wait for a real inference rather than a socket bind. Third, use priority classes and preemption so a batch job can absorb spare capacity and be evicted the moment serving needs it back. That is the mechanism that lets you run high utilization and still keep latency headroom.
Change ten: the utilization number on your dashboard is probably lying
The metric everyone puts on the wall first reports the fraction of a sampling window during which at least one kernel was resident on the device. A kernel that uses two percent of the available parallelism and runs continuously reads as one hundred percent utilized. Teams look at that number, conclude the fleet is saturated, and buy more hardware to solve a problem that was never a capacity problem.
Read the profiling counters instead. Streaming multiprocessor activity and occupancy tell you how much of the chip is engaged. Tensor pipe activity tells you whether the units you are actually paying for are doing anything. Framebuffer usage tells you how much device memory is committed. Put those next to the naive number and the picture usually changes.
Signal quality — how well each metric answers "is this GPU doing work"
Our rating of how much each signal tells you. The default counter is the one on most dashboards.
The number that governs spend is simpler than any of these: GPU-hours allocated minus GPU-hours doing work, attributed to a team. Idle hardware costs exactly what busy hardware costs. At ten dollars per GPU-hour, an eight-GPU node held overnight by a finished notebook burns close to a thousand dollars while producing nothing, and the only reason anyone finds it is that somebody built the report. Publish allocated-and-idle by team weekly, set queue quotas so idle allocations expire, and the behavior changes without anyone needing to police it.
When Kubernetes is right, and when it is overhead
| Situation | Verdict | Why |
|---|---|---|
| Several teams, mixed training and inference, shared fleet | Kubernetes | Quota, preemption and multi-tenancy are the hard part, and this is what the platform is for |
| One team, one long-running training job, dedicated hardware | Something simpler | A batch scheduler or a managed training service does this with a fraction of the operational surface |
| Inference services already deployed on Kubernetes | Kubernetes | You are adding a node group, not a platform. Keep the rollout, service mesh and observability you already run |
| Bursty experimentation, no steady baseline | Managed or rented | Cluster overhead is fixed and the workload is not. Rent until there is a baseline to size against |
| Regulated data with a controlled boundary and on-premises hardware | Kubernetes | One control plane across owned and rented capacity, and the boundary is yours to define |
| No platform engineering ownership available | Not yet | Everything in this article is a system somebody has to run. Unowned, it degrades into an expensive pet cluster |
Before the first GPU node joins a shared cluster
- GPU Operator installed, with the validator passing and driver and toolkit versions pinned
- Node taints in place and the extended-resource toleration admission controller enabled
- Separate node groups per device type, with node affinity written against hardware labels
- Topology Manager set to single-numa-node and the admission failures understood
- Gang-scheduling admission in front of the scheduler before the second team arrives
- Queues with quota, borrowing between cohorts, and written preemption rules
- A sharing mechanism chosen per workload class, with the isolation contract documented
- Fast-path networking verified with a real bandwidth test, not assumed from the pod spec
- Sharded checkpoints to node-local storage with asynchronous upload
- DCGM health checks feeding node conditions, with automated cordon and drain
- Per-rank step time exported so a straggler is visible the same day
- Allocated-and-idle GPU-hours reported by team, on a schedule, to the people who own the budget
A staged way to get there
Rollout sequence
Telemetry lands second on purpose. Every decision after it, from how much to share a device to which node group to grow, is a measurement question, and teams that install the sharing mechanism before the counters end up tuning against a number that does not mean what they think it means. The health loop wants a real test too. Plant a fault, confirm the node cordons, confirm the job requeues somewhere healthy. A remediation controller that has never fired is a remediation controller you have no evidence works.
Bottom line
GPUs on Kubernetes are not hard because the hardware is exotic. They are hard because the resource is indivisible, unenforced, and quietly failure-prone, and the platform was designed for resources that are none of those things. The work is putting back what the scheduler assumed: admission that respects a job's shape, isolation that holds when a neighbor misbehaves, a network path you have measured rather than configured, health signals the control plane can act on, and one honest utilization number. Do those and Kubernetes is a good place to run this. Skip them and you have bought fast hardware and built a slow cluster around it.
Frequently asked questions
Not through the scheduler, which only allocates whole devices. Sharing happens below it, through time-slicing, Multi-Process Service, or hardware partitioning with MIG. Pick based on isolation rather than on memory size: time-slicing gives you none, MPS gives you a memory cap in a shared fault domain, and MIG gives you a hardware boundary at the cost of a drain and reset to change layout.
Fragmentation. Free capacity is spread across nodes in pieces too small for the request, and the default scheduler will not consolidate. Aggregate free count is the wrong thing to alert on. Alert on largest contiguous placement available per node group, and add gang admission with queues so partial placements stop occurring in the first place.
You need all-or-nothing admission. Kueue provides it in front of the default scheduler with quota and cohort borrowing, and Volcano provides it as a replacement scheduler with gang plugins and queue fairness. Without one of them, a job that gets partially placed holds hardware while it waits for ranks that will never arrive, and two such jobs can deadlock each other indefinitely.
Two likely causes. The collective library fell back to TCP over the pod network instead of the fast path, which you confirm by reading the initialization output with debug logging on and comparing measured bandwidth to line rate. Or one device is throttling and setting the pace for every rank in a synchronous all-reduce, which you catch by exporting per-rank step time and alerting on the spread.
Not the default device utilization percentage, which reports only whether a kernel was resident and reads near one hundred percent for work using a small fraction of the chip. Use streaming multiprocessor occupancy and tensor pipe activity for whether the silicon is engaged, framebuffer usage for memory pressure, and allocated-but-idle GPU-hours by team for what it is costing you.