Compute2026-10-16

The Trade-offs of GPU Sharing — Choosing Among MIG, MPS, and Time-Slicing by Isolation, Interference, and Fragmentation

#gpu-sharing#mig#mps#time-slicing#kubernetes-dra

An inference service has a whole GPU to itself, yet outside peak hours most of its SMs (Streaming Multiprocessors) sit idle. An operator places a batch job on the same GPU, and both pods schedule normally, but the inference service's tail latency begins to creep up. The two jobs ran on different SMs, yet they shared the same L2 cache and memory bandwidth, and neither the scheduler nor the monitoring metrics showed any sign of that contention. The utilization gained by sharing a GPU is paid for in places like this, where no dashboard looks.

This post examines how much isolation each multi-tenant GPU sharing technique gives up and how much utilization it gains. It first covers the trend that made sharing a design problem and three criteria for choosing a technique, then explains how MIG, MPS and time-slicing, vGPU, and Kubernetes device allocation work. It closes with problems that remain open, such as measuring interference and overcommitting memory.

More GPUs, Stagnant Utilization

GPUs are allocated one whole device at a time, while the resources a workload actually uses are often smaller. In MLaaS in the Wild, presented at NSDI 2022, researchers at HKUST and elsewhere analyzed two months of training and inference job traces submitted by more than 1,300 users to a production cluster of 6,742 GPUs. Low GPU utilization and long queueing delays stood out as the key problems, and most task instances requested less than one GPU. Understanding GPU Resource Interference One Level Deeper, from researchers at ETH Zurich and the University of Toronto, starts from the same observation that even resource-intensive AI applications leave GPUs heavily underutilized. When the allocation unit is larger than what workloads use, idle capacity grows as GPUs are added.

Scheduler design widens the gap as well. In Tiresias, presented at NSDI 2019, researchers at the University of Michigan and elsewhere named three difficulties in managing distributed-training clusters: the all-or-nothing nature of jobs that must receive every requested GPU at once, unpredictable training time, and the inflexibility of GPU sharing that relies on preemption and resumption. Tiresias orders jobs by a two-dimensional priority combining how many GPUs a job has used and for how long, but the unit of sharing is time, and each GPU is still held by one job. In the Kubernetes device plugin framework, extended resources are integers only, cannot be overcommitted, and are not shared between containers. Because the scheduler does not split a device on its own, the sharing model has to be settled when the cluster is designed.

Three Criteria for Choosing a Sharing Technique

The first criterion is whether a workload is memory-bound or compute-bound, because partitioning SMs does not remove contention on the memory path. In the interference study, two kernels ran on disjoint SM sets carved out with CUDA Green Contexts, and contention for L2 cache capacity and for L2 and global memory bandwidth remained even so. Two copy kernels on separate SM sets slowed each other down by more than 2×, depending on input size. The size of that slowdown is tied to the L2 capacity and memory configuration of the GPU used, but the structure carries across generations: partitioning SMs does not partition L2 or the memory path. Sizing a slice has its own trap. A kernel with 6.25% achieved occupancy suffered an 8.57× latency increase when restricted to that fraction of SMs. Sizing a slice from a profiler's occupancy figure alone underestimates what the workload needs.

The second criterion is how much isolation tenants require, and the third is the risk of fragmentation. Among the sharing mechanisms, only MIG provides memory protection, SM performance isolation, memory bandwidth QoS, and error isolation together; CUDA streams and MPS have no error isolation. The practical question for isolation is therefore whether one tenant's fatal fault may terminate another tenant's process. Fragmentation is the cost running the other way. The more a GPU is cut into fixed-size slices, the stronger the isolation, but the more slices are left that no request fits. A workload that competes for memory bandwidth and also needs isolation calls for hardware partitioning that divides even the memory controllers, but fragmentation remains. Choosing software sharing to avoid fragmentation brings interference and fault propagation instead. Picking a familiar technique without weighing the three criteria lets interference accumulate without anyone knowing what was given up.

Hardware Partitioning: MIG's Fixed Profiles and Fragmentation

MIG (Multi-Instance GPU) partitions a GPU's memory path and SMs in hardware. Its basic allocation units are the memory slice, roughly one eighth of total memory capacity and bandwidth including the corresponding memory controllers and cache, and the SM slice, roughly one seventh of the SMs in MIG mode. One of each makes a GPU slice, and GPU slices combined with engines such as copy engines form a GPU instance (GI). A compute instance (CI) inside a GI owns a subset of the parent GI's SM slices but shares memory and engines with the other CIs in that GI. The boundary for memory bandwidth QoS and error isolation is thus the GI, and a CI only divides SMs within a GI.

Fragmentation is visible directly in the profile table. Profiles differ by GPU model; the Blackwell-generation B200 (180GB) entry in the Supported MIG Profiles table serves as an example. 1g.23gb has 1/8 of memory and 1/7 of SMs, with up to 7 instances; 1g.45gb has 2/8 of memory and 1/7 of SMs, up to 4; and 2g.45gb has 2/8 of memory and 2/7 of SMs, up to 3. 3g.90gb has 4/8 of memory and 3/7 of SMs, up to 2, 4g.90gb has 4/8 of memory and 4/7 of SMs, and 7g.180gb takes the whole GPU.

The leftover resources in these combinations come from the allocation units of 1/8 memory and 1/7 SM. Three 2g.45gb instances leave 2/8 of memory and 1/7 of SMs outside any instance, and seven 1g.23gb instances leave 1/8 of memory. The 1g.45gb profile is aimed at memory-bound workloads, but four of them consume all the memory while leaving 3/7 of the SMs unused. Placement creates fragmentation too. Profiles must be placed so they do not overlap, and a GI's physical position determines which GIs can sit beside it, so depending on where the free slices are, a new instance may not fit. MIG delivers the strongest isolation, in short, at the price of fitting workloads to a handful of fixed profiles, and whatever does not fit is left as fragmentation.

Compute Sharing: Fault Isolation Under MPS and Time-Slicing

MPS (Multi-Process Service) lets work from multiple processes run on the same GPU at the same time. Without MPS, each process receives time slices of the whole GPU in turn, and work from different contexts cannot run concurrently. Under MPS, a control daemon (nvidia-cuda-mps-control) manages the server's lifecycle, and the server (nvidia-cuda-mps-server) owns the GPU connection and scheduling resources its clients share. Since Volta, each client has its own GPU address space and submits work directly to the GPU. Resource limits come from CUDA_MPS_ACTIVE_THREAD_PERCENTAGE, which caps the share of SMs, and CUDA_MPS_PINNED_DEVICE_MEM_LIMIT, which caps device memory. The percentage, however, reserves no dedicated resources, so under contention no client is guaranteed a minimum amount of compute, and kernels from different clients may run on the same SM.

On Ampere and later GPUs, static SM partitioning closes that gap. When an operator explicitly assigns the SMs each client may use, the clients gain spatial isolation from one another. The partitioning unit, the chunk, is 8 SMs on discrete GPUs of Hopper and newer, and on earlier generations it is 4 SMs under Legacy MPS v2 and 2 SMs under MPS v3. MPS v3 does not automatically distribute SMs left over after partitioning. MPS can now dedicate SMs, then, but the fixed unit strands some SMs, and isolation of L2 and memory bandwidth remains outside what it guarantees.

Fault isolation is weaker still. One client's fatal fault is reported to every client sharing that GPU, without indicating which client caused it. The server enters a FAULT state and rejects new requests with CUDA_ERROR_MPS_SERVER_NOT_READY, returning to ACTIVE only after all affected clients have exited. With time-slicing enabled, NVIDIA/k8s-device-plugin advertises one GPU as several replicas and interleaves work through CUDA time-slicing. Workloads that receive replicas of the same GPU can each access all of its memory and run in the same fault domain, so if one crashes, they all do. Requesting more than one replica does not guarantee a proportional amount of compute. Software sharing creates almost no fragmentation, but every tenant on the device shares the interference and the fault domain.

Virtualization and Scheduling: vGPU, Device Plugins, and DRA

vGPU divides a GPU among virtual machines. Isolation between those VMs comes from the CPU's virtualization extensions, which hand a guest's sensitive instructions to the hypervisor through VM exits (see hardware virtualization and KVM). Time-sliced vGPUs divide processing cycles under one of three schedulers. The default, best effort, lets a vGPU use cycles others leave idle, while equal share splits cycles evenly among running vGPUs, so each one's share changes as vGPUs come and go. Fixed share gives each vGPU a fixed share so that its performance does not shift when others start or stop. The latter two enforce strict round-robin, and with eight or fewer vGPUs the default time slice is 2 ms. On a MIG-enabled GPU, the scheduler can be set independently for each GI, layering time-sliced vGPUs on top of MIG. vGPU is a layer that stacks hardware and time partitioning, and the choice of scheduler decides whether utilization or predictability comes first.

In Kubernetes, GPUs reach pods through device plugins. A plugin registers a ResourceName such as nvidia.com/gpu with the kubelet's Registration gRPC service, reports the device list and state changes through ListAndWatch, and exposes devices to a container through Allocate when the container is created. MIG instances are advertised as separate resources such as nvidia.com/mig-3g.90gb, and under time-slicing or MPS one GPU is advertised as several, but either way the scheduler sees only integer counts. Hardware placement constraints and actual usage therefore never enter the bin-packing decision. In Beware of Fragmentation, presented at USENIX ATC 2023, researchers at HKUST and elsewhere showed what follows. Allowing tasks to request fractions of a GPU left hundreds of GPUs unallocatable across a large cluster, and classic packing heuristics handled this poorly. FGD (Fragmentation Gradient Descent), which the researchers implemented as a Kubernetes scheduler, defines expected fragmentation over the workload distribution as a statistical measure and places each task where that measure grows least. In an emulation of more than 6,200 GPUs driven by production traces, FGD cut unallocated GPUs by up to 49% compared with existing packing schedulers.

Kubernetes is narrowing this gap with DRA (Dynamic Resource Allocation), stable since v1.35. Drivers publish devices and their attributes through ResourceSlices, and workloads request devices through ResourceClaims that reference a DeviceClass, filtering on attributes with CEL expressions. KEP-4815, DRA Partitionable Devices, beta since v1.36, declares resources such as a GPU's memory slices as SharedCounters, and each partition draws on those counters through ConsumesCounters. Two 1g partitions on different memory slices can therefore be allocated together, while a 1g and a 2g partition that compete for the same memory slice cannot. Only partitions listed in the ResourceSlice can be allocated, however, so arbitrary slicing is out of scope. DRA makes hardware placement constraints visible to the scheduler, but deciding which sizes to carve out in which positions still falls to drivers and operators.

Interference Measurement and Memory Overcommit

Deploying these techniques on a real cluster leaves several problems open.

First, measuring interference between tenants. Because the profiler relies on deterministic kernel replay, co-located kernels are hard to profile, and the interference study proposes a kernel-level interference estimator only as a first step. Under MIG, profiling of shared resources is not supported, and with time-slicing enabled, DCGM-Exporter cannot associate metrics with containers. Tenants experience interference, but no instrumentation path attributes its cause per tenant. In Predictable LLM Serving on GPU Clusters, researchers at Harvard, Northeastern, and UC San Diego exposed one more gap in measurement. MIG isolates compute and HBM but not the shared PCIe path, and in some cloud environments the PCIe topology is opaque, so contention must be inferred from counters.

Second, memory overcommit and QoS guarantees. Kubernetes extended resources cannot be overcommitted, yet each time-slicing replica can access all of the GPU's memory, so the scheduler's integer accounting and actual memory use come apart. KEP-5075, DRA Consumable Capacity, lets separate claims take shares of one device's capacity, but pods referencing the same request share resources without any isolation guarantees, and burst handling is left to individual drivers. On the runtime side, MPS v3 memory partitioning, available from CUDA 13.4 on devices without MIG, sets a hard limit and a soft limit for each cgroup v2 cgroup. An allocation beyond the hard limit fails with CUDA_ERROR_OUT_OF_MEMORY. A client above its soft limit merely becomes an eviction candidate, and MPS force-terminates such clients only under real memory pressure. Because this handling is reactive, the cost of memory overcommit is paid by terminating the process of a tenant that exceeded its share. As long as capacity accounting in the scheduler and enforcement at runtime live in different layers, QoS remains a promise made at scheduling time.

Third, repartitioning as workloads change. MIG Manager requires that no user workloads be running on the GPUs being configured, so before applying a new configuration it stops all GPU pods, including the device plugin, and restarts them once the change is applied. Changing a single profile becomes an operation that halts every GPU workload on that node. The researchers who studied PCIe contention acknowledge the same limitation. A profile change can require briefly pausing the tenant and reloading model state, and bursty workloads may notice that short disruption. The Kubernetes scheduler also does not support preemption for DRA resources, so even a high-priority pod must wait for a device to free up. Where the cost of re-carving partitions as load shifts stays below the benefit of reduced fragmentation has yet to be worked out.

Summary

The gap between the GPU allocation unit and actual usage shows up repeatedly in production traces and interference studies, and schedulers that treat GPUs only as integer extended resources have widened it. MIG provides memory bandwidth QoS and error isolation but leaves fragmentation in its fixed-profile combinations. MPS can now divide SMs and memory capacity, yet it does not isolate L2 or memory bandwidth and propagates fatal faults to every client, while time-slicing divides time without isolation. FGD and DRA's Partitionable Devices have turned fragmentation into a problem the scheduler can handle, but interference and runtime QoS remain outside the scheduler.

Choosing a GPU sharing technique ultimately comes down to answering three questions in order. Whether the co-located workloads compete for the same L2 and memory bandwidth decides whether partitioning SMs alone is enough. Whether one tenant's fatal fault may kill another tenant's process decides how far MPS and time-slicing can be used. Whether the distribution of request sizes fits the fixed profiles decides how much fragmentation MIG will leave behind. And holding to that choice across the cluster requires that the unit the scheduler sees match the unit the hardware actually cuts.

References

← Blog