Sharing one GPU between multiple LLMs on Kubernetes (MIG, Time-slicing, MPS)

Exploration guide. Goal: run two independent LLM workloads
(vLLM served through KServe LLMInferenceService) on a single physical GPU.
There are three ways to do this on NVIDIA GPUs. This document explains each one,
gives a step-by-step guide to enable it, a working example to deploy, and
a comparison to help choose. Everything is self-contained — copy/paste ready.

Environment

Single-node cluster used for this exploration:

Item	Value
Node	`hidon` (control-plane + worker)
OS / Kubernetes	Ubuntu 24.04, Kubernetes v1.33, containerd
GPUs	8× NVIDIA H100 SXM5 80GB (DGX H100), Hopper, compute 9.0
Driver / CUDA	595.91.07 / runtime 13.2
GPU stack	NVIDIA GPU Operator already installed (Helm), all pods healthy

Quick checks:

kubectl get nodes -o wide
lspci | grep -i nvidia                       # 8x GH100 [H100 SXM5 80GB]
kubectl get pods -n gpu-operator-resources   # driver, device-plugin, mig-manager, ...

The GPU Operator manages everything (driver, device plugin, MIG manager). This is
important: enabling any of the three sharing modes is done through operator
configuration (a node label or a ConfigMap), never with manual nvidia-smi
commands on the host. In this cluster the GPU operator namespace is
gpu-operator-resources and the ClusterPolicy is named cluster-policy.

Background: three ways to share one GPU

By default Kubernetes treats a GPU as one indivisible unit — only one pod can
use it (nvidia.com/gpu: 1). To place two LLMs on one GPU we need one of:

	MIG	Time-slicing	MPS
What it does	Cuts the GPU into physically separate mini-GPUs	Lets pods take turns on the whole GPU	Lets pods run at the same time, sharing the cores
Compute sharing	Hardware partitions	Temporal (round-robin, context switch)	Spatial (concurrent on different SMs)
Memory isolation	Yes (hardware)	No	Soft cap (optional)
Fault isolation	Yes	Weak	Weakest (shared context)
Needs `--gpu-memory-utilization`?	No (each slice has fixed memory)	Yes	Yes
Latency behaviour	Best / predictable	Jittery under load	Smooth
Simple analogy	Walled mini-GPUs	One-lane road, traffic light	Multi-lane highway
Best for	Production, multi-tenant, strict QoS	Dev / test, light bursty loads	Two cooperative models, good throughput

Key idea for Time-slicing and MPS: the GPU memory is not partitioned, so both
vLLM processes see the full 80 GB. By default vLLM grabs 90% of memory
(--gpu-memory-utilization 0.9) on startup, so the second one would fail with
out-of-memory. You must cap each instance, e.g. --gpu-memory-utilization 0.4
so that 0.4 + 0.4 = 0.8 of 80 GB fits with headroom. MIG does not need this
because each MIG slice already has its own fixed memory.

Option A — MIG (Multi-Instance GPU)

MIG splits one physical GPU into up to 7 fully isolated instances. Each instance
has its own compute cores, memory, and cache. This is the only option with real
hardware isolation.

MIG profiles on H100 80GB

A profile name like 3g.40gb means 3 compute slices + 40 GB memory.

Profile	Instances per GPU	Memory each	Use case
`1g.10gb`	7	10 GB	Many tiny models
`2g.20gb`	3	20 GB	Small models
`3g.40gb`	2	40 GB	Two LLMs (used here)
`4g.40gb`	1 (+unused)	40 GB	—
`7g.80gb`	1	80 GB	Whole GPU (no split)

MIG partitions are fixed-size. You cannot create an arbitrary size such as 33 GB.

Step 1 — Enable MIG

The operator's MIG manager watches a node label. Setting it to all-3g.40gb
splits every GPU into 2×40 GB.

kubectl label node hidon nvidia.com/mig.config=all-3g.40gb --overwrite

Step 2 — Verify

# Wait until the state becomes "success"
kubectl get node hidon -o json | jq -r '.metadata.labels["nvidia.com/mig.config.state"]'

# Node now advertises 16 GPUs (8 GPUs x 2 slices)
kubectl get node hidon -o json | jq -r '.status.allocatable["nvidia.com/gpu"]'   # 16

# Product label reflects MIG
kubectl get node hidon -o json | jq -r '.metadata.labels["nvidia.com/gpu.product"]'
# NVIDIA-H100-80GB-HBM3-MIG-3g.40gb

# Confirm the instances on the hardware
DRV=$(kubectl get pod -n gpu-operator-resources -l app=nvidia-driver-daemonset -o name | head -1)
kubectl exec -n gpu-operator-resources $DRV -- nvidia-smi mig -lgi   # 2x MIG 3g.40gb per GPU

Step 3 — Deploy two LLMs

Because we use the single MIG strategy, each 40 GB slice looks like a normal GPU,
so each workload simply requests nvidia.com/gpu: 1. No memory flag needed —
the slice already has a fixed 40 GB. The example serves the public model
Qwen/Qwen2.5-0.5B-Instruct (no HuggingFace token required).

kubectl apply -f - <<'EOF'
apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: llm-a
  namespace: default
spec:
  model:
    uri: hf://Qwen/Qwen2.5-0.5B-Instruct
    name: Qwen/Qwen2.5-0.5B-Instruct
  storageInitializer:
    enabled: false
  replicas: 1
  router:
    scheduler: {}
    route: {}
  template:
    nodeSelector:
      nvidia.com/gpu.present: "true"
    volumes:
      - name: kserve-pvc-source
        emptyDir: {}
    initContainers:
      - name: storage-initializer
        image: kserve/storage-initializer:v0.17.0
        args: ["hf://Qwen/Qwen2.5-0.5B-Instruct", "/mnt/models"]
        env:
          - { name: HF_HUB_ENABLE_HF_TRANSFER, value: "0" }
          - { name: HF_HUB_DOWNLOAD_TIMEOUT, value: "60" }
        volumeMounts:
          - { name: kserve-pvc-source, mountPath: /mnt/models }
    containers:
      - name: main
        image: ghcr.io/llm-d/llm-d-cuda:v0.4.0
        args: ["--enforce-eager"]
        securityContext:
          runAsNonRoot: false
        env:
          - { name: VLLM_LOGGING_LEVEL, value: INFO }
        volumeMounts:
          - { name: kserve-pvc-source, mountPath: /mnt/models, readOnly: true }
        resources:
          requests: { cpu: "1", memory: "8Gi", nvidia.com/gpu: "1" }
          limits:   { cpu: "4", memory: "16Gi", nvidia.com/gpu: "1" }
---
apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: llm-b
  namespace: default
spec:
  model:
    uri: hf://Qwen/Qwen2.5-0.5B-Instruct
    name: Qwen/Qwen2.5-0.5B-Instruct
  storageInitializer:
    enabled: false
  replicas: 1
  router:
    scheduler: {}
    route: {}
  template:
    nodeSelector:
      nvidia.com/gpu.present: "true"
    volumes:
      - name: kserve-pvc-source
        emptyDir: {}
    initContainers:
      - name: storage-initializer
        image: kserve/storage-initializer:v0.17.0
        args: ["hf://Qwen/Qwen2.5-0.5B-Instruct", "/mnt/models"]
        env:
          - { name: HF_HUB_ENABLE_HF_TRANSFER, value: "0" }
          - { name: HF_HUB_DOWNLOAD_TIMEOUT, value: "60" }
        volumeMounts:
          - { name: kserve-pvc-source, mountPath: /mnt/models }
    containers:
      - name: main
        image: ghcr.io/llm-d/llm-d-cuda:v0.4.0
        args: ["--enforce-eager"]
        securityContext:
          runAsNonRoot: false
        env:
          - { name: VLLM_LOGGING_LEVEL, value: INFO }
        volumeMounts:
          - { name: kserve-pvc-source, mountPath: /mnt/models, readOnly: true }
        resources:
          requests: { cpu: "1", memory: "8Gi", nvidia.com/gpu: "1" }
          limits:   { cpu: "4", memory: "16Gi", nvidia.com/gpu: "1" }
EOF

kubectl get llminferenceservices -n default      # wait for READY = True

Step 4 — Check each pod got its own MIG slice

PA=$(kubectl get pods -n default | grep -E 'llm-a-kserve-[0-9a-z]+-' | grep -v router | awk '{print $1}')
kubectl exec -n default $PA -c main -- nvidia-smi -L
# GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-...)
#   MIG 3g.40gb  Device 0: (UUID: MIG-...)   <- one isolated 40 GB slice

Both LLMs reach READY=True, each running on its own isolated 40 GB MIG slice.

Revert MIG

kubectl delete llminferenceservice llm-a llm-b -n default
kubectl label node hidon nvidia.com/mig.config=all-disabled --overwrite
kubectl get node hidon -o json | jq -r '.status.allocatable["nvidia.com/gpu"]'   # back to 8

Option B — Time-slicing

Time-slicing tells the device plugin to advertise each GPU as several "replicas".
Multiple pods can then be scheduled on one GPU and the hardware round-robins
between them (one process runs at a time, then a context switch). There is no
isolation — the memory flag is mandatory.

Step 1 — Make sure MIG is off

kubectl label node hidon nvidia.com/mig.config=all-disabled --overwrite
# wait for state=success and allocatable nvidia.com/gpu = 8

Step 2 — Enable time-slicing

Create the device-plugin config (replicas: 4 per GPU) and point the ClusterPolicy
at it.

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: gpu-operator-resources
data:
  any: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        resources:
          - name: nvidia.com/gpu
            replicas: 4
EOF

kubectl patch clusterpolicy cluster-policy --type merge \
  -p '{"spec":{"devicePlugin":{"config":{"name":"time-slicing-config","default":"any"}}}}'

Step 3 — Verify

kubectl get node hidon -o json | jq -r '.status.allocatable["nvidia.com/gpu"]'                 # 32 (8 x 4)
kubectl get node hidon -o json | jq -r '.metadata.labels["nvidia.com/gpu.sharing-strategy"]'   # time-slicing
kubectl get node hidon -o json | jq -r '.metadata.labels["nvidia.com/gpu.replicas"]'           # 4
kubectl get node hidon -o json | jq -r '.metadata.labels["nvidia.com/gpu.product"]'            # ...-SHARED

Step 4 — Deploy two LLMs (with memory cap)

Each vLLM requests nvidia.com/gpu: 1 and sets --gpu-memory-utilization 0.4
so both fit in the shared 80 GB.

kubectl apply -f - <<'EOF'
apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: ts-llm-a
  namespace: default
spec:
  model:
    uri: hf://Qwen/Qwen2.5-0.5B-Instruct
    name: Qwen/Qwen2.5-0.5B-Instruct
  storageInitializer:
    enabled: false
  replicas: 1
  router:
    scheduler: {}
    route: {}
  template:
    nodeSelector:
      nvidia.com/gpu.present: "true"
    volumes:
      - name: kserve-pvc-source
        emptyDir: {}
    initContainers:
      - name: storage-initializer
        image: kserve/storage-initializer:v0.17.0
        args: ["hf://Qwen/Qwen2.5-0.5B-Instruct", "/mnt/models"]
        env:
          - { name: HF_HUB_ENABLE_HF_TRANSFER, value: "0" }
          - { name: HF_HUB_DOWNLOAD_TIMEOUT, value: "60" }
        volumeMounts:
          - { name: kserve-pvc-source, mountPath: /mnt/models }
    containers:
      - name: main
        image: ghcr.io/llm-d/llm-d-cuda:v0.4.0
        args: ["--enforce-eager", "--gpu-memory-utilization", "0.4"]  # required
        securityContext:
          runAsNonRoot: false
        env:
          - { name: VLLM_LOGGING_LEVEL, value: INFO }
        volumeMounts:
          - { name: kserve-pvc-source, mountPath: /mnt/models, readOnly: true }
        resources:
          requests: { cpu: "1", memory: "8Gi", nvidia.com/gpu: "1" }
          limits:   { cpu: "4", memory: "16Gi", nvidia.com/gpu: "1" }
---
apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: ts-llm-b
  namespace: default
spec:
  model:
    uri: hf://Qwen/Qwen2.5-0.5B-Instruct
    name: Qwen/Qwen2.5-0.5B-Instruct
  storageInitializer:
    enabled: false
  replicas: 1
  router:
    scheduler: {}
    route: {}
  template:
    nodeSelector:
      nvidia.com/gpu.present: "true"
    volumes:
      - name: kserve-pvc-source
        emptyDir: {}
    initContainers:
      - name: storage-initializer
        image: kserve/storage-initializer:v0.17.0
        args: ["hf://Qwen/Qwen2.5-0.5B-Instruct", "/mnt/models"]
        env:
          - { name: HF_HUB_ENABLE_HF_TRANSFER, value: "0" }
          - { name: HF_HUB_DOWNLOAD_TIMEOUT, value: "60" }
        volumeMounts:
          - { name: kserve-pvc-source, mountPath: /mnt/models }
    containers:
      - name: main
        image: ghcr.io/llm-d/llm-d-cuda:v0.4.0
        args: ["--enforce-eager", "--gpu-memory-utilization", "0.4"]  # required
        securityContext:
          runAsNonRoot: false
        env:
          - { name: VLLM_LOGGING_LEVEL, value: INFO }
        volumeMounts:
          - { name: kserve-pvc-source, mountPath: /mnt/models, readOnly: true }
        resources:
          requests: { cpu: "1", memory: "8Gi", nvidia.com/gpu: "1" }
          limits:   { cpu: "4", memory: "16Gi", nvidia.com/gpu: "1" }
EOF

kubectl get llminferenceservices -n default        # wait for READY = True

Step 5 — Verify the pods are sharing a GPU

PA=$(kubectl get pods -n default | grep -E 'ts-llm-a-kserve-[0-9a-z]+-' | grep -v router | awk '{print $1}')
kubectl exec -n default $PA -c main -- nvidia-smi -L
# GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-...)

Both LLMs run on time-sliced GPU replicas, each limited to 40% of GPU memory.

Revert time-slicing

kubectl delete llminferenceservice ts-llm-a ts-llm-b -n default
kubectl patch clusterpolicy cluster-policy --type json \
  -p '[{"op":"remove","path":"/spec/devicePlugin/config"}]'
# node returns to allocatable nvidia.com/gpu = 8, sharing-strategy = none

Option C — MPS (Multi-Process Service)

MPS is like time-slicing (same replica model in Kubernetes) but the processes run
concurrently through the MPS daemon instead of taking turns. This gives better
utilization and smoother latency. It can also enforce a soft per-client memory cap.

Step 1 — Make sure MIG is off

kubectl label node hidon nvidia.com/mig.config=all-disabled --overwrite

Step 2 — Enable MPS

Same idea as time-slicing but the sharing key is mps.

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: mps-config
  namespace: gpu-operator-resources
data:
  any: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      mps:
        resources:
          - name: nvidia.com/gpu
            replicas: 4
EOF

kubectl patch clusterpolicy cluster-policy --type merge \
  -p '{"spec":{"devicePlugin":{"config":{"name":"mps-config","default":"any"}}}}'

The operator starts an extra nvidia-device-plugin-mps-control-daemon pod that
runs the MPS server.

Step 3 — Verify

kubectl get node hidon -o json | jq -r '.status.allocatable["nvidia.com/gpu"]'                 # 32 (8 x 4)
kubectl get node hidon -o json | jq -r '.metadata.labels["nvidia.com/gpu.sharing-strategy"]'   # mps
kubectl get pods -n gpu-operator-resources | grep mps                                          # mps-control-daemon

Step 4 — Deploy two LLMs (with memory cap)

The workload is the same as Option B, Step 4 (request one GPU + set
--gpu-memory-utilization 0.4), because MPS also uses the nvidia.com/gpu
resource. Apply the same two LLMInferenceService manifests (rename them to
mps-llm-a / mps-llm-b if you want to run alongside the others).

Revert MPS

kubectl delete llminferenceservice ts-llm-a ts-llm-b -n default
kubectl patch clusterpolicy cluster-policy --type json \
  -p '[{"op":"remove","path":"/spec/devicePlugin/config"}]'

Testing an LLM after deployment (any option)

Once a service is READY=True, send it a request. The simplest way is to
port-forward to the model pod and call the OpenAI-compatible API that vLLM exposes:

POD=$(kubectl get pods -n default | grep -E 'kserve-[0-9a-z]+-' | grep -v router | head -1 | awk '{print $1}')
kubectl port-forward -n default $POD 8000:8000 &

curl -s http://localhost:8000/v1/models | jq
curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen/Qwen2.5-0.5B-Instruct",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}]
  }' | jq

Comparison summary

Question	MIG	Time-slicing	MPS
Real memory isolation?	✅ Yes	❌ No	⚠️ Soft cap
Real compute isolation?	✅ Yes	❌ No	⚠️ Shared cores
Run at the same time?	✅	❌ (take turns)	✅
Crash of one affects others?	No	Unlikely	Possible
Need `--gpu-memory-utilization`?	No	Yes	Yes
How to enable	Node label	Device-plugin ConfigMap	Device-plugin ConfigMap
GPUs advertised (this node)	16	32	32
Setup difficulty	Low	Low	Low–medium
Best fit	Production multi-tenant	Dev / test, light loads	Two cooperative LLMs, throughput

Which one to choose

Need strong isolation / SLAs / untrusted tenants → MIG.
Just want to pack a few light or bursty jobs cheaply → Time-slicing.
Want two models truly running in parallel with good throughput → MPS.

Only one sharing mode is active at a time. Always disable MIG before
switching to time-slicing or MPS, and remove the device-plugin config before
switching back to MIG.

Uh oh!

[Exploration] testing fractional GPUs for llm workloads #587

Description

Context

What needs to get done

Definition of Done

Activity

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions