The DevOps Engineer’s Guide to GPU Infrastructure on Kubernetes

GPU infrastructure on Kubernetes has gone from a niche concern handled by dedicated ML platform teams to a core competency every DevOps and platform engineer needs in 2026. The explosion of AI workloads — LLM inference, model fine-tuning, embedding generation, computer vision pipelines — means GPU scheduling, monitoring, and cost management are now mainstream platform engineering responsibilities, not specialist territory.

This guide is the comprehensive, practical reference for running GPU workloads on Kubernetes: the device plugin architecture, GPU sharing strategies (MIG, time-slicing, MPS), scheduling for mixed CPU/GPU clusters, monitoring with DCGM, and the cost optimisation patterns that prevent your GPU bill from spiralling out of control. Every configuration is tested against current NVIDIA GPU Operator and Kubernetes 1.30+.

HARDWARE CONTEXT

This guide focuses on NVIDIA GPUs (A100, H100, L40S, A10G) as they represent the overwhelming majority of production Kubernetes GPU workloads in 2026. AMD GPU support (via the AMD GPU Operator) follows similar patterns with ROCm-specific configuration differences noted where relevant.

The Foundation: NVIDIA GPU Operator

The NVIDIA GPU Operator is the single most important piece of GPU infrastructure tooling — it automates the deployment of the NVIDIA driver, container toolkit, device plugin, DCGM monitoring exporter, and MIG manager as a coordinated set of components. Before the GPU Operator existed, each of these had to be manually installed and version-matched on every GPU node — a significant operational burden.

# Install NVIDIA GPU Operator via Helm
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator --create-namespace \
  --set driver.enabled=true \
  --set toolkit.enabled=true \
  --set devicePlugin.enabled=true \
  --set dcgmExporter.enabled=true \
  --set migManager.enabled=true

# Verify GPU is visible to the cluster
kubectl get nodes -o json | jq '.items[].status.capacity["nvidia.com/gpu"]'

# Label GPU nodes for scheduling
kubectl label nodes <gpu-node> node-pool=gpu-inference
kubectl taint nodes <gpu-node> nvidia.com/gpu=present:NoSchedule

GPU Sharing Strategies: MIG vs Time-Slicing vs MPS

MIG: Hardware-Level GPU Partitioning

Multi-Instance GPU, available on A100 and H100 GPUs, physically partitions a single GPU into up to 7 isolated instances, each with dedicated compute cores and memory. This is the only sharing strategy that provides genuine hardware-level isolation — a workload in one MIG partition cannot affect or see workloads in another partition.

# Configure MIG profile via GPU Operator ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: mig-parted-config
  namespace: gpu-operator
data:
  config.yaml: |
    version: v1
    mig-configs:
      all-1g.10gb:
        - devices: all
          mig-enabled: true
          mig-devices:
            1g.10gb: 7  # 7 instances of 1/7 GPU + 10GB memory each
      mixed-balanced:
        - devices: all
          mig-enabled: true
          mig-devices:
            2g.20gb: 2  # 2 instances of 2/7 GPU + 20GB
            1g.10gb: 3  # 3 instances of 1/7 GPU + 10GB

# Apply the MIG profile to specific nodes
kubectl label node <gpu-node> nvidia.com/mig.config=all-1g.10gb --overwrite

# Pod requesting a MIG slice instead of a whole GPU
resources:
  limits:
    nvidia.com/mig-1g.10gb: 1  # Request one 1g.10gb MIG instance

Time-Slicing: Software-Based Sharing

# Time-slicing config — multiple pods share one physical GPU
# via rapid context switching (no hardware isolation)
apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: gpu-operator
data:
  any: |
    version: v1
    sharing:
      timeSlicing:
        resources:
        - name: nvidia.com/gpu
          replicas: 4  # Each physical GPU appears as 4 schedulable units

helm upgrade gpu-operator nvidia/gpu-operator \
  --set devicePlugin.config.name=time-slicing-config

# CAUTION: time-slicing provides no memory isolation —
# a memory-hungry process can OOM other processes sharing the GPU
# Best for: dev/test, low-priority batch, NOT production inference with SLOs

Scheduling: Mixed CPU/GPU Clusters

Node pool separation and taints

# Separate GPU node pool with taint to prevent non-GPU pods scheduling there
# (EKS managed node group example)
eksctl create nodegroup \
  --cluster prod-cluster \
  --name gpu-pool \
  --node-type p4d.24xlarge \
  --node-labels 'node-pool=gpu-inference' \
  --node-taints 'nvidia.com/gpu=present:NoSchedule' \
  --nodes-min 0 --nodes-max 8

# Deployment with matching toleration and node affinity
spec:
  template:
    spec:
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node-pool
                operator: In
                values: [gpu-inference]

Priority classes: protecting inference from batch preemption

# Ensure latency-sensitive inference preempts batch training jobs
# when GPU capacity is constrained
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata: { name: gpu-inference-critical }
value: 1000000
globalDefault: false
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata: { name: gpu-batch-training }
value: 100
globalDefault: false
---
# Inference deployment uses high priority
spec:
  template:
    spec:
      priorityClassName: gpu-inference-critical
---
# Training job uses low priority — will be preempted if GPU capacity needed
spec:
  template:
    spec:
      priorityClassName: gpu-batch-training

Monitoring: GPU Observability with DCGM

# Install DCGM Exporter for Prometheus-compatible GPU metrics
helm install dcgm-exporter gpu-helm-charts/dcgm-exporter -n monitoring

# ServiceMonitor for Prometheus Operator scraping
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata: { name: dcgm-exporter, namespace: monitoring }
spec:
  selector:
    matchLabels: { app: dcgm-exporter }
  endpoints:
  - port: metrics
    interval: 15s

# Key alerts to configure:
- alert: GPUMemoryNearLimit
  expr: DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL > 0.95
  for: 5m
  annotations: { summary: 'GPU memory above 95% — risk of OOM' }

- alert: GPUOverheating
  expr: DCGM_FI_DEV_GPU_TEMP > 85
  for: 2m
  annotations: { summary: 'GPU temperature critical — check cooling' }

- alert: GPUUnderutilised
  expr: avg_over_time(DCGM_FI_DEV_GPU_UTIL[1h]) < 10
  for: 1h
  annotations: { summary: 'GPU under 10% utilised for 1h — cost waste candidate' }

Cost Optimisation for GPU Workloads

1. Right-size GPU type to workload: an H100 for a 7B parameter model is significant overspend; match GPU memory to model size with headroom, not the largest available instance
2. Use MIG for multiple small inference workloads rather than dedicating whole GPUs to each — can reduce GPU count needed by 60–70% for workloads that fit in 1g.10gb slices
3. Spot/preemptible instances for training jobs with checkpoint/resume — training jobs tolerate interruption far better than inference; savings of 60–90% vs on-demand GPU pricing
4. Scale-to-zero for batch and dev GPU node pools using Karpenter or Cluster Autoscaler — GPU nodes sitting idle overnight are the single largest preventable GPU cost
5. Set GPU utilisation alerts (shown above) and review weekly — idle GPU allocation is invisible until you measure it explicitly

Common GPU Infrastructure Mistakes

• Forgetting the toleration: a pod requesting nvidia.com/gpu without the matching toleration for the GPU node taint will be Pending forever with a confusing scheduling error
• Using whole-GPU allocation for small models: a 7B parameter model uses ~14GB VRAM — running it on a full 80GB A100 wastes 66GB of capacity that MIG or time-slicing could have shared
• No startup probe tuning: GPU workloads with large model weights need extended startup probe timing (covered in our Platform Engineering for AI Workloads guide) — default Kubernetes probe timing kills loading pods
• Ignoring driver version compatibility: GPU Operator manages driver versions, but mixing self-installed drivers with Operator-managed ones causes version conflicts that crash the device plugin
• No PriorityClass separation: without priority classes, a burst of low-priority batch jobs can starve latency-sensitive inference of GPU capacity during contention

GETTING STARTED

Install the NVIDIA GPU Operator on a single GPU node first (not your whole fleet). Verify nvidia.com/gpu shows in node capacity. Deploy one inference workload with proper tolerations, probes, and priority class. Add DCGM monitoring before you add a second GPU node — utilisation visibility from day one prevents the ‘we have no idea if these GPUs are being used’ problem that plagues most GPU infrastructure six months in.