Slicing an NVIDIA GPU with MIG and Shipping It to Kubernetes for vLLM Inference

A practical, end-to-end walkthrough: from nvidia-smi mig on the host, to a working device plugin, to a vLLM deployment that pulls a model straight from Hugging Face.

Why MIG?

When you put a modern data-center GPU (A100, H100, …) into a Kubernetes cluster, the default device plugin behavior is brutally simple: one pod gets one whole GPU. For training that’s fine. For inference workloads — where a 7B–30B parameter LLM rarely needs 80 GB of HBM — it’s incredibly wasteful.

Multi-Instance GPU (MIG) lets you carve a single physical GPU into up to 7 hardware-isolated instances. Each instance has its own slice of SMs, L2 cache, memory controllers and HBM. From the OS, the kernel, the container runtime and Kubernetes’ perspective, each slice looks and behaves like an independent GPU.

That’s the unlock: instead of one pod hogging the card, you can land seven inference pods on the same physical device, each with predictable, isolated performance.

⚠️ Mental model warning (this trips everyone up, including me). A 1g.18gb slice is not a thin shim on top of unified GPU memory. It is initialized as a GPU with exactly 18 GB of HBM, period. If your model + KV cache + activations don't fit in 18 GB, requesting more slices won't magically pool the memory across them — you need a bigger profile, or you need tensor-parallel sharding across multiple slices. Plan capacity per-slice, not per-card.

The Architecture We’re Building

Three layers:

1. Host — we use nvidia-smi mig to partition one specific GPU into 7× 1g.18gb instances.
2. Kubernetes node — the NVIDIA device plugin (running as a DaemonSet) discovers the MIG instances and advertises them to the kubelet as the schedulable resource nvidia.com/mig-1g.18gb.
3. Workload — a vLLM pod requests N slices via resources.limits like it would request CPU or memory.

Let’s walk through each layer.

Step 1 — Partition the GPU on the Host

Assume a node with multiple GPUs. We want to slice GPU index 3 into seven equal instances. MIG must already be enabled on that GPU (nvidia-smi -i 3 -mig 1); after that:

# List the available GPU instance profiles on GPU 3
sudo nvidia-smi mig -lgip -i 3

# Create 7 × 1g.18gb instances on GPU 3
# (profile ID 19 = 1g.18gb on A100-80GB — confirm from the -lgip output)
sudo nvidia-smi mig -cgi 19,19,19,19,19,19,19 -C -i 3

What just happened:

• -cgi creates the GPU Instances (the SM + memory partitions).
• -C immediately creates a matching Compute Instance inside each GI, which is what containers actually attach to.
• -i 3 scopes everything to the GPU at index 3 — the other GPUs on the node stay untouched and continue to expose themselves as full devices.

Verify:

nvidia-smi -i 3
# You should see 7 MIG devices listed under GPU 3

This is also why the next step matters: because our node has both a partitioned GPU and full GPUs, the device plugin needs to handle both at once.

Step 2 — Deploy the NVIDIA Device Plugin in mixed MIG mode

The device plugin has three MIG strategies:

StrategyWhen to usenoneNo MIG anywhere. Plugin exposes nvidia.com/gpu only.singleEvery GPU on the node is partitioned identically. Exposes nvidia.com/gpu as if it were a slice.**mixed**Some GPUs are partitioned, some aren't — or different profiles coexist. Exposes resources like nvidia.com/mig-1g.18gb alongside nvidia.com/gpu.

Our node has GPU 3 sliced and the rest untouched, so **mixed is the only correct choice**.

Here’s a stripped-down DaemonSet — no proprietary registry, no cluster-management noise, just the parts that matter:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin-ds
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
  template:
    metadata:
      labels:
        name: nvidia-device-plugin-ds
    spec:
      runtimeClassName: nvidia
      priorityClassName: system-node-critical
      nodeSelector:
        nvidia.com/gpu.available: "true"
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: nvidia-device-plugin-ctr
          image: nvcr.io/nvidia/k8s-device-plugin:v0.17.1
          securityContext:
            privileged: true
          env:
            - name: MIG_STRATEGY
              value: mixed
            - name: FAIL_ON_INIT_ERROR
              value: "false"
            - name: PASS_DEVICE_SPECS
              value: "true"
            - name: DEVICE_LIST_STRATEGY
              value: envvar
            - name: DEVICE_ID_STRATEGY
              value: uuid
            - name: NVIDIA_VISIBLE_DEVICES
              value: all
            - name: NVIDIA_DRIVER_CAPABILITIES
              value: all
            - name: NVIDIA_MIG_MONITOR_DEVICES
              value: all
          volumeMounts:
            - name: device-plugin
              mountPath: /var/lib/kubelet/device-plugins
      volumes:
        - name: device-plugin
          hostPath:
            path: /var/lib/kubelet/device-plugins

Apply it and inspect the node:

kubectl apply -f nvidia-device-plugin.yaml
kubectl describe node <node-name> | grep -E "nvidia.com|Allocatable" -A2

You should now see something like:

nvidia.com/mig-1g.18gb:    7        # the slices on GPU 3

That’s it — Kubernetes now treats each MIG slice as a first-class schedulable resource.

Step 3 — Run vLLM on the Slices

Now the fun part. We’ll deploy vLLM and have it serve a model directly from Hugging Face — no NFS, no PVCs, no shared filesystem. vLLM downloads the weights into the container’s HF cache on first start.

The key line is in resources: we ask for nvidia.com/mig-1g.18gb: "4" — four slices, which the scheduler will pin to the same physical GPU (they can only come from one card by definition).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: qwen-vllm
  namespace: vllm
  labels:
    app: qwen-vllm
spec:
  replicas: 1
  selector:
    matchLabels:
      app: qwen-vllm
  template:
    metadata:
      labels:
        app: qwen-vllm
    spec:
      runtimeClassName: nvidia
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          imagePullPolicy: IfNotPresent
          args:
            - --model
            - Qwen/Qwen2.5-7B-Instruct       # pulled from Hugging Face
            - --tensor-parallel-size
            - "4"                            # shard across 4 MIG slices
            - --gpu-memory-utilization
            - "0.95"
            - --max-model-len
            - "16384"
            - --host
            - 0.0.0.0
            - --port
            - "8000"
          env:
            # For gated/private models, mount this from a Secret instead.
            - name: HUGGING_FACE_HUB_TOKEN
              value: ""
          ports:
            - containerPort: 8000
          readinessProbe:
            httpGet: { path: /health, port: 8000 }
            initialDelaySeconds: 60
            periodSeconds: 5
          livenessProbe:
            httpGet: { path: /health, port: 8000 }
            initialDelaySeconds: 60
            periodSeconds: 10
          resources:
            requests:
              cpu: "2"
              memory: 40Gi
              nvidia.com/mig-1g.18gb: "4"
            limits:
              cpu: "4"
              memory: 80Gi
              nvidia.com/mig-1g.18gb: "4"
---
apiVersion: v1
kind: Service
metadata:
  name: qwen-vllm
  namespace: vllm
spec:
  selector:
    app: qwen-vllm
  ports:
    - port: 80
      targetPort: 8000

A quick smoke test once it’s Ready:

kubectl -n vllm port-forward svc/qwen-vllm 8080:80

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [{"role":"user","content":"Hello from a MIG slice!"}]
  }'

You’re now talking to an OpenAI-compatible endpoint running on 4 hardware-isolated GPU slices.

Capacity Planning: How Big a Model Fits on a Slice?

This is where the “each slice is its own GPU” rule bites hard. A rough rule of thumb on a 1g.18gb slice (≈18 GB usable HBM):

What lives in the sliceApprox. budgetModel weights (fp16)~2 × params_billions GBKV cachescales with max-model-len × batch × layersCUDA / framework overhead1–2 GB

So a 7B fp16 model (~14 GB) fits in one slice — barely — with a small context window. Want longer context or a bigger model? Either:

• Use a larger MIG profile (2g.36gb, 3g.45gb, …), or
• Tensor-parallel across multiple slices with vLLM’s --tensor-parallel-size, as we did above.

Just remember: tensor parallelism splits the weights, but each slice still needs to hold its shard plus its share of activations and KV cache. You cannot escape the 18 GB-per-slice ceiling.

Production Notes

A few things I’d add before declaring victory:

• Pin the image tag. vllm/vllm-openai:latest is great for a demo, terrible for reproducibility. Pin to a specific version.
• HF token via Secret. For gated models (Llama, Gemma, …), mount HUGGING_FACE_HUB_TOKEN from a Secret, never inline.
• Persistent HF cache. First boot redownloads weights. Mount a PVC at /root/.cache/huggingface if your pods get rescheduled often — otherwise you pay the download tax every time.
• HPA needs a custom metric. CPU-based autoscaling is useless here; scale on vllm:num_requests_running or queue depth scraped via Prometheus.
• MIG config is host-state. It survives reboots only if you use the NVIDIA GPU Operator’s MigManager or persist it via systemd. Treat the manual nvidia-smi mig -cgi as bootstrap, not as your source of truth in production.

Wrap-up

With less than 100 lines of YAML and three nvidia-smi commands, we turned a single A100 into seven independently schedulable GPUs, taught Kubernetes to schedule them, and ran a real LLM inference server on top — without giving up the ability to use the rest of the node's GPUs as full devices.

MIG won’t make sense for every workload (training, big models, anything memory-bound past 40 GB), but for the long tail of small-and-medium inference services, it’s the single highest-leverage knob you can turn on a modern GPU cluster.

If you found this useful, follow for more posts on running LLMs on Kubernetes the boring, reliable way.