Enable Dynamic MPS GPU Sharing under Kubernetes Cluster

• Written at: Thursday, March 13, 2025 (GMT+8)

Getting GPU support

• Reference this page
• Installing GPU operator
• Or manually install nvidia driver, cuda library, and install nvidia-device-plugin

Setup Nebuly AI’s Nvidia Device Plugin

• A Nebuly-AI fork version of Nvidia Device Plugin, enabling mps support.
• Reference this page

Disable general nvidia device plugin

1. Find the nvidia device plugin daemonset

• If you use GPU-operator : run kubectl get daemonset -n gpu-operator
• Else run kubectl get daemonset -n nvidia-device-plugin (or namespace of your self installed nvidia device plugin)

2. Take note of the name of daemonset, then run kubectl edit daemonset <daemonset name> -n <namespace>

3. Modify node affinity to this configuration

• issues : sometimes it only works when there is only one matchExpression section, so you may need to remove other matcheExpressions

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: nos.nebuly.com/gpu-partitioning
              operator: NotIn
              values:
                - mps

4. Label the mps nodes : kubectl label node <nodename> nos.nebuly.com/gpu-partitioning=mps

5. Restart daemonset: kubectl rollout restart daemonset <daemonset name> -n <namespace>

6. Run kubectl get po -n <nvidia device plugin namespace> , and check if the device plugin pods doesn't run on nodes with label nos.nebuly.com/gpu-partitioning

Install Nebuly AI device plugin

Install the Nebuly AI nvidia device plugin

helm install oci://ghcr.io/nebuly-ai/helm-charts/nvidia-device-plugin \
  --version 0.13.0 \
  --generate-name \
  -n nebuly-nvidia \
  --create-namespace

Verify installation by kubectl get po -n nebuly-nvidia

Debug

You may encounter the device plugin crashed

1. Run kubectl describe po -n nebuly-nvidia
2. If you see this, it means the device plugin is not running on a nvidia container runtime

Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  22m                  default-scheduler  Successfully assigned nebuly-nvidia/nvidia-device-plugin-1741689090-b2fsg to xcmff-b38w12057146d2dwe
  Normal   Pulling    22m                  kubelet            Pulling image "ghcr.io/nebuly-ai/k8s-device-plugin:v0.13.0"
  Normal   Pulled     22m                  kubelet            Successfully pulled image "ghcr.io/nebuly-ai/k8s-device-plugin:v0.13.0" in 10.189235161s (10.189253044s including waiting)
  Normal   Created    20m (x5 over 22m)    kubelet            Created container set-compute-mode
  Warning  Failed     20m (x5 over 22m)    kubelet            Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "nvidia-smi": executable file not found in $PATH: unknown
  Normal   Pulled     20m (x4 over 22m)    kubelet            Container image "ghcr.io/nebuly-ai/k8s-device-plugin:v0.13.0" already present on machine
  Warning  BackOff    2m9s (x93 over 22m)  kubelet            Back-off restarting failed container set-compute-mode in pod nvidia-device-plugin-1741689090-b2fsg_nebuly-nvidia(12b89aae-56db-437f-aa0a-32ec0

3. To fix this, run kubectl get daemonset -n nebuly-nvidia, to get the daemonset name

4. Run kubectl edit daemonset <daemonset name> -n nebuly-nvidia, add:

spec:
  template:
    spec:
      runtimeClassName: nvidia

1. Save and Restart daemonset , kubectl rollout restart daemonset <daemonset name> -n nebuly-nvidia
2. Check again the device plugins and see if it is running, the logs inside should show something like this (the label may not be default, it is ok , we will fix it next):

Defaulted container "nvidia-device-plugin-sidecar" out of: nvidia-device-plugin-sidecar, nvidia-mps-server, nvidia-device-plugin-ctr, set-compute-mode (init), set-nvidia-mps-volume-permissions (init), nvidi a-device-plugin-init (init)
W0311 14:07:44.711186
33 client_config.go:608] Neither --kubeconfig nor --master was specified. Using the inclusterConfig. This might not work.
time="2025-03-11T14:07:44Z" level=info msg="Waiting for change to 'nvidia.com/device-plugin.config' label" 
time="2025-03-11T14:07:44Z" level=info msg="Label change detected: nvidia.com/device-plugin.config=" 
time="2025-03-11T14:07:44Z" level=info msg="No value set and no default set. Attempting fallback strategies: [named single]" 
time="2025-03-11T14:07:44Z" level=info msg="Attempting to find config named: default" 
time="2025-03-11714:07:447" level=info msa="Updating to config: default" 
time="2025-03-11T14:07:44Z" level=info msg="Already configured. Skipping update..." 
time="2025-03-11T14:07:44Z" level=info msg="Waiting for change to 'nvidia.com/device-plugin.config' label"

Configure MPS partition

1. Run kubectl get configmap -n nebuly-nvidia and take note of the nos-device-plugin-config config map name ( it may be nos-device-plugin-configs)
2. Run kubectl edit configmap <nos config map name> -n nebuly-nvidia
3. Inside the file, replace this under the data section to set a mps partition configuration of two 2GB memory slice on the device 0 GPU.

flags:
  migStrategy: none
sharing:
  mps: 
    failRequestsGreaterThanOne: true
    resources:
      - name: nvidia.com/gpu
        rename: nvidia.com/gpu-2gb
        memoryGB: 2
        replicas: 2
        devices: ["0"]

4. Set the node label to let device plugin reference this config map’s default config kubectl label node nvidia.com/device-plugin.config=default

5. save configmap nd restart the device plugin pod kubectl delete po <device-plugin pod name> -n nebuly-nvidia

6. Run kubectl describe node and check if the mps node has the correct partition

7. Try a test pod:

$ kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: mps-partitioning-example
spec:
  hostIPC: true 
  securityContext:
    runAsUser: 1000 
  containers:
    - name: sleepy
      image: "busybox:latest"
      command: ["sleep", "120"]
      resources:
        limits:
          nvidia.com/gpu-4gb: 1  
EOF

Setup NOS for MPS Dynamic Partition

• Reference this page, and make sure you met the prerequisites

1. Install cert-manager since it is a dependency of NOS

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.17.0/cert-manager.yaml

2. Install NOS via helm

helm install oci://ghcr.io/nebuly-ai/helm-charts/nos \
  --version 0.1.2 \
  --namespace nebuly-nos \
  --generate-name \
  --create-namespace

3. Verify installation kubectl get po -n nebuly-nos

4. Run kubectl describe node, check the mps node and make sure the node label value for nvidia.com/device-plugin.config is not deafult anymore, (likely change to node name, which is done by NOS)

5. Restart the device plugin pod of nebuly-ai nvidia device plugin to let it read the new config set by NOS: kubectl delete po $(kubectl get po -n nebuly-nvidia) -n nebuly-nvidia

6. Try a test pod! Check if NOS helps repartition the GPU slices.

$ kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: mps-partitioning-example
spec:
  hostIPC: true 
  securityContext:
    runAsUser: 1000 
  containers:
    - name: sleepy
      image: "busybox:latest"
      command: ["sleep", "120"]
      resources:
        limits:
          nvidia.com/gpu-13gb: 1  
          nvidia.com/gpu-19gb: 1 
EOF

Debug

Issue 1

You may encounter the issue :

Events:
  Type     Reason       Age                From               Message
  ----     ------       ----               ----               -------
  Normal   Scheduled    63s                default-scheduler  Successfully assigned nebuly-nos/nos-1741745534-gpu-agent-6tx7z to xcmff-b38w12057146d2dwe
  Warning  FailedMount  32s (x7 over 64s)  kubelet            MountVolume.SetUp failed for volume "run-nvidia" : hostPath type check failed: /run/nvidia is not a directory

1. It means it cannot find the nvidia driver in a non nvidia runtime container, and assumes you have gpu-operator installed , but stilled cannot find the drive on the path /run/nvidia.
2. To fix this, find the daemonset of the nebuly gpu agent: kubectl get daemonset -n nebuly-nos
3. Run kubectl edit daemonset <daemonset name> -n nebuly-nos, and add:

spec:
  template:
    spec:
      runtimeClassName: nvidia

4. Rerun daemonset:

kubectl rollout restart daemonset <daemonset name> -n nebuly-nos

Issue 2

If you found out there is no partitioning happen on the gpu

1. Run kubectl get po -n nebuly-nos, and find the gpu partitioner pod, and see its logs using kubectl logs
2. Check if any node label is not set , such as nvidia.com/gpu.count=1, nvidia.com/gpu.product, or nvidia.com/gpu.memory.
3. Properly set the labels by referencing other nodes with the same gpu that havs the label, or refernce to the gpu spec.

Reference:

• How to Increase GPU Utilization in Kubernetes with NVIDIA MPS
• Dynamic MIG Partitioning in Kubernetes
• Nebuly-NOS
• Nebuly-AI/k8s-device-plugin
• Nvidia GPU operator
• Nvidia K8s device plugin
• Enable Dynamic MPS GPU Sharing under Kubernetes Cluster
• Getting GPU support
• Setup Nebuly AI’s Nvidia Device Plugin
• Disable general nvidia device plugin
• Install Nebuly AI device plugin
• Configure MPS partition
• Setup NOS for MPS Dynamic Partition