[Under]Pod-Based Hybrid K8s-Slurm GPU Scheduling with Slinky Operator and KEDA — the Under Pattern

Introduction

The Under pattern takes a fundamentally different approach to K8s-Slurm integration: instead of bridging two separate systems, it runs Slurm itself as Kubernetes Pods. The Slinky Operator deploys slurmctld and slurmd as managed workloads inside the cluster, and KEDA dynamically scales slurmd Pods to share GPUs between HPC and AI jobs. This article shows how we built a hybrid environment where HPC users submit via sbatch and AI users submit via kubectl — with GPU isolation handled by K8s Pod scheduling rather than Slurm partition engineering.

The Problem: slurmd Pods Monopolize GPUs

When Slinky deploys slurmd as K8s Pods, each Pod requests a GPU through K8s nvidia-device-plugin:

With 4 GPU workers (1 GPU each) and 4 slurmd replicas, every GPU is allocated to slurmd Pods. From the K8s scheduler’s perspective, GPU availability is zero. PyTorchJob, MPIJob, and other K8s native GPU workloads stay Pending indefinitely.

This is the core tension of the Under pattern: Slurm needs slurmd Pods to run HPC jobs, but those Pods consume the very GPUs that K8s native jobs also need.

The Under Architecture

The solution is to dynamically control slurmd Pod replicas with KEDA, releasing GPUs to K8s when Slurm doesn’t need them.

Two key design decisions:

First, GPU allocation authority shifts dynamically. When Slurm jobs are queued, KEDA scales up slurmd Pods that claim GPUs via K8s resource requests. When Slurm jobs finish, KEDA scales them down, returning GPUs to the K8s resource pool for native workloads.

Second, no bridging layer is needed. Unlike the Converged pattern (slurm-bridge) or the Distant pattern (interLink/Virtual Kubelet), Under runs HPC and AI jobs through completely independent paths. Slurm jobs go through slurmctld. K8s jobs go through kube-scheduler. They share GPUs through K8s nvidia-device-plugin, not through a scheduling bridge.

Why GPU Double-Booking Doesn’t Happen

K8s nvidia-device-plugin manages all physical GPU assignments. When a slurmd Pod starts with nvidia.com/gpu: 1, that GPU is marked as consumed in the K8s scheduler. Slurm GRES (AutoDetect=nvidia) detects the same GPU inside the Pod. Slurm jobs execute within the slurmd Pod and can only access the GPU that K8s assigned to it.

When a K8s native AI job requests nvidia.com/gpu: 1, the scheduler assigns a GPU that no slurmd Pod is using. The two paths never share the same physical GPU.

This means nvidia-device-plugin provides GPU exclusivity for both Slurm and K8s paths. No Dual Partition (Converged) or structural separation (Distant) is required.

Test Environment

We validated this architecture on AWS EC2 with 5 nodes:

• Control node (t3.medium): K8s control plane + Slinky Operator + KEDA
• GPU Worker 1–4 (g4dn.xlarge × 4): kubelet + slurmd Pod + Tesla T4

Software stack: K8s v1.34, Slinky Operator v1.0.0 (Helm OCI), Slurm 25.11-ubuntu24.04 (containerized), NVIDIA Device Plugin v0.17.0, NVIDIA Driver 580, containerd, KEDA v2.16.1, Kubeflow Training Operator v1.8.1, cert-manager v1.19.2

How Slinky Changes Slurm Infrastructure

The defining feature of the Under pattern is that all Slurm components run as K8s Pods. There is no dedicated Slurm head node. slurmctld runs as a StatefulSet, slurmd instances run as a NodeSet CRD, and the Slinky Operator manages their lifecycle.

What this changes compared to traditional Slurm:

• slurmctld: host daemon → K8s Pod (StatefulSet with PersistentVolume)
• slurmd: host daemon on each worker → K8s Pod (NodeSet CRD, one per GPU worker)
• State persistence: local disk → hostPath PV (/var/spool/slurmctld)
• Failure recovery: manual or systemd → K8s automatic Pod restart
• Scaling: manual node addition → change NodeSet replicas
• GPU detection: manual gres.conf → AutoDetect=nvidia (automatic)

The Slinky Operator provides 6 CRDs (NodeSet, Controller, LoginSet, RestAPI, Accounting, Token) and deploys via Helm chart from the OCI registry at ghcr.io/slinkyproject/charts/slurm.

Helm Values: NodeSet and GPU GRES

The entire Slurm cluster is deployed as a single Helm release. GPU GRES configuration is embedded in the NodeSet definition:

Three things to note:

• nvidia.com/gpu: “1” makes K8s nvidia-device-plugin assign a physical GPU to each slurmd Pod
• Gres: gpu:tesla:1 configures the Slurm GRES definition; AutoDetect=nvidia discovers the GPU inside the Pod
• fsGroup: 401 matches the Slurm user UID in Slinky’s 25.11 container image (different from typical Slurm installs that use UID 64030 or 901)

When a slurmd Pod starts, K8s nvidia-device-plugin and Slurm GRES both point to the same physical GPU. Slurm jobs execute within the Pod boundary and can only access what K8s allocated.

slurmctld State Persistence

slurmctld runs as a StatefulSet with a hostPath PV for /var/spool/slurmctld:

The Slinky Helm chart auto-creates a PVC named statesave-slurm-controller-0. The PV uses claimRef to pre-bind to it. nodeAffinity pins slurmctld to the control plane node so it always recovers its state after restart.

HPC Job Submission: sbatch via slurmctld Pod

Since slurmctld runs as a Pod (not a host daemon), sbatch commands are executed inside the slurmctld Pod via kubectl exec:

kubectl exec -n slurm $(kubectl get pods -n slurm \ -l app.kubernetes.io/name=slurmctld \ -o jsonpath=’{.items[0].metadata.name}’) \ — sbatch — wait /mnt/share/shared/jobs/run.sh

An OpenFOAM sbatch script:

!/bin/bash

SBATCH — job-name=Bench_8M_N2_NT2

SBATCH — nodes=2

SBATCH — ntasks-per-node=2

SBATCH — output=log_output.log

SBATCH — error=log_error.log

source /mnt/share/shared/tools/openfoam12/etc/bashrc blockMesh decomposePar mpirun -np $SLURM_NTASKS icoFoam -parallel reconstructPar

#SBATCH directives, MPI execution, and shared filesystem access (/mnt/share) all work exactly as in traditional Slurm. Standard Slurm job scripts require no modification — only the submission path (via kubectl exec) differs from a conventional Slurm setup.

Both slurmctld and slurmd Pods mount /mnt/share/shared via hostPath, providing the shared filesystem semantics that MPI jobs require.

AI Job Submission: Standard kubectl

K8s native AI jobs are submitted with standard kubectl apply. No bridging layer (slurm-bridge, interLink) is involved:

This is a pure K8s native job. Unlike the Converged pattern, there is no schedulerName: slurm-bridge-scheduler. Unlike the Distant pattern, there are no slurm.vk.io/flags annotations. GPU allocation goes through K8s nvidia-device-plugin as usual.

The catch: if slurmd Pods are occupying all GPUs, K8s native jobs stay Pending until KEDA scales slurmd down. This dynamic handoff is the core mechanism described next.

Scheduling Flow Comparison

HPC jobs (sbatch):

• Submission: kubectl exec → sbatch
• Scheduler: slurmctld Pod
• GPU allocation: Slurm GRES (inside Pod)
• Execution: slurmd Pod
• Runtime: containerd (Pod container)
• Visible in squeue: yes

AI jobs (kubectl):

• Submission: kubectl apply -f job.yaml
• Scheduler: kube-scheduler
• GPU allocation: nvidia-device-plugin
• Execution: Job Pod (kubelet)
• Runtime: containerd (Pod container)
• Visible in squeue: no

An important asymmetry: K8s native jobs do not appear in Slurm’s job queue. In the Converged and Distant patterns, all jobs flow through slurmctld and are visible in squeue/sacct. In Under, the two scheduling paths are independent. Unified monitoring requires Prometheus + Grafana to cover both Slurm metrics (via Slinky) and K8s metrics (via kube-state-metrics).

Running HPC + AI Simultaneously

We validated hybrid execution with several mixed workload patterns.

HPC parallel execution: Two OpenFOAM instances submitted via sbatch, each using 2 nodes and 2 MPI tasks. slurmd Pods scale up to handle the jobs, with GPUs allocated through Slurm GRES.

Mixed HPC + AI execution: OpenFOAM (sbatch) and MLPerf Training (PyTorchJob) running simultaneously. The slurmd Pod and PyTorchJob Pod are placed on different GPU workers. nvidia-device-plugin ensures they use different physical GPUs.

Heavy mixed load: OpenFOAM×2 + MLPerf Training×2 + Inference×1 submitted simultaneously. When GPU requests exceed the 4 available GPUs, both schedulers queue excess jobs. As preceding jobs complete, subsequent jobs are dispatched automatically.

Results: Across all tested patterns, sbatch and kubectl jobs executed correctly alongside each other. GPU metrics (dcgm-exporter → Prometheus → AWS Managed Prometheus) confirmed GPU exclusivity — no double-booking occurred in any test run.

KEDA Dynamic GPU Allocation

KEDA is the mechanism that makes Under hybrid execution possible. It monitors Slurm’s job queue via Prometheus and scales slurmd Pods accordingly.

ScaledObject Definition

The scaling flow:

1. Idle (idleReplicaCount: 0): No slurmd Pods running. All 4 GPUs available for K8s native jobs.
2. Slurm job submitted: slurm_partition_jobs_pending rises above 0. KEDA scales slurmd Pods from 0 to at least minReplicaCount: 1.
3. Jobs running: slurm_partition_nodes_alloc stays above 0, preventing premature scale-down.
4. Jobs complete + cooldown (300s): Metrics drop to 0. After 300 seconds, KEDA scales slurmd back to 0. GPUs return to K8s.

Slinky Metrics for KEDA

KEDA triggers consume Slinky’s Prometheus metrics. Setting controller.metrics.enabled: true in the Helm values exposes a /metrics/partitions endpoint on slurmctld’s port 6817:

Key metrics:

• slurm_partition_jobs_pending: Pending job count (scale-up trigger)
• slurm_partition_nodes_alloc: Allocated node count (prevents scale-down during execution)
• slurm_partition_jobs_max_job_nodes_nohold: Max requested nodes per job (handles multi-node jobs)

Comparison with Converged and Distant Patterns

The three patterns solve the same problem — hybrid K8s/Slurm GPU scheduling — through fundamentally different mechanisms:

• GPU worker daemons: Converged runs kubelet + slurmd on every worker. Distant runs slurmd only (no kubelet). Under runs kubelet + slurmd Pods.
• GPU isolation: Converged uses Dual Partition (static separation). Distant is structurally impossible to double-book (no nvidia-device-plugin on workers). Under uses Pod-level resource requests (dynamic separation via KEDA).
• Slurm head: Converged and Distant use a physical node. Under uses a Pod.
• Container runtime: Converged and Under use containerd. Distant uses Apptainer.
• K8s-Slurm bridge: Converged uses slurm-bridge-scheduler. Distant uses interLink (Virtual Kubelet). Under needs no bridge — the two paths are independent.
• K8s ecosystem: Converged and Under support full K8s features (DaemonSet, Service, Ingress). Distant is limited to Pod submission (workers have no kubelet).
• Job management: Converged and Distant unify all jobs in squeue/sacct. Under has separate tracking — Slurm jobs in squeue, K8s jobs in kubectl.
• Best fit: Converged for K8s-centric integration. Distant for HPC-centric environments. Under for cloud-native operations with minimal Slurm infrastructure.

The Converged pattern says: “add Slurm to your K8s cluster.” The Distant pattern says: “add a K8s interface to your Slurm cluster.” The Under pattern says: “run Slurm inside your K8s cluster.”

Prerequisites and Setup Notes

slurmctld statesave directory permissions

The Slurm user UID in Slinky’s 25.11 container image is 401, which differs from typical Slurm installations (UID 64030 or 901). The hostPath directory must be chown 401:401, and the Helm values must set fsGroup: 401.

Slinky chart PVC binding bug

The controller.persistence.existingClaim field in Slinky Helm chart v1.0.0 does not work due to a template bug. Instead, use PV claimRef to pre-bind to the chart’s auto-created PVC name (statesave-slurm-controller-0).

cert-manager webhook timing

The Slinky Operator depends on cert-manager for webhook certificates. A cert-manager Pod being Ready does not guarantee the webhook API is functional. Before installing the operator, verify that the webhook health endpoint (/healthz) returns ok.

login Pod requires SSSD/LDAP

Slinky’s login Pod CrashLoops without SSSD/LDAP configuration. For evaluation, disable it (loginsets.slinky.enabled: false) and use kubectl exec into the slurmctld Pod directly.

KEDA cooldownPeriod tuning

A short cooldown causes unnecessary slurmd Pod restarts between consecutive Slurm jobs. Pod startup + GPU allocation + Slurm node registration takes 30–60 seconds. Set cooldownPeriod to 300 seconds or more to avoid thrashing.

Conclusion

The Under pattern provides hybrid K8s/Slurm GPU scheduling with three characteristics:

1. Pod-based Slurm: slurmctld and slurmd run as K8s Pods, eliminating dedicated Slurm infrastructure. Failure recovery, scaling, and GPU detection are all managed by K8s.
2. Dynamic GPU sharing: KEDA scales slurmd Pods based on Slurm job queue metrics. When Slurm is idle, GPUs are available for K8s native workloads. When Slurm jobs arrive, slurmd Pods scale up and claim GPUs.
3. Independent scheduling paths: HPC jobs go through slurmctld, AI jobs go through kube-scheduler. GPU exclusivity is guaranteed by K8s nvidia-device-plugin at the Pod level — no partition engineering or structural constraints needed.

The design principle: the Converged pattern separates GPU pools with static partitions. The Distant pattern eliminates double-booking structurally by keeping GPU workers Slurm-only. The Under pattern shares GPUs dynamically through Pod lifecycle management — at the cost of time-division rather than simultaneous GPU sharing, and separate job tracking for each path.

Choose Under when you want Slurm’s HPC capabilities without maintaining separate Slurm infrastructure, and your workload pattern tolerates time-sharing GPUs between HPC and AI phases.

—

References

• Slinky Project GitHub: https://github.com/SlinkyProject/slinky
• Slinky Operator: https://github.com/SlinkyProject/slurm-operator
• KEDA (Kubernetes Event-Driven Autoscaling): https://keda.sh/
• NVIDIA Device Plugin for Kubernetes: https://github.com/NVIDIA/k8s-device-plugin
• Slurm GRES (Generic Resources): https://slurm.schedmd.com/gres.html