[Distant]Hybrid K8s-Slurm GPU Scheduling with interLink and Apptainer

Introduction

The Distant pattern keeps Kubernetes and Slurm operationally separate: K8s runs only as a control plane, GPU workers run only slurmd and Apptainer, and interLink bridges the gap by converting Pod specs into sbatch scripts. This article shows how we built a hybrid environment where HPC users submit via sbatch and AI users submit via kubectl — both landing in the same Slurm queue on the same GPU cluster.

The Problem: Two User Communities, One GPU Cluster

In AI/HPC environments, different teams want different submission interfaces:

HPC users want sbatch, #SBATCH directives, MPI, and shared filesystems. Their workflows are built around OpenFOAM, NAS Parallel Benchmarks, STREAM, and IOR. Asking them to learn kubectl is a non-starter.

AI users want kubectl, Kubeflow Training Operator, PyTorchJob CRDs, and the K8s ecosystem. Their workflows are built around MLPerf Training, distributed PyTorch, and Triton inference serving.

The naive approach — separate GPU clusters for each team — wastes resources. When HPC jobs finish early, those GPUs sit idle while AI jobs queue. What you want is a single GPU pool that both teams can submit to through their native interfaces.

The Distant Architecture

The solution is to consolidate K8s control plane and Slurm head on a single node, while keeping GPU workers under Slurm-only management. interLink’s Virtual Kubelet translates kubectl submissions into sbatch commands, so both paths converge on the same slurmctld.

Two key design decisions:

First, GPU workers run slurmd only — no kubelet. K8s exists only as a control plane and never directly manages GPU nodes. Slurm GRES is the sole GPU allocation mechanism. This eliminates the GPU double-booking problem that the Converged pattern must solve with dual partitions.

Second, kubectl submissions also go through sbatch. The interLink Slurm Sidecar converts Pod specs into sbatch scripts and submits them to slurmctld. Both HPC jobs (direct sbatch) and AI jobs (kubectl → interLink → sbatch) enter the same slurmctld queue.

Why GPU Double-Booking Doesn’t Happen

In the Converged pattern, GPU workers run both kubelet and slurmd. This creates two independent GPU tracking systems — K8s nvidia-device-plugin and Slurm GRES — that don’t know about each other’s allocations. Without careful partition engineering, the same GPU gets assigned to two jobs.

The Distant pattern avoids this entirely. GPU workers have no kubelet, so nvidia-device-plugin never runs. Slurm GRES is the only GPU allocation mechanism. Even interLink-originated jobs go through sbatch → slurmctld → GRES allocation. Double-booking is structurally impossible, and only a single Slurm partition is needed.

Test Environment

We validated this architecture on AWS EC2 with 3 nodes:

• Control node (t3.large): K8s control plane + slurmctld + interLink API + Slurm Sidecar + NFS server
• GPU Worker 1 (g4dn.xlarge): slurmd + Apptainer + Tesla T4
• GPU Worker 2 (g4dn.xlarge): slurmd + Apptainer + Tesla T4

Software stack: K8s v1.29.15, interLink / Virtual Kubelet 0.6.1-pre4, Slurm 25.11.0, Apptainer 1.4.5, NVIDIA Driver 580.95.05, Kubeflow Training Operator v1.8.1

How interLink Converts Pods to Slurm Jobs

interLink operates as a three-layer translation pipeline:

The Virtual Kubelet bridges K8s API and interLink API. The interLink API Server normalizes requests. The Slurm Sidecar builds the actual sbatch command.

interLink Configuration

InterlinkConfig.yaml (API Server)

InterlinkAddress: http://0.0.0.0 InterlinkPort: 3000 SidecarURL: http://localhost SidecarPort: 4001 ExportPodData: true DataRootFolder: /var/interlink/data

SlurmConfig.yaml (Slurm Sidecar)

SidecarPort: “4001” SbatchPath: “/usr/bin/sbatch” ScancelPath: “/usr/bin/scancel” SqueuePath: “/usr/bin/squeue” SinfoPath: “/usr/bin/sinfo” SingularityPath: “/usr/bin/apptainer” DataRootFolder: “/var/interlink/data” BashPath: “/bin/bash” Namespace: “interlink”

Note that SingularityPath points to the Apptainer binary. interLink uses the Singularity-compatible API internally, and Apptainer (the Singularity fork) is fully command-compatible.

Virtual Kubelet ConfigMap

The Virtual Kubelet runs with hostNetwork: true and connects to the interLink API Server on localhost. The Accelerators field declares how many GPUs the virtual node advertises to the K8s scheduler.

slurm.conf: Single Partition

ClusterName=distant SlurmctldHost=distant-k8s-control(10.0.7.10)

AuthType=auth/munge SchedulerType=sched/backfill SelectType=select/cons_tres SelectTypeParameters=CR_Core

GresTypes=gpu

Single partition: HPC and interLink jobs share the same queue

PartitionName=gpu Nodes=ip-10–0–7–21,ip-10–0–7–22 Default=YES MaxTime=INFINITE State=UP

GPU workers (Slurm slurmd + Apptainer only, no K8s kubelet)

NodeName=ip-10–0–7–21 NodeAddr=10.0.7.21 CPUs=4 RealMemory=15500 Gres=gpu:1 State=UNKNOWN NodeName=ip-10–0–7–22 NodeAddr=10.0.7.22 CPUs=4 RealMemory=15500 Gres=gpu:1 State=UNKNOWN

Unlike the Converged pattern’s Dual Partition setup, only one partition is needed. Since GPU workers have no kubelet, there is no K8s nvidia-device-plugin competing with Slurm GRES. Both sbatch and interLink jobs land in the same gpu partition.

HPC Job Submission: Direct sbatch

OpenFOAM MPI parallel computation submitted via sbatch:

!/bin/bash

SBATCH — job-name=Bench_8M_N2_NT2

SBATCH — nodes=2

SBATCH — ntasks-per-node=2

SBATCH — output=log_output.log

SBATCH — error=log_error.log

source /opt/openfoam12/etc/bashrc blockMesh decomposePar mpirun -np $SLURM_NTASKS icoFoam -parallel

sbatch — wait run.sh

HPC users submit jobs the way they always have. MPI execution, shared filesystem (NFS) access, #SBATCH directives — the entire Slurm-native workflow is preserved.

AI Job Submission: kubectl + interLink

K8s users submit Pods via kubectl apply. interLink converts the Pod spec to an sbatch script and executes it as an Apptainer container on the GPU worker.

Three additions are required for interLink routing:

• nodeSelector routes the Pod to the virtual node (interlink-slurm)
• tolerations accepts the virtual node’s taint
• slurm.vk.io/flags annotation passes Slurm-specific options (GPU count, time limit)

Important: The actual GPU allocation is controlled by the slurm.vk.io/flags annotation (— gres=gpu:1), not by resources.limits. The nvidia.com/gpu: 1 limit is used by the K8s scheduler to check virtual node capacity, but Slurm ignores it. Keep both values consistent.

Pod-to-sbatch conversion summary:

• image → apptainer pull docker://… (converted to SIF format)
• slurm.vk.io/flags → #SBATCH directives (actual GPU allocation)
• command + args → apptainer exec — nv image.sif /bin/bash -c “…”
• nvidia.com/gpu → used by K8s scheduler only, not passed to Slurm

The first Docker-to-SIF conversion takes about 10 minutes for large images (e.g., nvcr.io/nvidia/pytorch:24.12-py3). Subsequent runs use the cached SIF and start immediately.

Container Runtime: Apptainer vs containerd

A distinctive feature of the Distant pattern is that all container execution uses Apptainer, regardless of submission path. Here are the typical differences in this configuration:

• Runtime: containerd (K8s native) vs Apptainer (HPC native)
• Image format: OCI Image vs SIF (Singularity Image Format)
• GPU access: nvidia-container-toolkit vs apptainer exec — nv
• Privileges: root is common in containerd vs unprivileged by default in Apptainer
• MPI integration: configured within Pod vs directly leverages host MPI
• Filesystem access: volume mounts vs direct host filesystem references

Apptainer is designed for HPC environments. It runs unprivileged by default and integrates naturally with host filesystems and MPI libraries.

Scheduling Flow Comparison

HPC jobs (sbatch):

• Submission: sbatch run.sh
• Scheduler: slurmctld directly
• GPU allocation: Slurm GRES
• Execution: slurmd
• Runtime: host process or Apptainer
• Partition: gpu (Default)
• Visible in squeue: yes

AI jobs (kubectl):

• Submission: kubectl apply -f job.yaml
• Scheduler: kube-scheduler → interLink → slurmctld
• GPU allocation: Slurm GRES (via interLink)
• Execution: slurmd
• Runtime: Apptainer
• Partition: gpu (Default)
• Visible in squeue: yes

Both paths allocate GPUs through Slurm GRES, and both are executed by slurmd. interLink-originated jobs appear in squeue and sacct just like native Slurm jobs. Since we use sched/backfill with accounting_storage/slurmdbd, the same scheduling policy and accounting apply to all jobs regardless of submission path.

Running HPC + AI Simultaneously

We validated hybrid execution with several mixed workload patterns. Here are representative examples.

HPC parallel execution: Two OpenFOAM instances submitted via sbatch, each using 2 nodes and 2 MPI tasks. This exercises pure HPC workloads on shared filesystem as a baseline.

Mixed HPC + AI execution: Three jobs running simultaneously:

• OpenFOAM (MPI parallel CFD) — submitted via sbatch
• MLPerf Training (PyTorch BERT fine-tuning) — submitted via kubectl → interLink
• MLPerf Inference (Triton serving) — submitted via kubectl → interLink

All three jobs enter the same slurmctld queue. OpenFOAM primarily consumes CPU and memory, while MLPerf uses GPUs. Slurm’s TRES management tracks CPU, memory, and GPU holistically, preventing over-allocation.

Heavy mixed load: OpenFOAM×2 + MLPerf Training×2 + Inference×1 submitted simultaneously. With only 2 GPU workers (2 GPUs), the job count exceeds available resources. Slurm’s backfill scheduler queues excess jobs and dispatches them as preceding jobs complete.

Results: Across all tested patterns, mixed sbatch and interLink submissions executed correctly. Unified job tracking via sacct, metrics collection via Prometheus + Pushgateway, and GPU exclusivity all functioned as expected. interLink-originated jobs appeared in squeue identically to native Slurm jobs, confirming operational transparency.

Benefits of the Unified Control Node

Consolidating K8s control plane and Slurm head onto a single node provides several advantages:

Simplified orchestration: Shell scripts on the control node can run both sbatch — wait and kubectl apply in parallel, then collect accounting data via sacct. No SSH hops to remote hosts.

Zero network overhead for interLink: The entire Virtual Kubelet → interLink API → Slurm Sidecar → slurmctld communication path stays on localhost. No firewall rules or cross-node latency.

NFS origin: The control node serves as the NFS server, exporting /var/interlink to GPU workers. Pod data (logs, output files) generated by interLink is accessible from workers via NFS.

Comparison with the Converged Pattern

The Converged pattern (using slurm-bridge) also achieves hybrid K8s/Slurm execution, but through a fundamentally different architecture:

• GPU worker daemons: Converged runs kubelet + slurmd on every worker. Distant runs slurmd only.
• Container runtime: Converged uses containerd for K8s jobs, host processes for Slurm jobs. Distant uses Apptainer for everything.
• K8s-to-Slurm bridge: Converged uses slurm-bridge-scheduler. Distant uses interLink (Virtual Kubelet).
• GPU double-booking risk: Converged has it (solved by Dual Partitions). Distant doesn’t (structurally impossible).
• K8s ecosystem: Converged supports full K8s features (DaemonSet, Service, Ingress). Distant is limited to Pod submission (GPU workers have no kubelet).
• Best fit: Converged for K8s-centric AI/HPC integration. Distant for HPC-centric environments adding a K8s interface.

The Converged pattern says: “add Slurm to your K8s cluster.” The Distant pattern says: “add a K8s interface to your Slurm cluster.”

Prerequisites and Setup Notes

interLink version compatibility

interLink is an actively developed CNCF Sandbox project. Configuration formats and APIs change between versions. We used the 0.6.x series. The Virtual Kubelet ConfigMap format (InterlinkURL, Accelerators, etc.) is version-dependent — check the release notes.

ephemeral-storage patch

interLink 0.6.x does not natively support ephemeral-storage resources. After the virtual node registers, you need to patch it manually:

kubectl patch node interlink-slurm — type merge \

-p ‘{“status”:{“capacity”:{“ephemeral-storage”:”100Gi”},

“allocatable”:{“ephemeral-storage”:”100Gi”}}}’ \

-- subresource=status

Docker-to-SIF conversion caching

The first conversion of a Docker image to SIF format takes ~10 minutes for large images. Pre-convert images in production to avoid cold-start delays.

NFS export configuration

interLink’s Pod data directory (/var/interlink/data) is shared to GPU workers via NFS. The NFS export must include no_root_squash — otherwise Apptainer may fail to access Pod data on the workers.

Conclusion

The Distant pattern provides hybrid K8s/Slurm GPU scheduling with three guarantees:

1. Unified GPU management: GPU workers run slurmd only, making Slurm GRES the sole GPU allocator. Double-booking is structurally impossible — no dual partitions needed.
2. Dual submission paths: HPC users keep sbatch with full MPI/shared-filesystem workflows. AI users get kubectl with Kubeflow and PyTorchJob support.
3. Unified accounting: Every job flows through slurmctld, so sacct provides a single view of all job history regardless of submission method.

The design principle: the Converged pattern adds Slurm scheduling to K8s nodes. The Distant pattern adds a K8s submission interface to a Slurm cluster. By keeping GPU management exclusively under Slurm, the Distant pattern avoids the dual-management complexity that Converged must solve with partition engineering — at the cost of limiting K8s features to Pod submission only.

Choose Distant when your GPU cluster is fundamentally HPC-centric and you want to offer a K8s interface without restructuring the existing Slurm infrastructure.

—

References

• interLink GitHub: https://github.com/interlink-hq/interlink
• interLink Documentation: https://interlink-hq.github.io/interlink/
• Virtual Kubelet: https://github.com/virtual-kubelet/virtual-kubelet
• Apptainer (Singularity): https://apptainer.org/
• Slurm GRES (Generic Resources): https://slurm.schedmd.com/gres.html