[Compare]Validation C: Same-Condition E2E Comparison (6/6 Patterns)

Introduction

Kobayashi — Senior Architect, Tas Design Group.

This article reports the results of measuring and fairly comparing E2E (End-to-End) time across 6 patterns for integrating Kubernetes and Slurm in HPC environments, using the same node, same image, and same workload.

Purpose

In Validation B, measurements were under mixed conditions (different nodes, different versions), making fair comparison impossible. This validation unifies the following conditions to quantify the true E2E overhead of all 6 patterns.

• Same node: ip-10–0–3–21 (Tesla T4, g4dn.xlarge)
• Same image: yolo-bench:v2
• Same PyTorch: 2.5.1+cu121
• Same workload: YOLOv8n inference (bus.jpg)
• Warm 3 measurements: Cold excluded, model DL completed

Measurement Conditions

Common Conditions

Pattern-specific Configuration

Final Results (All 6 Patterns Confirmed)

E2E Time Ranking (Warm Average)

Visualization

E2E Time Comparison (Shorter is Better)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Over (k3s)               |████████| 4.02s (+0%)
Under (Slinky)           |█████████| 4.67s (+16%)
Adjacent-K8s             |██████████| 4.77s (+19%)
Adjacent-Slurm (Docker)  |██████████| 5.10s (+27%)
Distant (interLink)      |█████████████████| 8.64s (+115%)
Converged (slurm-bridge) |██████████████████████| 11.33s (+182%)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Individual Pattern Details

Over (Fastest: 4.02s)

k3s launched on ip-10–0–3–21 with GPU access enabled via nvidia-container-runtime.

Configuration: k3s (single-node) + nvidia-container-runtime + NVIDIA device plugin

Reasons for Fastest:

• k3s lightweight scheduler is fast
• Direct GPU access via nvidia-container-runtime
• Single-node optimization is effective

Note: Conflicts with existing K8s cluster’s kubelet/kube-proxy, so existing services were stopped during measurement.

Under (4.67s)

Python script executed directly inside Slinky Operator’s slurmd Pod.

Configuration: Custom slurmd image (yolo-slurmd:v6) = slurmd + PyTorch 2.5.1+cu121 + ultralytics

Reasons for Fast:

• slurmd is resident, no container startup overhead
• Inference starts with only Python process startup
• Slinky’s K8s integration overhead is minimal

Adjacent-K8s (4.77s)

yolo-bench:v2 container executed as K8s Job (default-scheduler).

Characteristics:

• Standard K8s Job execution path
• Stable configuration with containerd + NVIDIA device plugin
• Scheduling overhead is approximately 0.1s (t_ack — t_submit)
• Very stable with standard deviation of 0.00

Adjacent-Slurm (5.10s)

Docker container (yolo-bench:v2) executed via sbatch.

Characteristics:

• sbatch Ack is fast at ~7ms
• Docker container startup is main overhead cause

Reference (Apptainer path): E2E 7.86s, latency 1999ms, approximately 2.7s slower than Docker.

Distant (8.64s)

K8s Pod executed as Slurm job via interLink Virtual Kubelet.

Configuration: interLink VK 0.4.0-pre2 + interlink-sidecar-slurm + Singularity

Characteristics:

• VK → sidecar → Slurm conversion overhead is minimal at ~0.15ms
• Latency is high at ~2s because Singularity container startup overhead is included
• Practical at +0.17s (+2%) compared to direct sbatch (8.47s)

Reference (direct sbatch): E2E 8.47s, latency 2309ms (reference value without interLink)

Converged (11.33s)

K8s Pod scheduled as Slurm job via slurm-bridge-scheduler.

Overhead Analysis:

• kubectl apply: ~162ms
• slurm-bridge + Slurm execution: ~9.98s
• Inference latency: ~1.19s

Pod → Slurm Job conversion by slurm-bridge incurs approximately 6.5s overhead (difference from Adjacent-K8s).

Overhead Structure Analysis

E2E Time Breakdown

Breaking down E2E time of 6 patterns into “inference latency” and “overhead.”

Visualization (Breakdown)

E2E Breakdown (Inference Latency + Overhead)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Over       |████|██████|  Inference 1.21s + OH 2.81s = 4.02s
Under      |████|████████|  Inference 1.19s + OH 3.48s = 4.67s
Adj-K8s    |████|█████████|  Inference 1.14s + OH 3.63s = 4.77s
Adj-Slurm  |████|██████████|  Inference 1.19s + OH 3.91s = 5.10s
Distant    |████████|██████████████|  Inference 2.05s + OH 6.59s = 8.64s
Converged  |████|████████████████████████████|  Inference 1.19s + OH 10.14s = 11.33s
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Comparison with Validation A

Comparing Validation A (steady-state FPS) and this validation (E2E) under identical conditions.

Key Points:

• Steady-state performance (FPS): Approximately 77–83 FPS across all patterns, maximum difference 8%
• E2E time: 4.02–11.33 seconds, maximum difference 182%
• Conclusion: Steady-state performance is nearly equivalent, but E2E differences are due to “scheduling + container startup” overhead

Key Findings

1. Over is Fastest (4.02s)

Combination of k3s lightweight scheduler and nvidia-container-runtime is most efficient. However, coexistence with existing K8s cluster requires consideration.

2. Under is Second (4.67s)

Container startup overhead avoided by resident slurmd. K8s integration possible while leveraging Slurm features.

3. Adjacent-K8s / Adjacent-Slurm Converge to 4.8–5.1s

Standard K8s Job and Slurm + Docker E2E are similar. Latency is also equivalent at ~1.2s.

4. Converged Adds +6.5s from slurm-bridge Conversion

The 6.5s difference between Adjacent-K8s (4.77s) and Converged (11.33s) is slurm-bridge overhead. Design decision needed to accept overhead in exchange for functional value.

5. Distant (interLink) is Essentially Equivalent to Direct sbatch

interLink route (8.64s) is +0.17s (+2%) compared to direct sbatch (8.47s). VK → sidecar → Slurm conversion overhead is minimal, practical for use cases submitting Slurm jobs from K8s API.

6. Singularity Initialization is Main Cause of 2s+ Latency

Distant pattern latency (2053ms) is higher than other patterns (~1200ms) because Singularity exec initialization overhead is included. Can be reduced with resident/cache.

Discussion

Scheduling Layer Lightness Directly Impacts E2E

E2E ranking correlates with scheduling layer lightness.

Converged / interLink Conversion is Tradeoff for Functional Value

Converged and Distant (interLink) are inferior in E2E but provide the following functional value.

In production, select based on balance between functional requirements and overhead.

Distant Can Be Accelerated with Singularity Resident/Cache

Distant pattern latency (2053ms) is dominated by Singularity initialization time. Can be reduced with:

• Singularity instance resident
• Image cache optimization
• Pre-extraction of SIF images

Pattern Selection Guidelines

interLink Investigation Case Study

Introducing troubleshooting encountered during Distant pattern measurement.

1. Singularity Image Path Issue

Symptom: docker:// prefix applied, Singularity cannot resolve

Solution: Specify absolute path directly in Pod spec image field

image: /home/ubuntu/containers/yolo-bench_v2.sif

2. ServiceAccount Token Mount Failure

Symptom: mount source .../token doesn't exist with exitCode 255

Solution: Disable ServiceAccount token mount in Pod spec

automountServiceAccountToken: false

3. Virtual Kubelet CrashLoopBackOff

Symptom: VK Pod crashes repeatedly, slurm-virtual-node is NotReady

Solution: Confirmed stable operation with VK 0.4.0-pre2. Secret watch error remains but does not affect Pod execution.

Conclusion

This validation quantified E2E for 6 patterns under identical conditions (ip-10–0–3–21 + yolo-bench:v2 + PyTorch 2.5.1+cu121 + Warm 3 runs) as follows.

Key Findings:

1. Steady-state performance is equivalent across all patterns (~30 FPS / latency ~1.2s)
2. E2E differences are due to startup overhead (scheduling + container startup)
3. Over / Under are fastest (benefit of resident processes)
4. Converged has large overhead in exchange for functional value
5. Distant (interLink) is equivalent to direct sbatch (conversion overhead +2%)

Related Documents

• Validation A: Steady-State Inference Performance 5-Pattern Same-Condition Comparison
• Validation B: E2E Overhead Under Production Conditions
• Detailed Result Data