Building a High-Performance Computing (HPC) Cluster from Scratch: A Step-by-Step Guide — Part 6: Slurm Workload Manager Setup

In Part 5, we configured LMOD that allows users to easily load different versions of software without conflicts across the cluster.

In this part, we install and configure Slurm, the most widely used open-source workload manager in HPC. Slurm handles job scheduling, resource allocation (CPUs, GPUs, memory), queuing, and fair-share policies across all 12 compute nodes.

Why Slurm?

Slurm (Simple Linux Utility for Resource Management) is the de-facto standard in research and academic HPC. It excels at:

• Allocating CPUs and GPUs efficiently
• Supporting partitions (our High-Density, Balanced GPU, and CPU-only tiers)
• Fair-share scheduling and Quality of Service (QoS)
• Integration with MUNGE (already set up in Part 3)

Step 1: Create the Slurm System User (All Nodes)

This user must exist with the same UID/GID on every node.

sudo groupadd -g 100 slurm
sudo useradd -m -c "Slurm Workload Manager" -d /var/lib/slurm -u 100 -g slurm -s /bin/false slurm

• groupadd -g 100: Creates a group with a specific Group ID (any number below 1000 is a common convention for system services).
• useradd: Creates the slurm user with home directory /var/lib/slurm, no login shell, and matching UID.

Verify:

id slurm

Step 2: Slurm Installation and Configuration on the HeadNode

1. Install Slurm Controller

sudo dnf install slurm-slurmctld -y

2. Create Required Directories

sudo mkdir -p /etc/slurm /var/spool/slurmctld /var/log/slurm
sudo chown slurm:slurm /var/spool/slurmctld /var/log/slurm

3. Create Main Configuration File — /etc/slurm/slurm.conf

sudo nano /etc/slurm/slurm.conf

Note:

To check on the hardware config on compute nodes, just run the following command and copy the result into slurm.conf file on Headnode

sudo slurmd -C

Here is a solid base configuration (customize node names and resources as needed):

ClusterName=researchcluster
SlurmctldHost=headnode
GresTypes=gpu

ProctrackType=proctrack/cgroup
TaskPlugin=task/cgroup
SelectType=select/cons_res
SelectTypeParameters=CR_Core_Memory

SchedulerType=sched/backfill
# SchedulerType=sched/builtin   # Use for simple testing

SlurmctldPidFile=/var/run/slurmctld.pid
SlurmdPidFile=/var/run/slurmd.pid
SlurmdSpoolDir=/var/spool/slurmd
StateSaveLocation=/var/spool/slurmctld
TmpFS=/stash

# Logging
SlurmctldDebug=info
SlurmctldLogFile=/var/log/slurm/slurmctld.log
SlurmdDebug=info
SlurmdLogFile=/var/log/slurm/slurmd.log

# Node Definitions (examples - adapt to your actual hostnames)
NodeName=headnode CPUs=64 RealMemory=500000 State=UNKNOWN

# High-Density GPU Nodes (4 GPUs each)
NodeName=high-density-[01-02] CPUs=112 RealMemory=1030000 Gres=gpu:nvidia:4 State=UNKNOWN

# Balanced GPU Nodes (1 GPU each)
NodeName=gpu-standard-[01-05] CPUs=112 RealMemory=1030000 Gres=gpu:nvidia:1 State=UNKNOWN

# CPU-only Nodes
NodeName=cpu-only-[01-05] CPUs=64 RealMemory=786000 State=UNKNOWN

# Partitions (Tiers)
PartitionName=high-density Nodes=high-density-[01-02] Default=NO MaxTime=7-00:00:00 State=UP PriorityTier=20
PartitionName=gpu-standard Nodes=gpu-standard-[01-05] Default=YES MaxTime=INFINITE State=UP PriorityTier=10
PartitionName=cpu-only Nodes=cpu-only-[01-05] Default=NO MaxTime=14-00:00:00 State=UP PriorityTier=5

Key directives explained:

• GresTypes=gpu: Enables Generic Resource Scheduling for GPUs.
• SelectTypeParameters=CR_Core_Memory: Allows core + memory based allocation.
• PartitionName: Defines our three tiers with different priorities and time limits.

4. Create cgroup Configuration

sudo nano /etc/slurm/cgroup.conf

CgroupAutomount=yes
ConstrainCores=yes
ConstrainRAMSpace=yes
ConstrainDevices=yes
AllowedRAMSpace=90 #adjust to your needs

Step 3: GPU Configuration (GRES) on Compute Nodes

On every GPU node, create the GRES file:

sudo nano /etc/slurm/gres.conf

For High-Density nodes (4 GPUs):

Name=gpu Type=nvidia File=/dev/nvidia[0-3]

For Balanced GPU nodes (1 GPU):

Name=gpu Type=nvidia File=/dev/nvidia0

Step 4: Firewall Configuration (All Nodes)

On HeadNode:

sudo firewall-cmd --permanent --add-port=6817/tcp
sudo firewall-cmd --reload

On Compute Nodes:

sudo firewall-cmd --permanent --add-port=6818/tcp
sudo firewall-cmd --reload

Step 5: Copy Configuration Files to All Compute Nodes

From the HeadNode, copy the files to every compute node (use a loop or Ansible for efficiency):

for node in gpu-standard-01 gpu-standard-02 ... cpu-only-01 ...; do
    scp /etc/slurm/slurm.conf $node:/etc/slurm/
    scp /etc/slurm/cgroup.conf $node:/etc/slurm/
    scp /etc/slurm/gres.conf $node:/etc/slurm/   # Only for GPU nodes
done

Step 6: Start Slurm Services

On HeadNode:

sudo systemctl enable --now slurmctld
sudo systemctl status slurmctld

On All Compute Nodes:

sudo dnf install slurm-slurmd -y
sudo systemctl enable --now slurmd
sudo systemctl status slurmd

Step 7: Verification Commands

From the HeadNode, run these important checks:

sinfo                    # View all nodes and their state
sinfo -N -l              # More detailed node info
scontrol show nodes      # Detailed node configuration
scontrol show partition  # View partitions

A healthy output should show all nodes as idle or mixed.

Test submitting a simple job:

srun --partition=gpu-standard --gres=gpu:1 --nodelist=gpu-standard-01 hostname

Best Practices

• Keep slurm.conf and cgroup.conf identical across all nodes.
• Use consistent node naming conventions (e.g., high-density-01).
• Back up /etc/slurm/ regularly.
• For more control, enable Slurm accounting (slurmdbd) and set up Quality of Service (QoS) rules.
• Monitor logs: tail -f /var/log/slurm/slurmctld.log

What’s Next?

In Part 6, we will continue with more Slurm setup.

You now have a working job scheduler! Users can submit jobs across your 12-node cluster with proper GPU awareness.