Building a High-Performance Computing (HPC) Cluster from Scratch: A Step-by-Step Guide — Part 5: LMOD Environment Modules System

In Part 4, we configured NFS shared storage so that all nodes can access common directories.

In this part, we set up LMOD — the modern environment modules system. LMOD is one of the most critical tools for usability in any professional HPC cluster.

Why Do We Need LMOD?

Modern HPC clusters host dozens (or hundreds) of software packages, libraries, compilers, and toolchains that often have multiple versions of the same software (e.g., CUDA 12.4, 12.6; Python 3.10, 3.11, 3.12; TensorFlow, PyTorch, etc.).

Without a modules system:

• Users would manually modify PATH, LD_LIBRARY_PATH, PYTHONPATH, etc., leading to conflicts and “it works on my machine” problems.
• Reproducibility across nodes and over time becomes nearly impossible.

LMOD (Lua-based Modular Environment) solves this by allowing users to dynamically load and unload software environments with simple commands like:

• module load cuda/12.6
• module load python/3.11
• module avail
• module list

LMOD is developed by TACC (Texas Advanced Computing Center) and has become the de-facto standard in most research HPC centers due to its speed, hierarchical module support, and excellent dependency handling.

Part 6.1: LMOD Installation on the HeadNode

1. Install Build Dependencies

sudo dnf install -y lua lua-devel lua-posix lua-filesystem tcl-devel

• These packages provide the Lua runtime and development libraries that LMOD depends on.

2. Download and Build LMOD

We install LMOD into /opt/apps following best practices (this allows easy future upgrades):

cd /tmp
wget https://github.com/TACC/Lmod/archive/refs/tags/8.7.tar.gz
tar -xvf 8.7.tar.gz
cd Lmod-8.7

./configure --prefix=/opt/apps
make -j$(nproc)
sudo make install

• — prefix=/opt/apps: Installs LMOD into a versioned directory structure.
• make -j$(nproc): Uses all available CPU cores to speed up compilation.

3. Create Symbolic Link for Easy Upgrades

sudo ln -s /opt/apps/lmod/lmod /opt/apps/lmod/lmod-current

4. Initialize LMOD System-Wide

Create symbolic links so that every user automatically gets the module command:

sudo ln -s /opt/apps/lmod/lmod/init/profile /etc/profile.d/z00_lmod.sh
sudo ln -s /opt/apps/lmod/lmod/init/cshrc /etc/profile.d/z00_lmod.csh

5. Create Module Directory Structure

sudo mkdir -p /opt/modulefiles/Core
sudo mkdir -p /etc/lmod

6. Configure Module Search Path

echo "/opt/modulefiles/Core" | sudo tee /etc/lmod/.modulespath

7. Test on HeadNode

source /etc/profile.d/z00_lmod.sh
module avail
module --version

Part 6.2: LMOD Setup on All Compute Nodes

Since /opt/apps and /opt/modulefiles will be shared via NFS (from Part 4), we only need to configure the shell initialization on each compute node.

1. Install Runtime Dependencies

sudo dnf install -y lua-filesystem lua-posix

2. Link Initialization Scripts

sudo ln -s /opt/apps/lmod/lmod/init/profile /etc/profile.d/z00_lmod.sh

3. Refresh Environment and Test

source /etc/profile.d/z00_lmod.sh
export MODULEPATH=/opt/modulefiles/Core
module avail

Repeat this process (or use Ansible) on all 12 compute nodes.

Part 6.3: Creating Your First Modulefiles

Modulefiles are written in Lua and tell the shell how to modify environment variables when a module is loaded.

Example: Create a CUDA Module

1. Create directory structure:

sudo mkdir -p /opt/modulefiles/Core/cuda

1. Create the module file:

sudo nano /opt/modulefiles/Core/cuda/12.6.lua

help([[ CUDA Toolkit 12.6]])
whatis("Version: 12.6")
whatis("Description: NVIDIA CUDA Toolkit for GPU computing")
local root = "/usr/local/cuda-12.6"
prepend_path("PATH", root .. "/bin")
prepend_path("LD_LIBRARY_PATH", root .. "/lib64")
prepend_path("LIBRARY_PATH", root .. "/lib64")
prepend_path("INCLUDE", root .. "/include")
prepend_path("MANPATH", root .. "/doc/man")
setenv("CUDA_HOME", root)
setenv("CUDA_VERSION", "12.6")

Key Lua functions explained:

• prepend_path(): Adds directories to environment variables without overwriting existing ones.
• setenv(): Sets specific environment variables.
• help() and whatis(): Provide information when users run module help or module whatis.

Make the Module Available

module avail
module load cuda/12.6
nvcc --version
module unload cuda/12.6

Part 6.4: Advanced LMOD Features & Best Practices

• Hierarchical Modules: You can create module categories (e.g., Core, Compiler, MPI, Applications).
• Spider Cache (for performance): When you have many modules, enable spider cache:

sudo mkdir -p /opt/modulefiles/cache module spider

• Default Versions: Create a .version file inside a module directory to set a default version.
• Module Collections: Users can save their favorite combinations with module save myenv and restore with module restore myenv.
• Naming Convention: Use package/version (e.g., gcc/13.2, openmpi/4.1.6).

Recommended Practices:

• Keep all software installations in /opt/apps/<package>/<version>.
• Document every module with meaningful help() text.
• Use versioned modulefiles (never overwrite old versions).
• Regularly run module spider to update the cache.
• Back up the entire /opt/modulefiles directory.

Verification Across the Cluster

From any node (including compute nodes):

module avail
module load cuda/12.6
echo $PATH | grep cuda
module list
module unload cuda

Part 6.5: How to use LMOD?

1. Core Module Commands Reference

Every interaction with the software stack uses the module command (or the shorthand ml).

Finding Software

• module avail (or ml av): Lists all software packages currently visible and available. Tip: Filter with a keyword: module avail cuda
• module spider: The most powerful search tool. It searches all modules, even those hidden behind dependencies. Example: module spider python

Managing Your Environment

• module load <package> (or ml <package>): Loads the software into your current shell session. Example: module load git/2.47.3
• module list (or ml): Shows currently loaded modules in your session.
• module unload <package>: Removes a module from your environment.
• module purge: Unloads all modules — useful for starting clean.
• module swap: old/new Replaces one version with another seamlessly.

Handy Shorthand (ml):

• ml → module list
• ml git → module load git
• ml -git → module unload git
• ml av → module avail

2. Understanding Module Output

When you run module avail, you will see output similar to this:

---------------------------- /opt/modulefiles/Core -----------------------------

   gcc/24.1         python/3.7(L)       testutils/1.0

--------------------- /opt/apps/lmod/modulefiles/Core ---------------------
   lmod    settarg

  Where:
   L:  Module is loaded
   D:  Default module

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching
any of the "keys".

• (L) = Currently loaded in your session
• (D) = Default version (loaded if you don’t specify a version)
• /opt/modulefiles/Core = Primary location for your software stack

Useful Commands:

module --default avail     # Show only default modules
module overview            # Summary of available modules
module keyword cuda        # Search by keyword

Part 6.6: Best Practices for Using Modules

Rule 1: Always Load Modules in Job Scripts

Instead of putting absolute paths like /opt/apps/python/3.7/bin/python into your job submission or execution scripts, always load the module at the beginning of your workflow:

# Clean environment
module purge

# Load exact versions needed
module load gcc/24.1
module load pyhton/3.7

# Your commands
python train_model.py

This ensures perfect reproducibility even months later.

Rule 2: Use module purge Before Loading New Toolchains

This prevents version conflicts between Python, CUDA, compilers, etc.

Rule 3: Create Module Collections (Advanced)

module save my_ml_env          # Save current loaded modules
module restore my_ml_env       # Restore later

Part 6.6: Troubleshooting Common Issues

1. “Command not found” after loading a module → Run hash -r to refresh Bash’s command cache.
2. Module doesn’t appear in module avail → Use module spider <name> — it may require a prerequisite (e.g., load a compiler first).
3. Library or version conflicts → Run module purge and reload only the minimum required modules.
4. Module list is too long → Use module spider or module keyword for targeted searches.

What’s Next?

In Part 6, we will install and configure Slurm, the powerful workload manager that will let users submit jobs across all 12 compute nodes with proper CPU and GPU resource allocation.

Your cluster now has a professional, user-friendly software environment thanks to LMOD!