Building a High-Performance Computing (HPC) Cluster from Scratch: A Step-by-Step Guide — Part 4: NFS Shared Storage Configuration

In Part 3, we configured MUNGE for secure inter-node authentication and LDAP + SSSD for centralized user management. In this part, we configure NFS (Network File System) — the shared storage layer that allows users to access the same files (home directories, projects, and data) from any node in the cluster.

This is a critical step because it enables users to seamlessly access their files from any node in the cluster (HeadNode or any of the 12 compute nodes).

Why NFS in This Phase?

NFSv4 is simple, reliable, and easy to deploy, making it an excellent Phase 1 shared filesystem for most research clusters. Later, as your data grows beyond 50–100 TB, you can migrate to a parallel filesystem such as BeeGFS, Lustre, or Weka.

We will:

• Export shares from the HeadNode
• Mount them on all compute nodes using both static mounts (/etc/fstab) and autofs for better reliability

Planned Shared Directories

We will create and share the following common directories:

• /scratch → Shared scratch space (temporary files, can be cleaned periodically)
• /projects → Research project directories
• /home → User home directories (via NFS)
• /data → Large input/output datasets (read-heavy)
• /software → Optional shared software area

Part 4.1: NFS Server Setup on the HeadNode

1. Install NFS Utilities

sudo dnf install nfs-utils -y

• dnf install: Uses Red Hat’s package manager to download and install software.
• nfs-utils: Contains all the tools and daemons required to run an NFS server.

2. Create the Shared Directories

sudo mkdir -p /scratch /projects /data /home
sudo chmod 755 /scratch /projects /data
sudo chmod 755 /home

• mkdir -p: Creates directories and any necessary parent directories.
• chmod 755: Sets permissions so the owner has full access and others can read/execute (standard for shared directories).

3. Configure NFS Exports

Edit the exports file:

sudo nano /etc/exports

Add the following lines (adjust subnets to match your cluster network, e.g., 10.28.0.0/16):

/scratch       10.28.0.0/16(rw,sync,no_root_squash,no_subtree_check)
/projects      10.28.0.0/16(rw,sync,no_root_squash,no_subtree_check)
/data          10.28.0.0/16(ro,sync,root_squash,no_subtree_check)
/home          10.28.0.0/16(rw,sync,no_root_squash,no_subtree_check)

Explanation of options:

• rw = Read and write access
• ro = Read-only (for sensitive data)
• sync = Write data to disk before replying (safer, slightly slower)
• no_root_squash = Allows root on compute nodes to act as root on the share (required for Slurm)
• no_subtree_check = Improves performance

4. Apply Exports and Start Services

sudo exportfs -rav

• exportfs: Command to manage NFS exports.
• -r: Re-export all directories (apply changes).
• -a: Export all directories listed in /etc/exports.
• -v: Verbose output (shows what was exported).

sudo systemctl enable --now nfs-server rpcbind

• systemctl enable — now: Enables the service to start automatically at boot and starts it immediately.
• nfs-server: Main NFS daemon.
• rpcbind: Required supporting service for RPC (Remote Procedure Call) communication.

5. Configure Firewall on HeadNode

sudo firewall-cmd --permanent --add-service=nfs
sudo firewall-cmd --permanent --add-service=rpc-bind
sudo firewall-cmd --permanent --add-service=mountd
sudo firewall-cmd --reload

• --permanent: Makes the rule survive reboots.
• --add-service: Opens the necessary firewall ports for NFS.
• --reload: Applies the new firewall rules immediately.

Part 4.2: NFS Client Setup on All Compute Nodes

1. Install NFS Utilities

sudo dnf install nfs-utils -y

2. Create Mount Points

sudo mkdir -p /scratch /projects /data /home

3. Mount NFS Shares Using /etc/fstab (Static Mount)

Edit the fstab file:

sudo nano /etc/fstab

Add these lines at the end (replace headnode with your actual HeadNode hostname or IP):

headnode:/scratch    /scratch    nfs    defaults,hard,rsize=32768,wsize=32768  0 0
headnode:/projects   /projects   nfs    defaults,hard,rsize=32768,wsize=32768  0 0
headnode:/data       /data       nfs    defaults,hard,ro,rsize=32768,wsize=32768 0 0
headnode:/home       /home       nfs    defaults,hard,rsize=32768,wsize=32768  0 0

Important options explained:

• defaults — Standard mount options
• hard — If the server becomes unavailable, the client will keep retrying (recommended for HPC)
• rsize / wsize — Read/write block size in bytes (32768 = 32KB improves performance)
• timeo=600 — Timeout value in tenths of a second

Mount all shares:

sudo mount -a

• mount -a: Mounts all filesystems listed in /etc/fstab that are not yet mounted.

Verify:

df -h | grep headnode

Part 4.3: Recommended — AutoFS for More Reliable Mounting

Autofs automatically mounts shares when accessed and unmounts them after inactivity which is better for stability in large clusters.

1. Install and Enable AutoFS

sudo dnf install autofs -y
sudo systemctl enable --now autofs

2. Create AutoFS Map File

sudo nano /etc/auto.share

Add the following:

/scratch     -rw,soft,intr,no_root_squash,rsize=32768,wsize=32768    headnode:/scratch
/projects    -rw,soft,intr,no_root_squash,rsize=32768,wsize=32768    headnode:/projects
/data        -ro,soft,intr,root_squash,rsize=32768,wsize=32768       headnode:/data
/home        -rw,soft,intr,no_root_squash,rsize=32768,wsize=32768    headnode:/home

3. Configure AutoFS Master Map

sudo nano /etc/auto.master

Add at the bottom:

/-    /etc/auto.share    --timeout=300

• /- indicates a “direct map,” which allows you to mount directly to an absolute path.
• — timeout=300 tells autofs to unmount the folder if it hasn’t been used for 300 seconds (prevents stale mount hangs).

3. Restart and Test AutoFS

sudo systemctl restart autofs

Test by accessing the directories:

ls /scratch
ls /projects

Verification Commands

From any compute node:

showmount -e headnode          # Shows what the server is exporting
df -h | grep headnode          # Shows mounted NFS shares
mount | grep nfs               # Detailed mount information

Best Practices and Troubleshooting

• Use showmount -e headnode from a compute node to see available exports.
• Monitor NFS performance with nfsstat -c (client) and nfsstat -s (server).
• For better performance with large files, increase rsize and wsize (32768 or 65536).
• If you see “Permission Denied” errors → check UID/GID consistency via LDAP and no_root_squash.
• Stale mounts? Use sudo umount -f /mountpoint or restart autofs.
• Always test with a normal user account, not just root.

What’s Next?

In Part 5, we will install and configure LMOD, the environment modules system that allows users to easily load different versions of software (CUDA, Python, TensorFlow, etc.) without conflicts.

You now have centralized users (LDAP) and shared storage (NFS), now the cluster is starting to feel unified!