Setting Up NetApp Trident on Talos Linux: A Complete Guide

Introduction

Integrating enterprise storage solutions with Kubernetes can be challenging, especially when running on immutable operating systems like Talos Linux. In this guide, I’ll walk you through the complete process of setting up NetApp Trident CSI driver on Talos Linux, including the critical configurations and common pitfalls I encountered during implementation.

NetApp Trident is a fully supported open-source dynamic storage orchestrator for Kubernetes that enables persistent storage provisioning from NetApp ONTAP systems. Combined with Talos Linux’s security-focused, immutable design, this setup provides a robust foundation for production Kubernetes clusters.

Prerequisites

Before starting, ensure you have:

• A running Talos Linux Kubernetes cluster

• NetApp ONTAP storage system (A220 or similar)

• Administrative access to both systems

• talosctl and kubectl CLI tools installed and configured

Part 1: Configuring Talos Linux

Talos Linux requires specific kernel modules and system extensions to support iSCSI and multipath storage. Unlike traditional Linux distributions, Talos uses an immutable approach where all configurations must be declared in the machine configuration.

Required System Extensions

Add the following system extensions to your Talos configuration:

system_extensions = [

“siderolabs/iscsi-tools”,

“siderolabs/trident-iscsi-tools”,

“siderolabs/multipath-tools”,

“siderolabs/util-linux-tools”,

… # The other extensions like hypervisor-agent

]

Key extensions explained:

• trident-iscsi-tools: Provides the tools and binaries used by the NetApp Trident CSI for iSCSI. It contains lsscsi, ls, free, pgrep, cat and dd.
• iscsi-tools: Provides iscsi-tools
• multipath-tools: Enables device mapper multipathing for redundant storage paths
• util-linux-tools: Provides blkid, mkfs, and other utilities needed for volume formatting

Kernel Modules Configuration

Load the necessary device mapper kernel modules:

kernel_modules = [

{ name = “dm_multipath” },

{ name = “dm_round_robin” }

]

These ker modules enable:

• dm_multipath: Core multipath support
• dm_round_robin: Round-robin path selection algorithm for load balancing

Multipath Configuration

Configure the multipath daemon with appropriate settings:

extension_service_configs = [

{ name = “multipathd”

config_files = [{ content = <<-EOT

defaults {

user_friendly_names yes

find_multipaths no

path_selector “round-robin 0”

}

EOT

mount_path = “/etc/multipath.conf”

}]}]

Configuration breakdown:

• user_friendly_names yes: Creates human-readable device names

• find_multipaths no: Disables automatic multipath detection (recommended for Trident)

• path_selector “round-robin 0”: Uses round-robin I/O path selection

Applying Talos Configuration

After updating your machine configuration, apply it to your cluster:

Apply configuration to control plane

talosctl -n <control-plane-ip> apply-config — file controlplane.yaml

Apply configuration to worker nodes

talosctl -n <worker-ip> apply-config — file worker.yaml

Reboot nodes to load new kernel modules and extensions

talosctl -n <node-ip> reboot

Important: A reboot is required for kernel modules and system extensions to take effect.

Part 2: Configuring NetApp ONTAP

Now let’s configure the NetApp storage system to support both iSCSI (block) and NFS (file) protocols.

Step 1: Create Storage Virtual Machine (SVM)

An SVM (Storage Virtual Machine) provides isolated storage services with its own network interfaces and volumes.

1. Log into NetApp System Manager

2. Navigate to Storage → Storage VMs

3. Click Add to create a new SVM

4. Configure the SVM:

Name: svm_timewarp_k8s (or your preferred name)

Protocols: Enable both iSCSI and NFS

Data LIFs: We’ll configure these in the next steps

[SCREENSHOT PLACEHOLDER: NetApp System Manager — Create SVM dialog]

Step 2: Configure Network Interfaces (LIFs)

Create dedicated Logical Interfaces (LIFs) for storage traffic:

Management LIF

Purpose: Management and API access

• IP Address: 10.20.1.73 (example)

• Protocol: HTTPS

iSCSI Data LIF

• Purpose: Block storage traffic

• IP Address: 10.20.1.75 (example)

• Protocol: iSCSI

• Port: Dedicated network port for storage traffic

NFS Data LIF

• Purpose: File storage traffic

• IP Address: 10.20.1.76 (example)

• Protocol: NFS

• Port: Can share with iSCSI or use dedicated port

[SCREENSHOT PLACEHOLDER: NetApp System Manager — Network Interfaces configuration]

Step 3: Configure iSCSI Settings

1. Navigate to Protocols → iSCSI

2. Enable iSCSI on the SVM

3. Create an iGroup (initiator group):

• Name: trident-iscsi

• OS Type: Linux

• Protocol: iSCSI

• Initiators: Add your Kubernetes node IQNs (found via talosctl -n <node-ip> read /etc/iscsi/initiatorname.iscsi)

[SCREENSHOT PLACEHOLDER: NetApp System Manager — iSCSI iGroup configuration]

Step 4: Configure NFS Export Policy

1. Navigate to Protocols → NFS

2. Ensure NFS v3 and v4.1 are enabled

3. Create or modify the default export policy:

• Client Match: Your Kubernetes node subnet (e.g., 10.21.9.0/24)

• Access Protocol: NFS3, NFS4

• Read-Only: No

• Read-Write: Yes

• Superuser: sys

• Security: sys (or krb5 if using Kerberos)

[SCREENSHOT PLACEHOLDER: NetApp System Manager — NFS Export Policy configuration]

Step 5: Create Credentials Secret

Create a Kubernetes secret with your NetApp credentials:

kubectl create namespace trident

kubectl create secret generic netapp-ontap-credentials \

— from-literal=username=admin \

— from-literal=password=’YourSecurePassword’ \

-n trident

Part 3: Installing and Configuring Trident

Step 1: Install Trident Operator

Install Trident using Helm:

Add NetApp Helm repository

helm repo add netapp-trident https://netapp.github.io/trident-helm-chart

helm repo update netapp-trident

Install Trident operator

helm install trident-operator netapp-trident/trident-operator \

— namespace trident \

— create-namespace \

— version 100.2510.0

Wait for the operator to be ready:

kubectl get pods -n trident

Expected output:

NAME READY STATUS RESTARTS AGE

trident-controller-xxxxx 6/6 Running 0 2m

trident-node-linux-xxxxx 2/2 Running 0 2m

trident-operator-xxxxx 1/1 Running 0 3m

Step 2: Configure Backend Storage

Create backend configurations for both iSCSI and NFS protocols.

iSCSI Backend Configuration

apiVersion: trident.netapp.io/v1

kind: TridentBackendConfig

metadata:

name: ontap-iscsi-backend

namespace: trident

spec:

version: 1

storageDriverName: ontap-san

managementLIF: “10.20.1.73”

dataLIF: “10.20.1.75”

svm: “svm_timewarp_k8s”

igroupName: “trident-iscsi”

credentials:

name: netapp-ontap-credentials

storage:

• labels:

encryption: “false”

protocol: “iscsi”

zone: “dev”

defaults:

spaceReserve: “none”

snapshotPolicy: “default”

snapshotReserve: “10”

splitOnClone: “false”

encryption: “false”

tieringPolicy: “none”

NFS Backend Configuration

apiVersion: trident.netapp.io/v1

kind: TridentBackendConfig

metadata:

name: ontap-nfs-backend

namespace: trident

spec:

version: 1

storageDriverName: ontap-nas

managementLIF: “10.20.1.73”

dataLIF: “10.20.1.76”

svm: “svm_timewarp_k8s”

credentials:

name: netapp-ontap-credentials

storage:

• labels:

performance: “standard”

protocol: “file”

encryption: “false”

zone: “dev”

defaults:

spaceReserve: “none”

snapshotPolicy: “default”

snapshotReserve: “10”

splitOnClone: “false”

encryption: “false”

tieringPolicy: “none”

exportPolicy: “default”

securityStyle: “unix”

unixPermissions: “0755”

Apply the backend configurations:

kubectl apply -f trident-backend-iscsi.yaml

kubectl apply -f trident-backend-nfs.yaml

Verify backends are online:

kubectl get tridentbackendconfig -n trident

Step 3: Create Storage Classes

Define StorageClasses for different use cases.

iSCSI StorageClass (RWO — Block Storage)

apiVersion: storage.k8s.io/v1

kind: StorageClass

metadata:

name: ontap-iscsi

annotations:

storageclass.kubernetes.io/is-default-class: “true”

provisioner: csi.trident.netapp.io

parameters:

backendType: “ontap-san”

encryption: “false”

provisioningType: “thin”

fsType: “ext4”

allowVolumeExpansion: true

volumeBindingMode: WaitForFirstConsumer

NFS StorageClass (RWX — File Storage)

apiVersion: storage.k8s.io/v1

kind: StorageClass

metadata:

name: ontap-nfs

provisioner: csi.trident.netapp.io

parameters:

backendType: “ontap-nas”

encryption: “false”

fsType: “nfs”

allowVolumeExpansion: true

volumeBindingMode: WaitForFirstConsumer

Key configuration notes:

• volumeBindingMode: WaitForFirstConsumer: Delays volume binding until a pod is scheduled, preventing multi-attach errors

• fsType: “ext4”: Required for fsGroup to work with iSCSI volumes

• allowVolumeExpansion: true: Enables dynamic volume expansion

Apply the storage classes:

kubectl apply -f storage-classes.yaml

Step 4: Test Storage Provisioning

Create a test PVC to verify everything works:

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

name: test-pvc

spec:

accessModes:

• ReadWriteOnce

storageClassName: ontap-iscsi

resources:

requests:

storage: 1Gi

Apply them:

kubectl apply -f test-pvc.yaml

kubectl get pvc test-pvc

The PVC should show Pending status until a pod uses it (due to WaitForFirstConsumer).

Create a test pod:

apiVersion: v1

kind: Pod

metadata:

name: test-pod

spec:

containers:

• name: test

image: nginx:alpine

volumeMounts:

• name: data

mountPath: /data

volumes:

• name: data

persistentVolumeClaim:

claimName: test-pvc

Apply them:

kubectl apply -f test-pod.yaml

kubectl get pod test-pod

kubectl get pvc test-pvc

The PVC should now be Bound and the pod Running.

Part 4: Common Errors and Troubleshooting

Error 1: “device /dev/dm-X is not unformatted”

Symptom:

MountVolume.MountDevice failed: rpc error: code = Internal

desc = failed to stage volume: device /dev/dm-1 is not unformatted

Root Cause: The util-linux-tools extension is missing from Talos, so the Trident CSI driver cannot detect or format filesystems. The driver needs blkid to check if a device is formatted and mkfs utilities to create filesystems.

Solution:

1. Add siderolabs/util-linux-tools to your Talos system extensions

2. Apply the configuration and reboot nodes

3. Delete and recreate affected PVCs

Verify util-linux-tools is installed

talosctl -n <node-ip> list /usr/local/lib/extensions/util-linux-tools/

Check if blkid is available in Trident node pod

kubectl exec -n trident trident-node-linux-xxxxx -c trident-main — which blkid

Error 2: Multi-Attach Volume Errors

Symptom:

Multi-Attach error for volume “pvc-xxxxx”: Volume is already

exclusively attached to one node and can’t be attached to another

Root Cause: Using volumeBindingMode: Immediate causes the volume to be provisioned and attached before pod scheduling. If the pod is scheduled on a different node, Kubernetes cannot move the RWO volume.

Solution:

Change StorageClass to use WaitForFirstConsumer:

volumeBindingMode: WaitForFirstConsumer

This ensures volumes are provisioned on the same node where the pod is scheduled.

Error 3: Multipath Device Mapper Errors

Symptom:

device-mapper: table: 251:0: multipath: unknown path selector type

dm_addmap: libdm task=0 error: Invalid argument

Root Cause: The multipath configuration specifies a path selector (e.g., service-time, queue-length) whose kernel module is not loaded or available.

Solution:

1. Check available multipath modules:

talosctl -n <node-ip> read /lib/modules/$(uname -r)/kernel/drivers/md/

2. Use dm_round_robin which is widely available:

kernel_modules = [

{ name = “dm_multipath” },

{ name = “dm_round_robin” }

]

3. Update multipath.conf:

path_selector “round-robin 0”

Error 4: iSCSI Login Failures

Symptom:

iscsiadm: Could not login to [iface: default, target: iqn.xxx, portal: x.x.x.x,3260]

Root Cause: Network connectivity issues, incorrect iGroup configuration, or missing iSCSI initiator setup.

Solution:

1. Verify iSCSI initiator name on nodes:

talosctl -n <node-ip> read /etc/iscsi/initiatorname.iscsi

2. Ensure node IQNs are added to the NetApp iGroup

3. Test network connectivity to iSCSI LIF:

talosctl -n <node-ip> netstat | grep 3260