Kubernetes Storage Deep Dive: CSI Drivers, PV/PVC Lifecycle, StatefulSet Volume Semantics, and Node Failure Modes

1. CSI Drivers — The Storage Abstraction Layer:What Is CSI?

The Container Storage Interface (CSI) is a standardized API that decouples Kubernetes from storage vendor implementations. Before CSI, storage plugins were compiled directly into the Kubernetes codebase — meaning a bug in an EBS plugin could crash the kubelet. CSI solved this by pushing storage logic out-of-tree into separate, independently deployed drivers.

A CSI driver runs as a set of pods inside the cluster and exposes three core operations to Kubernetes:

• CreateVolume / DeleteVolume — provision and deprovision storage
• ControllerPublishVolume / ControllerUnpublishVolume — attach/detach volumes to nodes
• NodeStageVolume / NodePublishVolume — mount volumes into pod containers

CSI Driver Components

A typical CSI deployment consists of:

┌──────────────────────────────────────────────────────────┐
│                   CSI Controller Pod                      │
│  ┌─────────────────┐   ┌──────────────────────────────┐  │
│  │ external-        │   │ external-                    │  │
│  │ provisioner      │   │ attacher                     │  │
│  └─────────────────┘   └──────────────────────────────┘  │
│  ┌─────────────────────────────────────────────────────┐  │
│  │           CSI Driver (vendor plugin)                │  │
│  └─────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────┐
│                   CSI Node DaemonSet                      │
│  ┌──────────────────┐  ┌──────────────────────────────┐  │
│  │  node-driver-    │  │ CSI Driver (node plugin)     │  │
│  │  registrar       │  │ NodeStage / NodePublish       │  │
│  └──────────────────┘  └──────────────────────────────┘  │
└──────────────────────────────────────────────────────────┘

Sidecar Role external-provisioner Watches PVCs and calls CreateVolume on the driver external-attacher Calls ControllerPublish when a pod is scheduled to a node external-resizer Expands volumes by calling ControllerExpandVolume node-driver-registrar Registers the CSI driver socket with the kubelet

StorageClass and CSI

Every CSI driver is referenced through a StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: disk.csi.azure.com        # CSI driver name
parameters:
  skuName: Premium_LRS
  cachingMode: ReadOnly
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

**WaitForFirstConsumer** is critical — it delays volume creation until a pod is scheduled, ensuring the volume is created in the same availability zone as the node. Without this, you can get a pod stuck because its PVC was provisioned in zone-a but the pod scheduled to zone-b.

2. PV/PVC Lifecycle:The Four Phases of a Persistent Volume

Provisioning → Binding → Using → Releasing → Reclaiming

Phase 1: Provisioning

Volumes are provisioned in two ways:

Static provisioning — an administrator manually creates a PV:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: manual-pv
spec:
  capacity:
    storage: 50Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: disk.csi.azure.com
    volumeHandle: /subscriptions/.../disks/my-disk

Dynamic provisioning — Kubernetes calls the CSI driver automatically when a PVC is created that references a StorageClass.

Phase 2: Binding

The PVC controller matches a PVC to a PV based on:

• Storage capacity (PV must be >= PVC request)
• Access modes must be compatible
• StorageClass name must match
• Optional: selector label matching

Once matched, both the PV and PVC enter Bound state — this binding is exclusive and bidirectional. A ReadWriteOnce PV bound to one PVC cannot be claimed by another.

PVC (Pending) ──────────────────────► PVC (Bound)
                    Binding                │
PV  (Available) ─────────────────────► PV (Bound)

Phase 3: Using

A pod references the PVC by name:

volumes:
  - name: data
    persistentVolumeClaim:
      claimName: my-pvc

The kubelet then calls the CSI node plugin to stage and publish the volume into the pod’s mount namespace. The sequence is:

NodeStageVolume   → mounts device to a global staging path on the node
NodePublishVolume → bind-mounts the staging path into the pod's directory

Phase 4: Releasing and Reclaiming

When a PVC is deleted, the PV transitions to Released. What happens next depends on the ReclaimPolicy:

Policy Behavior Delete CSI DeleteVolume is called — the underlying disk is destroyed Retain PV stays in Released state; data is preserved but the PV must be manually cleaned and re-claimed Recycle Deprecated — ran rm -rf on the volume data

⚠️ Important: A PV in Released state cannot be automatically rebound to a new PVC even if data is intact. You must manually remove the .spec.claimRef field from the PV to make it Available again.

Access Modes

Mode Short Meaning ReadWriteOnce RWO One node can mount read-write ReadOnlyMany ROX Many nodes can mount read-only ReadWriteMany RWX Many nodes can mount read-write ReadWriteOncePod RWOP Only one pod cluster-wide (added in Kubernetes 1.22)

RWX requires a network filesystem like NFS, Azure Files, or CephFS. Block storage (Azure Disk, AWS EBS) only supports RWO.

3. StatefulSet Volume Semantics

Why StatefulSets Are Different

Deployments treat pods as interchangeable — any pod can be rescheduled anywhere and share a volume (or use ephemeral storage). StatefulSets treat each pod as a unique, stable identity with its own dedicated storage. This is essential for databases, message brokers (like Kafka), and any stateful workload where pod-0 must always reconnect to pod-0’s data.

VolumeClaimTemplates

StatefulSets define storage via volumeClaimTemplates rather than referencing a single PVC:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka-broker
spec:
  serviceName: kafka-headless
  replicas: 3
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: fast-ssd
        resources:
          requests:
            storage: 100Gi

Kubernetes automatically creates one PVC per pod replica using the naming convention:

<volumeClaimTemplate.name>-<statefulset.name>-<ordinal>

data-kafka-broker-0
data-kafka-broker-1
data-kafka-broker-2

Stable Identity Guarantees

StatefulSets provide three guarantees:

1. Stable network identity — pod names are deterministic (kafka-broker-0, not a random hash)
2. Ordered deployment — pods start in order (0, 1, 2) and terminate in reverse order (2, 1, 0) by default
3. Persistent storage — the PVC data-kafka-broker-0 always attaches to kafka-broker-0, regardless of which physical node it runs on

PVC Lifecycle Is Independent of the StatefulSet

This is a critical and often misunderstood point: deleting a StatefulSet does NOT delete its PVCs. The PVCs survive independently. This is intentional — it prevents accidental data loss. To fully clean up:

# Delete StatefulSet (pods are removed, PVCs remain)
kubectl delete statefulset kafka-broker

# Explicitly delete PVCs
kubectl delete pvc data-kafka-broker-0 data-kafka-broker-1 data-kafka-broker-2

Similarly, if a StatefulSet is scaled down from 3 to 2 replicas, the PVC data-kafka-broker-2 is not deleted — it stays around so that scaling back up to 3 can reattach the same data.

Headless Service Requirement

StatefulSets require a headless service (clusterIP: None) to give each pod a stable DNS entry:

kafka-broker-0.kafka-headless.namespace.svc.cluster.local
kafka-broker-1.kafka-headless.namespace.svc.cluster.local

This allows other pods and brokers to address each instance directly — essential for leader election, replication, and client routing.

4. Failure Modes When a Node Holding Local PVs Dies

This is the most operationally dangerous area of Kubernetes storage. The behavior differs significantly between network-attached storage and local storage.

Network-Attached Volumes (CSI block/file)

For volumes like Azure Disk, AWS EBS, or GCP PD:

Node dies
    │
    ├─► Node becomes NotReady (after node-monitor-grace-period, default 40s)
    │
    ├─► Pod marked for eviction (after pod-eviction-timeout, default 5min)
    │
    ├─► CSI external-attacher calls ControllerUnpublishVolume
    │       (detaches disk from dead node)
    │
    └─► Pod rescheduled to healthy node
            CSI attaches disk to new node → pod resumes

Total recovery time: typically 5–7 minutes with default settings. The disk is safely detached and reattached because it was never physically local to the node.

Local Volumes — A Different Beast

Local PVs are volumes tied to a specific node’s physical disk or directory:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-pv-node1
spec:
  capacity:
    storage: 500Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /mnt/disks/ssd1
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - node1               # ← permanently tied to this node

When node1 dies:

Node dies
    │
    ├─► Node becomes NotReady
    │
    ├─► Pod enters Terminating state
    │
    ├─► Kubernetes CANNOT reschedule the pod elsewhere
    │       because the PV has nodeAffinity pinning it to node1
    │
    └─► Pod stays Terminating INDEFINITELY
            (until node recovers OR manual intervention)

This is not a bug — it is intentional. Kubernetes cannot move a local PV because the data physically lives on that node’s disk.

Force-Deleting Stuck Pods

When a node is permanently dead and will not recover, you must manually intervene:

# Force delete the stuck pod (bypasses graceful termination)
kubectl delete pod kafka-broker-0 --grace-period=0 --force -n egindustrials-prod

# If the node object still exists, delete it too
kubectl delete node dead-node-name

After force-deleting, if the pod is part of a StatefulSet, it will be recreated — but it will again be stuck in Pending because the PV still has the dead node's affinity. You then have two choices:

Option A — Node recovers: Bring the node back online. Kubernetes automatically reattaches everything.

Option B — Node is permanently gone: Accept data loss, delete the local PV and PVC, and let the StatefulSet provision fresh storage:

kubectl delete pvc data-kafka-broker-0 -n egindustrials-prod
kubectl delete pv local-pv-node1
# StatefulSet will recreate PVC → new empty volume provisioned

The volumeBindingMode: WaitForFirstConsumer Trap with Local Volumes

If you use WaitForFirstConsumer with local storage, the PVC stays Pending until a pod is scheduled — but the pod can't be scheduled until the PVC is bound. On a dead node this creates a circular dependency that never resolves without manual intervention.

Comparison: Network vs Local Storage on Node Failure

Aspect Network-Attached (CSI) Local PV Pod reschedulable after node death? Yes /No Data survives node loss?Yes (disk persists)No (disk is on dead node) Recovery time 5–7 min (automatic) Manual intervention required Use case General workloads, databases High-performance I/O, NVMe-class Recommended mitigation None needed Application-level replication (e.g., Kafka replication factor ≥ 3)

Best Practices for Local PV Workloads

1. Always use replication at the application layer — Kafka with replication.factor=3 and min.insync.replicas=2 means losing one broker (and its local disk) is survivable without data loss.
2. Use pod disruption budgets (PDBs) to prevent voluntary disruptions from taking down too many replicas simultaneously:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: kafka-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: kafka

1. Set podManagementPolicy: Parallel only if your application handles concurrent startup safely — otherwise keep OrderedReady for safer rolling restarts.
2. Monitor node health proactively — use node problem detector and alert on NodeNotReady before it cascades into stuck pods.
3. Prefer network-attached storage for critical stateful workloads unless you have a specific, measured need for local disk I/O performance that cannot be met otherwise.

Summary

Concept Key Takeaway CSI Drivers Out-of-tree plugins that abstract storage via standardized Create/Attach/Mount APIs PV/PVC Lifecycle Provision → Bind → Use → Release → Reclaim; ReclaimPolicy controls what happens to data StatefulSet Volumes Each pod gets its own PVC via volumeClaimTemplates; PVCs survive StatefulSet deletion Node failure (network storage) Automatic recovery in ~5 min via CSI detach/reattach Node failure (local PV) Pod stuck indefinitely; requires manual force-delete and application-level replication to survive