From Pending to Running: Mastering Node Affinity in Kubernetes

Imagine this, You’re working on a Kubernetes project — whether it’s running on a local KIND cluster, Minikube or on a Bare-Metal. You define a PersistentVolume and attach a PersistentVolumeClaim. You configure a StatefulSet or Deployment. Everything seems perfect… until you deploy, and your pod just sits there.

No errors. No logs. Just stuck in “Pending.”

This kind of issue is surprisingly common, and it almost always boils down to Node Affinity — or the lack of it. And what’s worse? You’re not alone. Plenty of developers run into the same wall. In fact, there are several variations of this problem — PVC stays pending, PV is unused, or the pod fails to mount the volume. All of these point to the same root issue: your pod can’t land where the data physically exists.

Let us together learn how to solve this problem in this Article.

Table of Contents

⚡Understanding Node Affinity in Multi-Node Clusters

⚡The Scheduling Hierarchy: NodeSelector → Node Affinity → Pod Affinity

⚡What Is Node Affinity?

⚡Example: Using Node Affinity in a StatefulSet

⚡Node Affinity Rules: Required vs Preferred

⚡Node Affinity Operators Explained

⚡Troubleshooting Common Errors

⚡Best Practices

⚡Conclusion

🧩Understanding Node Affinity in Multi-Node Clusters

Let’s assume you have a cluster setup like this:

🔸1 Control Plane Node

🔸3 Worker Nodes

You create a PersistentVolume (PV) using hostPath: /data/mongodb. Now that folder only exists on one of those worker nodes.

But your pod can be scheduled on any of the three worker nodes. If it ends up on a different node, your PVC can’t bind to the PV — because the storage doesn’t exist there.

That’s why you see this dreaded message:

Warning  FailedScheduling  default-scheduler  0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims

And boom — your pod stays stuck in “Pending.”

📊The Scheduling Hierarchy

Before diving into Node Affinity, it’s helpful to understand Kubernetes scheduling logic:

🔷nodeSelector:

🔸The simplest way to schedule a pod on specific nodes.

🔸You match a pod to a node using key-value labels.

🔸It’s very limited: only exact matches, no complex logic.

🔷nodeAffinity:

🔸A more flexible and powerful alternative to nodeSelector.

🔸Supports complex matching rules using: In, NotIn, Exists, etc.

🔸Comes in two types: required and preferred (explained later)

🔷podAffinity:

🔸Used to schedule a pod on the same node or zone as another set of pods.

🔸Useful when pods need to communicate closely and quickly.

🔷podAntiAffinity:

🔸The opposite of pod affinity.

🔸Ensures that pods are not scheduled on the same node (or zone)

🔸Commonly used for high availability.

Node Affinity is usually the go-to when scheduling pods that require localized or node-specific resources.

🔍What Is Node Affinity?

Node Affinity is a Kubernetes feature that allows you to constrain a pod to only run on certain nodes based on node labels.

In simpler terms:

“Only run this pod on <your-node-name> — because that’s where my storage physically exists.”

Node Affinity is essential when using static storage like hostPath, local-path, or any custom setup in bare-metal, hybrid, or development environments.

📝Example: Using Node Affinity in a StatefulSet

Here’s a full StatefulSet manifest file that demonstrates how to use node affinity in a real scenario:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongodb
  namespace: easyshop
spec:
  serviceName: "mongodb-service"
  replicas: 1
  selector:
    matchLabels:
      app: mongodb
  template:
    metadata:
      labels:
        app: mongodb
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/hostname
                operator: In
                values:
                - <your-node-name>  # Replace with actual node name
      containers:
      - name: mongodb
        image: mongo:6.0
        ports:
        - containerPort: 27017
        volumeMounts:
        - name: mongodb-data
          mountPath: /data/db
  volumeClaimTemplates:
  - metadata:
      name: mongodb-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "standard"
      resources:
        requests:
          storage: 5Gi

🤔 Wait — How Do You Even Know Which Node Has That Storage Path?

You might be wondering:

“Okay, cool, I get that I need to target a specific node. But how on earth do I figure out which node actually holds the storage path I’m trying to use?”

That’s a crucial question, because if you pick the wrong node in your node affinity, your pods will stay stuck in Pending forever.

Here’s how you can figure it out:

Identify and Label the Correct Node

• 🖥️ First, list your cluster nodes:

kubectl get nodes -o wide

• 🔍 Then SSH into each node and check whether the path (e.g., /data/mongodb) exists:

ls /data/mongodb

• ✅ Once you’ve found the right node, label it with the key your affinity rule expects:

kubectl label nodes <your-node-name> kubernetes.io/hostname=<your-node-name>

Make sure the label you assign matches the one referenced in the matchExpressions section of your manifest.

⚖️Node Affinity Rules: Required vs Preferred

Here’s a full Deployment manifest using both types:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
  namespace: easyshop
spec:
  replicas: 2
  selector:
    matchLabels:
      app: webapp
  template:
    metadata:
      labels:
        app: webapp
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: disktype
                operator: In
                values:
                - ssd
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: zone
                operator: In
                values:
                - us-east-1a
      containers:
      - name: nginx
        image: nginx

Explanation:

🔸required: Pod will only schedule on nodes with disktype=ssd

🔸preferred: Tries to schedule in zone=us-east-1a but falls back if unavailable

requiredDuringSchedulingIgnoredDuringExecution

🔸This is a hard requirement.

🔸If the rule is not satisfied, the pod won’t be scheduled at all.

🔸Use this when the pod must run on specific types of nodes or in specific environments

preferredDuringSchedulingIgnoredDuringExecution

🔸This is a soft preference.

🔸The scheduler will try to satisfy the condition, but if it can’t, it will still schedule the pod somewhere else.

🔸Use this when you want to optimize for something (like zone locality, faster disks, etc.) but can still run elsewhere.

🔧Node Affinity Operators Explained

Here’s a deployment using all relevant node affinity operators:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: analytics-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: analytics
  template:
    metadata:
      labels:
        app: analytics
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: cpu
                operator: Gt
                values:
                - "4"
              - key: region
                operator: In
                values:
                - us-east
              - key: gpu
                operator: Exists
              - key: deprecated
                operator: DoesNotExist
      containers:
      - name: analytics
        image: my-analytics:latest

What’s happening here:

🔸Ensures node has more than 4 CPUs (Gt)

🔸Must be in region=us-east (In)

🔸Must have GPU label present (Exists)

🔸Must not be marked as deprecated (DoesNotExist)

Node Affinity Operators:

🔸**In* – Matches if the node label key has any value* from the specified list.

🔸**NotIn* – Matches if the node label key's value is not in* the given list.

🔸**Exists** – Matches if the node has the label key, regardless of value.

🔸**DoesNotExist* – Matches if the node does not have* the specified label key.

🔸**Gt* – Matches if the label’s value is greater than* the specified integer.

🔸**Lt* – Matches if the label’s value is less than* the specified integer.

⚠️ Troubleshooting: Still Not Working?

❓My pod is stuck in Pending. How do I find out what’s wrong? ✔️Run kubectl describe pod <pod-name> and look for events or warning messages such as “unbound PersistentVolumeClaims” or notes about node mismatches. These messages often point to affinity or volume binding issues.

❓My PersistentVolumeClaim (PVC) is stuck in Pending. What should I check? ✔️Verify that the storageClassName in your PVC matches exactly with the storageClassName defined in your PersistentVolume (PV). A mismatch here will prevent PVC from binding.

❓The PV is Bound but the pod is still Pending. What could be wrong? ✔️This usually means a node mismatch. Your PV is bound to a specific node (due to hostPath or local-path), but your pod’s node affinity or default scheduling is placing it on a different node. Make sure the pod’s affinity matches the node where the PV lives.

✅ Best Practices

1. Label with Intention: Use labels that describe node capabilities (e.g., gpu=true, zone=us-east-1a, env=prod).
2. Use required with Caution: Avoid hard requirements unless critical — they restrict scheduling flexibility.
3. Prefer preferred for Flexibility: Helps Kubernetes find best-effort placements without deadlocks.
4. Abstract Names with Labels: Don’t hardcode node names — use meaningful, reusable labels.
5. Match Local Paths Exactly: If using hostPath, ensure PVs point to actual, consistent paths across node restarts.
6. Test in Staging First: Don’t blindly deploy affinity constraints in prod. Validate them in test environments.
7. Combine With Other Constraints: Use affinity rules alongside resource requests, taints/tolerations, and priority classes.

🎯Conclusion

Node affinity is crucial when working with static, localized storage such as hostPath or custom volumes. If your PersistentVolume and pod don’t land on the same node, your workload simply won’t run as expected.

Understanding and properly configuring node affinity ensures your pods are scheduled correctly, preventing those frustrating “Pending” states and storage issues. This knowledge is invaluable for anyone working with Kubernetes clusters of any size, especially in development environments like KIND or Minikube.

🚀 Call to Action

Did this help you debug a stuck pod? Save the article, share it with your dev team, and help spread real-world Kubernetes debugging insights.

Got some insights related to PVs or node scheduling? Drop it in the comments and let us connect.

Written by: Gibran

Tested on: KIND