K8 — Taints & Tolerations

Taints and Tolerations are controlled at pod scheduling phase in kubernetes.

Taints are applied at Node level and tolerations at a pod level

What is Taint ?

In kubernetes a taint is a property applied to a node that acts like a rule saying “Do not schedule pods here unless they have a matching toleration.”

Why Use Taints?

• To reserve nodes for specific workloads (e.g., GPU nodes for ML jobs).
• To protect nodes during maintenance or when under resource pressure.
• To control pod placement beyond normal resource checks.

What Happens When a Node Has a Taint?

• When you apply a taint to a node, Kubernetes adds metadata to that node:

key=value:effect

This tells the scheduler: “Do not place pods here unless they tolerate this taint.”

What is Tolerations and its Work

• A toleration in the pod spec acts like a filter override: It says: “I can ignore this taint.”
• When the scheduler checks nodes, If the pod has a toleration matching the node’s taint (same key, operator, and effect), the node is considered eligible.
• Important: Tolerations do not force pods onto tainted nodes — they only allow it. Other scheduling rules (affinity, resources) still apply.

Toleration Effects:

• Effect = NoSchedule → Pod is not scheduled on that node.
• Effect = PreferNoSchedule → Scheduler tries to avoid that node but may still schedule if no better option.
• Effect = NoExecute → Pod is evicted if already running and cannot be scheduled again.

Scheduler’s Role

• The Kubernetes scheduler evaluates all nodes for a pod.
• If a node has a taint and the pod does not have a matching toleration, that node is excluded.

Real-Time Flow

• Step 1: Pod creation triggers scheduling.
• Step 2: Scheduler checks all nodes, Removes nodes where taints conflict with pod tolerations. If pod tolerations match with any node taints, its consideres that node.

A toleration matches a taint if:

Key matches.

Operator matches (Equal or Exists).

Value matches (if operator is Equal).

Effect matches (NoSchedule, PreferNoSchedule, NoExecute).

• Step 3: If a node does not have any taints, or if the pod’s tolerations do not match the node’s taints, the scheduler evaluates other nodes without taints. It then applies standard scheduling logic — checking CPU, memory, and other resource requirements — before selecting the best node for the pod.

Example

Node taint:

kubectl taint nodes node1 dedicated=gpu:NoSchedule

Pod toleration:

tolerations:
- key: "dedicated"
  operator: "Equal"
  value: "gpu"
  effect: "NoSchedule"

Result: Pod can be scheduled on node1.