Kubernetes Controllers Explained

In Kubernetes, controllers are one of the most important concepts to understand. So let’s break it down in a simple and clear way.

A Kubernetes controller is basically a piece of software that runs a continuous control loop. It constantly watches what’s happening in your cluster, compares it with what you have declared, and then takes actions to bridge any gap between the current state and the desired state.

The idea is simple:

• We write a declarative configuration (a manifest file). That file describes what needs to be deployed.
• Kubernetes then combines this with reactive reconciliation, and this is where controllers come into the picture.
• When declarative config and reactive reconciliation work together, we get self-healing systems. Kubernetes will actively maintain the application in its desired state.

How This Works (The Control Loop)

This whole magic happens because of control loops.

Here’s the simple flow:

1. The controller observes the current state of the resources it is responsible for.
2. It compares the current state with the desired state you declared in your manifest.
3. If both match, it waits for the next event to happen.
4. If they don’t match, the controller takes the corrective action needed to close that gap.

And once it finishes, it starts again from the beginning. This loop never ends : it keeps observing, comparing, deciding, and acting whenever necessary.

Because of this continuous loop, even if you deploy the application multiple times, or the application crashes, or the system processes the same event twice, the controller will not create duplicate resources. Even if you run it 100 times, the end result should always be just one.

A common question is: How does the controller actually know that something happened? It doesn’t keep pulling metrics from the API server every time as that would be inefficient.

How Kubernetes controllers actually watch events ?

Controllers don’t just emit events; they consume Kubernetes events as first-class resources to react to whatever is happening inside the cluster.

Think of it like they will listen to whats happening and announce what they are doing!

Here’s how the event-consumption pattern works:

1. When something happens in the cluster, an event is created by the Kubernetes API. This could be anything — Kubernetes pulling an image, the scheduler trying to place a Pod, a container failing to start, a Pod getting killed, and so on. Basically, any meaningful action inside the cluster usually generates an event.
2. Controllers, on the other hand, set up event watchers. They monitor specific event resources using labels and selectors so they only watch the events they care about.

There are mainly two types of events:

a) Normal events — things like “Pod created,” “Image pulled,” “Container started,” etc.

b) Warning events — failures like “Failed to create pod,” “Back-off,” “OOM,” and similar issues.

• Events are spread across the cluster. They can come from nodes, Pods, containers, schedulers — almost anything.
• But because events are literally everywhere, controllers need smart filtering to focus only on what actually matters to them.
• Another important part is that Kubernetes avoids creating duplicate events, which could cause chaos. This is done using keys to track events.
• Along with this, there is rate limiting, which implements backup strategies to avoid overwhelming.
• After all this filtering and handling, the controller finally processes the event. Based on the event, it might update some internal state, create a Pod, modify a resource, trigger another action, send an alert, or whatever logic it’s designed to handle.

Event Emission

-> Controllers don’t just consume events — they also emit events to broadcast what they’re doing to the rest of the cluster. This helps with observability, creates audit trails, and allows other controllers or components to react when needed.

So how does this work?

• When this happens, the controller uses the event recorder interface to create a new event. It starts when a controller performs some operation maybe scaling replicas, creating a Deployment, updating a Pod, or anything similar.
• Controllers emit different categories of events depending on what’s happening
• Operational events — normal actions like “Deployment created,” “Service created,” “Pod created,” etc.
• Warning events — issues like “Pod cannot be created,” “Insufficient resources,” “Failed scheduling,” and so on.
• When these warning events are emitted, they often trigger reactions across the cluster.

For example :

A “FailedScheduling” event may trigger the Cluster Autoscaler to spin up new nodes.

A Pod constantly crashing may cause replicas to be recreated. This is where Kubernetes shows its self-healing behavior.

representation of the key component

Every event has four main components like shown above :

1. Object — the resource involved (for example, a Pod).
2. Type — usually Normal or Warning.
3. Reason — a short reason like “Created,” “Failed,” “BackOff,” etc.
4. Message — a detailed human-readable explanation

• An important thing to remember is that events don’t live forever. “Kubernetes automatically deletes events after 1 hour by default”.
• If you need longer retention, you must ship them to external watchers

Event Aggregation

Kubernetes also performs event aggregation to reduce noise. If the same event keeps happening repeatedly, Kubernetes will group them instead of spamming you with hundreds of identical events.

For example, if a Deployment keeps failing to pull an image, Kubernetes won’t create 50 separate events — it will show one event and increase the count showing how many times it happened.

Based on the event structure, Kubernetes decides what type of event to emit, whether its a succesful one or a failure Once emitted:

• The event is stored in the API server.
• It becomes available to the whole ecosystem.
• Monitoring tools, controllers, and even simple CLI commands like kubectl events can consume it.

Reconciliation Loop :

This is basically how the controller actually works .

• The reconciliation loop continuously ensures that the reality of the cluster matches the desired state
• Events from cluster like periodic queues, resource updates, crashes, or someone directly modifying resources. Anything that updates the state of a resource can trigger reconciliation.
• Whenever any of these triggers fire, they all funnel into a single reconcile function. You can think of this function as the main entry point for handling all resource state changes.
• No matter what the event is, it goes through reconciliation, which helps maintain consistent behavior across the cluster.

Step-by-Step Flow

1. Reconciliation gets triggered This can happen due to many reasons — a Pod crash, changes by other controllers, a periodic timer, or someone updating a Deployment directly.
2. The controller fetches the resource It queries the API server to get the current state. It uses the namespace, name, and resource type to identify exactly which resource triggered the reconciliation. For example, if someone updates a Deployment in the production namespace, that specific Deployment becomes the target.
3. Controller checks resource existence If the resource no longer exists (maybe someone deleted it), the controller needs to handle cleanup. If it exists, then the core logic begins.
4. Compare desired vs current state This is the actual reconciliation logic. The controller checks whats written in manifest file, the current state and the desired state

• For example, if the manifest says 5 replicas, but only 3 are running in the cluster, the controller detects the drift.

5. Interaction planning Now the controller decides how to bridge the gap between the current and desired states. It may decide to: Create missing replicas , update images, adjust configurations, fix mismatches . The main job is always the same — make the cluster match the declared state.

6. Execute the actions Once the plan is ready, the controller applies the required changes — like creating Pods, updating the Deployment, fixing configurations, etc.

7. Verify the outcome After applying the changes, the controller checks whether they were successful.

• If it succeeded, it updates the status to show that everything is correct.
• If it failed, it records the failure in the resource status, so you can see it when inspecting the resource.

8. Return the final status Regardless of success or failure, the controller returns the updated status and the loop continues.

Kubernetes Watch API :

How controllers actually watch for changes happening inside the cluster?

Pictorial representation of the flow

1. The Kubernetes API server exposes a special API endpoint that streams resource changes in real time. Instead of controllers pulling the API server again and again, the API server pushes notifications whenever something changes. So whenever there’s some update or event in the cluster, controllers get notified through these watch streams. But here’s the catch: Raw watch streams can be overwhelming for controllers to handle directly. This is where the Informer comes into the picture.
2. Informers sit between the API watch streams and the controller. They maintain a local copy (cache) of resources and provide clear, filtered event notifications when something changes. So instead of dealing with raw watch streams, the controller receives neat, organized notifications that tell it exactly what changed.
3. To avoid having separate informers for every controller, Kubernetes uses Shared Informers. A shared informer can watch multiple events at the same time, reducing load on the API server.
4. Shared Informers also maintain a local cache, which means controllers don’t have to constantly hit the API server to get resource details, they simply read from the shared cache.

How the Watch → Informer → Controller Flow Works

1. Watch API streams events in real-time.
2. Informer receives the stream, stores the current state in its cache, and sends clean notifications.
3. If multiple controllers care about the same resource, the Shared Informer notifies all of them.
4. Each controller has its own event handlers, which take the event and place it into a work queue.
5. Every controller gets its own work queue, so they can process events at their own pace.
6. From the work queue, the controller triggers reconciliation whenever needed.

This design keeps the system efficient, scalable, and avoids overwhelming the API server.

There’s something called the resync period. By default (every ~10 hours), even if nothing has changed, the informer triggers a full reconciliation of all resources.

This ensures that the actual state still matches the desired state, and no silent drifts go unnoticed.

Types of Events Controllers Receive

Controllers receive five types of events:

1. Added — when a new resource is created
2. Modified — when an existing resource is changed
3. Deleted — when a resource is removed
4. Bookmarked — lightweight checkpoints to help with efficient re-sync
5. Error — something went wrong (like resource stuck in Pending)

Resource Versions

To keep everything consistent, Kubernetes tracks resource versions. This prevents outdated updates from overwriting newer ones.

For example: If version 1 of a Deployment exists and someone tries to update using the old version 1 again, the API server rejects it.

This avoids overwriting the newer state and prevents unnecessary reprocessing.

Controller Communication

Pictorial representation of the Steps

• Controllers never work alone — they operate like a coordinated team. Even though they don’t directly talk to each other, they still manage to work together smoothly. For example, when you create an Ingress, multiple controllers start working behind the scenes, even though they aren’t aware of each other.
• The primary way they communicate is through event-driven coordination.

When you create an Ingress resource:

• The Ingress Controller sees the event and sets up the necessary load balancer configuration.
• Cert-Manager notices TLS annotations and provisions certificates.
• The TLS controller may update Pods or configurations to point to the new certificate or endpoint.

None of them “talk” to each other directly — they simply react to the events or changes happening in the cluster.

-> Controllers can also share ownership of resources using Custom Resources (CRDs).

-> Multiple controllers can watch the same custom resource, each responsible for a different part of the process.

Example:

• One controller handles Pod creation
• Another controller handles database provisioning
• Another controller handles backups
• Another controller handles monitoring

Each controller works on its own section of the resource, usually by updating different parts of the status field. They don’t conflict with each other because the API design ensures clear separation of responsibilities.

-> A lot of elegant coordination happens through labels and annotations.

• Labels are used to signal intent. For example: “I’m the controller managing this resource” or “This component is ready to take traffic.”
• Annotations carry metadata between controllers — configuration details, state information, or instructions that other controllers can read and act upon.

-> Other controllers read these signals and take action based on them.

So even without explicit communication, Kubernetes controllers work together in a clean, decoupled, event-driven way — almost like they’re part of the same team.