Don’t Let Your Kubernetes Cluster Get Headless: A Guide to Control Plane High Availability

Kubernetes Architecture Knowledge Map

Understanding Kubernetes: A Comprehensive Guide

Overview

Kubernetes is designed to run containers at scale. While Kubernetes doesn’t run containers directly, it uses an abstraction called a Pod. A Pod can consist of one or more containers and is the smallest unit of compute that you can deploy to Kubernetes. Kubernetes runs these Pods on physical servers or virtual machines, depending on your environment, whether it is on-premises or in the cloud. These servers are known as Kubernetes nodes or workers.

You could potentially spin up one Pod here and there, but Kubernetes often manages hundreds or even thousands of these Pods. To orchestrate these containers, we use what used to be called masters, now known as control plane nodes. The control plane is responsible for making decisions about the cluster’s state, scheduling Pods, adding new nodes, and responding to events, such as automatically scaling Pods based on CPU or memory usage.

Impact of Control Plane Downtime

If the control plane goes down, your applications will continue to operate without downtime. For example, if you have an application exposed to the internet using ingress, clients will still be able to send requests and receive responses, with no noticeable interruptions. However, there are significant limitations:

• New Pods: Cannot be scheduled since the scheduler is part of the control plane.
• Scaling: Automatic scaling (horizontal or vertical) will not occur.
• Updates: You won’t be able to make updates, such as creating new deployments or updating existing ones.
• Self-Healing: Pods and nodes won’t be rescheduled automatically.
• Cron Jobs: Will not run, causing potential lag or issues.
• Third-Party Controllers: Like Airflow, which create Pods for batch jobs, will also be affected.

While the control plane is crucial, its failure doesn’t mean everything collapses immediately. You’ll have time to debug and recover before users notice.

High Availability

To avoid issues in production, we often run multiple control planes, typically three, with a load balancer in front of them. In this setup, only one control plane is active and makes decisions. If this node fails, another control plane node will become the leader and take over.

Control Plane Components

Etcd

The control plane stores the cluster state using the etcd database. Etcd is a distributed key-value store used by Kubernetes to keep track of cluster configuration and state.

• Consistency: Etcd ensures that once a write is acknowledged, all subsequent reads reflect that write. All nodes in the etcd cluster will see the updated value, regardless of which node is queried.
• High Availability: Provides fault tolerance, allowing the cluster to continue operating even if one node fails.
• Raft Consensus Algorithm: Manages leader election and ensures data consistency. An odd number of nodes is required to ensure a majority (quorum). The recommended number of nodes is three, though five nodes can increase reliability.
• Security: Uses TLS certificates for encrypting requests. You’ll need to renew certificates annually if using tools like kOps for provisioning.

Etcd stores data as key-value pairs and provides a watch mechanism for real-time updates, essential for maintaining the desired cluster state and synchronization.

Kube-Scheduler

The kube-scheduler watches for new Pods and schedules them to suitable nodes.

• Filtering: Excludes nodes that don’t meet the Pod’s requirements (e.g., CPU, memory).
• Scoring: Assigns scores to suitable nodes based on various criteria to find the best fit, such as resource availability and affinity rules.

Controller-Manager

The controller-manager regulates the cluster’s state by ensuring it matches the desired state defined by users. It includes:

• Node Controller: Manages node health and takes action if nodes become unavailable.
• Replication Controller: Ensures the specified number of Pod replicas are running. Creates or deletes Pods as needed.
• Endpoints Controller: Links Services with Pods and updates the Endpoints object.

Other controllers include Service Account and Token Controllers, Namespace Controller, Job Controller, DaemonSet, Deployment, ReplicaSet Controller, StatefulSet Controller, and CronJob Controller.

The controller-manager continuously monitors and adjusts the cluster state to match user-defined configurations.

Cloud-Controller-Manager

The cloud-controller-manager integrates Kubernetes with cloud providers. For instance, it creates AWS Elastic Load Balancers for Services of type LoadBalancer and manages cloud-specific resources like EBS volumes.

• Legacy: The in-tree cloud controller manager now primarily receives bug and security fixes. Most logic has shifted to external cloud controllers created by cloud providers.
• Modern Integration: Install external cloud managers like the AWS Load Balancer Controller or CSI EBS driver for better integration and feature updates.

API Server

The API server serves as the main interface for interacting with the Kubernetes cluster, handling all RESTful API requests from clients like kubectl, kubelet, and other components.

• API Gateway: Provides endpoints for API operations such as creating, updating, and retrieving resources.
• Authentication and Authorization: Verifies credentials and checks permissions.
• Validation: Ensures requests conform to expected schemas and rules.
• Admission Controllers: Apply custom policies and modifications to requests before they are processed.

The API server writes new resources to etcd and manages requests through various stages, including authentication, authorization, validation, and admission control.

Node Components

Kubelet

The kubelet is the primary node agent that ensures containers in a Pod are running as specified.

• Pod Management: Ensures containers are healthy and restarts them if necessary.
• Resource Management: Verifies nodes have enough resources for Pods.
• Storage Management: Manages volumes for persistent storage.
• Metrics and Logging: Collects and exposes metrics and logs.

Kube-Proxy

The kube-proxy manages network traffic and load balancing within the cluster.

• Network Proxy: Acts as a network proxy and load balancer for service traffic.
• Service Abstraction: Routes traffic to Pods and can use iptables or IPVS.
• Service Discovery: Updates routing rules based on new services and endpoints.
• Load Balancing: Distributes traffic among Pods and supports session affinity.

Container Runtime

The container runtime is responsible for running containers on a node. It manages their lifecycle, including creation, execution, and destruction.

• Container Runtime Interface (CRI): Defines the APIs for integrating different container runtimes with kubelet. Docker was previously the primary runtime but has been deprecated in favor of runtimes like containerd.