Kubernetes series - Readiness and Liveness probes

Kubernetes uses Readiness and Liveness probes to manage pod health and traffic handling effectively. These probes help determine when to restart a pod or mark it as ready to receive traffic.

If no Readiness probe is specified, a pod is considered ready as soon as all its containers are running. However, this assumption can be misleading because HTTP servers may take some time to become fully operational after starting.

Similarly, without a Liveness probe, Kubernetes cannot detect when a pod is stuck or failing, which may lead to prolonged downtime as the pod is not restarted automatically.

In this article, I’ll explain these probes in detail and provide practical examples.

Readiness probe

Readiness probes determine when a container is ready to handle incoming traffic. They are particularly useful for applications that require extra time to complete initial tasks, such as establishing network connections, loading essential files, or warming up caches.

When a readiness probe indicates that the container is ready, Kubernetes begins routing traffic to the pod. If the probe reports that the container is not ready, Kubernetes temporarily removes the pod from all relevant service endpoints, ensuring it doesn’t receive traffic until it is fully prepared.

Liveness probe

Liveness probes determine when a pod needs to be restarted. If a pod consistently fails its liveness probe, the kubelet automatically restarts it.

This is particularly useful when a pod encounters repeated failures and manual intervention is not feasible. By restarting the pod, Kubernetes helps minimize downtime and attempts to restore normal functionality automatically.

Example

This example uses a .NET project that simulates a long startup. You can find the project here: Net-Health-Checks. A Docker image named maiconghidolin/net-healthchecks has already been created.

Create the Server Deployment

Start by creating a deployment for the server and a service to expose the pods:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: server
  labels:
    app: server
spec:
  selector:
    matchLabels:
      app: server
  replicas: 1
  template:
    metadata:
      labels:
        app: server
    spec:     
      containers:
      - name: server
        image: maiconghidolin/net-healthchecks:latest
        imagePullPolicy: Always
        ports:
          - containerPort: 8080
        command:
          - sh
          - -c
          - |
            if [ ! -f /app/Resources/text.txt ]; then
              mkdir -p /app/Resources && echo "Response Ok" > /app/Resources/text.txt
            fi
            dotnet HealthChecks.dll

---
apiVersion: v1
kind: Service
metadata:
  name: server
  labels:
    app: server
spec:
  selector:
    app: server
  ports:
    - protocol: TCP
      name: http
      port: 80
      targetPort: 8080

Create the Consumer Client

apiVersion: apps/v1
kind: Deployment
metadata:
  name: client
  labels:
    app: client
spec:
  selector:
    matchLabels:
      app: client
  replicas: 1
  template:
    metadata:
      labels:
        app: client
    spec:
      containers:
      - name: client
        image: curlimages/curl:latest
        command: ["/bin/sh", "-c"]
        args: ["while true; do curl -s -w 'HTTP Code: %{http_code}\\n' <http://server/home> || echo 'Request failed'; sleep 1; done"]

Apply the configurations

Deploy the server and client to Kubernetes:

kubectl apply -f server.yaml
kubectl apply -f client.yaml

View Client Logs

To observe the logs from the client, run the following:

kubectl attach -i -t -n default <client-pod-name> -c client

You should see logs with the Response Ok message.

Scale the server

Scale the server deployment to 5 replicas:

kubectl scale deployment server --replicas 5

At this point, you may notice some Request failed responses. This happens because the new pods are still starting, and Kubernetes is routing traffic to them before they’re ready.

Add Readiness Probe to Server Deployment

To address this issue, add a readiness probe to ensure that the server pods are only marked as ready when they can actually handle traffic. Add the following to the server deployment under the container spec:

readinessProbe:
  httpGet:
    path: /home
    port: 8080
  initialDelaySeconds: 2
  periodSeconds: 3
  failureThreshold: 3

This configuration introduces a 2-second delay before checking readiness, and it will retry every 3 seconds for up to 3 failures. Once the server responds with a 200 OK status, the pod will be considered ready.

Reapply the updated server deployment and scale the server up again.

This time, the pods will take longer to be marked as ready, and you should no longer see the Request failed messages.

Simulating a failure

First, scale the server back down to a single replica, then simulate a failure in the pod by removing the text.txt file:

kubectl exec <server-pod-name> -- sh -c "rm Resources/text.txt"

After this, you’ll begin to see HTTP 500 responses, as the server is now failing. It will continue to fail until manual intervention is made.

To fix it, recreate the text.txt file:

kubectl exec <server-pod-name> -- sh -c "echo 'Response OK' > Resources/text.txt"

Once the file is restored, everything will work again. This highlights the importance of using multiple replicas — if one pod fails, others can continue to serve traffic, ensuring the service stays up.

However, manually intervening isn’t ideal. To avoid this, we can configure a liveness probe to automatically restart the pod after a failure.

Configuring the Liveness Probe

In the server deployment, under the container spec, add the following configuration for the liveness probe:

livenessProbe:
  httpGet:
    path: /home
    port: 8080
  initialDelaySeconds: 3
  periodSeconds: 4
  failureThreshold: 5

This is similar to the readiness probe, but with a higher failure threshold to allow more time for recovery. In this case, it will attempt 5 requests every 4 seconds.

Apply the updated server configuration:

kubectl apply -f server.yaml

Next, simulate the failure again by removing the text.txt file:

kubectl exec <server-pod-name> -- sh -c "rm Resources/text.txt"

After a short delay, you’ll see some requests failing. When the liveness probe fails (after approximately 20 seconds), the pod will be restarted, and all requests will start working again.

Scaling the Server and observing the behavior

Now, scale the server to 2 replicas and remove the text.txt file from one replica:

kubectl scale deployment server --replicas 2
kubectl exec <server-pod-name> -- sh -c "rm Resources/text.txt"

If you observe the traffic, you’ll notice that the readiness probe helps to route traffic away from the failing pod. Meanwhile, the liveness probe will eventually restart the unhealthy pod, ensuring that the service remains operational.

Conclusion

In conclusion, Kubernetes’ Readiness and Liveness probes are essential tools for ensuring application stability and minimizing downtime. The Readiness probe ensures that traffic is only routed to pods that are fully ready, preventing failed requests during pod startup. On the other hand, the Liveness probe automatically detects and restarts failing pods, reducing the need for manual intervention. By combining both probes, we can create a more resilient system where services remain available, even in the face of temporary failures. This approach improves the overall reliability of applications running in Kubernetes.

In the next article, I will show how to create a cluster with Kind. Stay tuned and see you next time.