AI Assisted Automatic Shell Re-Entry In Kubernetes Pod

Written by — paranjan@in.ibm.com, surajit.majumder@ibm.com, Prodip.Roy2@ibm.com

Kubernetes is a widely adopted container orchestration platform that dynamically manages the life cycle of application pods. However, pod restarts or evictions — triggered by system failures, resource constraints, or cluster maintenance — cause abrupt termination of active shell sessions, requiring users to manually reconnect. This paper proposes a novel, automated solution that leverages machine learning to identify the most appropriate replacement pod and restore the shell session seamlessly. The system enhances operational efficiency by reducing manual intervention and aligns with the principles of resilience and automation in cloud-native environments. Experimental results and architectural design validate the effectiveness of the approach, demonstrating its potential to improve developer productivity and cluster usability.

Keywords — Kubernetes, Pod Restart, Shell Session, Automatic Reconnection, Pod Similarity, Machine Learning, Feature Encoding, Distance Metrics, Resource Usage Metrics, Cloud-Native Resilience

1. Introduction

A pod in Kubernetes can go down due to various reasons including: node issues (like high resource usage, node failure), application errors within the pod (crashes, unhandled exceptions), resource constraints (memory, CPU limits), network problems, image pull failures, faulty pod configurations, node eviction due to pressure, pod disruptions from cluster maintenance, and issues with the Kubernetes control plane; essentially, anything that disrupts the normal operation of the containerized application running within the pod can cause it to fail. When any pod goes down then any existing shell connection from a server(local laptop) to the pod also gets disconnected. The Kubernetes user then has to write command to list the existing list of pods for that name-space. Then the user has to copy another command using the newly copied pod name to “exec” into the new pod. This is cumbersome and wastes time unnecessarily. If pods restarting is a part of how Kubernetes(or any container orchestrator) is supposed to behave in a cloud native environment, then there should also be a way to automatically enter into another pod. Currently, the user is left stranded and disconnected from the pod it was connected to earlier.

2. Brief Description of the Invention

This invention provides a system for maintaining persistent shell connections to Kubernetes pods by intelligently reconnecting users to the most similar pod when a pod restart or failure occurs. The system continuously captures and stores key operational features of each pod — such as resource usage, lifecycle status, probe results, and node metrics — at fixed time intervals.

Upon shell disconnection, the Kubernetes shell client triggers a similarity search request to the kube-api-server. The server encodes pod features using one-hot encoding or other suitable categorical-to-vector encoding techniques.

These feature vectors are then compared using distance metrics (e.g., Euclidean distance, Cosine Similarity) to identify the pod most similar to the disconnected one. The system reconnects the shell session to this identified pod automatically, displaying a message to the user regarding the reconnection. Users can enable or disable this auto-reconnect feature at the command-line (kubectl) or Deployment configuration level. This mechanism ensures minimal disruption during pod restarts, improving user experience and operational continuity in Kubernetes environments.

2. 1 Feature Capture and Storage

At a predefined interval T, the system captures and stores the latest operational features of each running Kubernetes pod. These features include, but are not limited to:

1. Node IP address.
2. Subnet of the pod.
3. Pod state (Running, Pending, etc.).
4. Pod age or uptime.
5. Restart count.
6. Readiness and liveness probe statuses.
7. Pod-level resource usage: CPU, memory, network, disk, and PVC.
8. Node-level resource usage: CPU, memory, network.
9. These features collectively represent the current runtime context and behavior of each pod.

2.2 Shell Disconnection Event Detection

When a user’s shell session to a pod is terminated due to pod failure, eviction, or restart, the Kubernetes shell client detects this disconnection and initiates a similarity-based reconnection process.

2.3 Reconnection Request to Kubernetes API Server

Upon detecting disconnection, the shell client issues a request to the Kubernetes API server, asking for the most similar pod (within the same Deployment, Replica Set, or relevant grouping) based on the latest feature data.

2.4 Feature Encoding

The Kubernetes API server encodes the most recent features of all candidate pods into numerical vectors using:

1. One-hot encoding for categorical features (e.g., pod state, subnet).
2. Direct numeric representation for continuous features (e.g., CPU usage).
3. Or alternative encoding methods (e.g., label encoding, embedding based vectors) as applicable.

This transforms the pod state into a high-dimensional feature space suitable for similarity computation.

2.5 Similarity Computation

To determine the most similar pod, the system calculates the distance between the feature vector of the disconnected pod and those of all available candidate pods using:

1. Default: Euclidean Distance
2. Optionally: Cosine Similarity, Manhattan Distance, Hamming Distance, Jaccard Similarity, Chebyshev Distance, Mahalanobis Distance, or Minkowski Distance depending on feature characteristics or configuration.

2.6 Pod Selection

The candidate pod with the minimum computed distance from the disconnected pod is selected as the most similar pod.

2.7 Automatic Reconnection

The shell client is then automatically reconnected to the selected similar pod. The user receives a message indicating:

”Your connection to the previous pod was lost. You are now connected to the most similar pod in the same deployment.”

2.8 User Control

Users may configure the auto-reconnection feature through:

• Command-line flags in kubectl,

• Environment variables, or

• Deployment-level annotations or specifications.

This allows for opt-in or opt-out behavior based on user or system preferences.

3. Reduction to Practice

This research introduces a set of key methodological innovations and are given below:

System Description and Workflow

i. The system continuously captures and stores the latest features of each pod at a fixed interval of T seconds. The collected features include:

a) IP address of the node on which the pod is deployed.

b) Subnet address of the node’s IP.

c) Current state of the pod (e.g., Running, Pending, Failed).

d) Age or duration since the pod was created.

e) Number of restarts for the pod.

f) Readiness probe status of the pod.

g) Liveness probe status of the pod.

h) CPU usage of the pod.

i) Memory usage of the pod.

j) Network usage of the pod.

k) Persistent Volume Claim (PVC) usage of the pod.

l) CPU usage of the node hosting the pod.

m) Memory usage of the node hosting the pod.

n) Network usage of the node hosting the pod.

o) Disk usage of the pod.

ii. When a shell connection to a pod is unexpectedly disconnected (due to pod failure, restart, or eviction), the Kubernetes shell client sends a request to the kube-api-server to identify and connect to the pod most similar to the disconnected one.

iii. The kube-api-server encodes the latest features of each pod in the same Deployment using One-Hot Encoding or other appropriate encoding methods. One-Hot Encoding is a technique used to convert categorical data into numerical vectors where each unique category is represented by a binary vector, with a ”1” in the position corresponding to the category and ”0” in all other positions, essentially creating a sparse vector where only one element is ”on” (1) for each category; this allows machine learning models to process categorical data effectively. Other categorical-to-vector encoding techniques can also be used.

iv. After One Hot encoding of the categorical data of each pod to vector, we have a vector representing the latest features of each pod of the Deployment.

v. The system would calculate the distance of the vector of the pod from which the shell got disconnected with all other vectors of all pods and their latest features using Euclidean distance. There are other distance measurement algorithms which can also be used to find the distance between vectors are explained below:

Euclidean Distance is like measuring the straightest and shortest path between two points.

Imagine you have a string, and you stretch it tight between two points on a map; the length of that string is the Euclidean distance. It tells you how far apart the two points are without any turns or bends, just like a bird would fly directly from one spot to another. This metric is based on the Pythagorean theorem and is widely utilized in various fields such as machine learning, data analysis, computer vision, and more.

There are other distance measurement algorithms which can also be used to find the distance between vectors are explained below:

a. Cosine Similarity measures the similarity between two vectors by calculating the cosine of the angle between them. It focuses on the direction of the vectors, returning a value between −1 and 1, where 1 means identical direction and 0 means completely unrelated.

b. Manhattan Distance measures the distance between two points by summing the absolute differences of their corresponding coordinates. It reflects how far two points are in a grid-like path, similar to navigating city blocks.

c. Hamming Distance measures the number of positions at which the corresponding symbols in two equal-length strings differ. It is commonly used to compare binary strings or error-detection codes, with a higher distance indicating more differences.

d. Jaccard Similarity measures the similarity between two sets by dividing the size of their intersection by the size of their union. It ranges from 0 to 1, where 1 indicates that the sets are identical and 0 means they have no elements in common.

e. Chebyshev Distance measures the greatest absolute difference between the coordinates of two vectors. For example it reflects the minimum number of moves a king would take to move between two squares on a chessboard.

f. Mahalanobis Distance measures the distance between a point and a distribution, taking into account the correlations between variables. Unlike Euclidean distance, it scales and de-correlates the data using the covariance matrix, making it effective for identifying outliers in multivariate datasets.

g. Minkowski Distance is a generalized distance metric that encompasses other distances like Euclidean and Manhattan by varying a parameter p. It is defined as the pth root of the sum of the absolute differences of the coordinates raised to the power p; when p=1, it becomes Manhattan Distance, and when p=2, it becomes Euclidean Distance.

vi. The pod which smallest distance from the disconnected pod’s vector is identified as the most similar pod to the pod from which the shell got disconnected.

vii. The Kubernetes shell client is then connected to the identified pod.

viii. The user will see a message on the shell client mentioning that the previous pod had got disconnected and the shell is now connected to the current pod.

ix. The user will have the option of turning this automatic reconnection feature on or off at the kubectl or Deployment level.

4. Advantages of the Invention

The proposed invention offers several advantages and are given below:

4.1 Seamless User Experience

Automatically reconnects shell sessions to a similar pod after disconnection, eliminating manual intervention and reducing downtime for developers and operators.

4.2 Improved Operational Efficiency

Saves valuable time by avoiding repetitive commands to identify and connect to new pods, streamlining Kubernetes pod management during pod restarts or failures.

4.3 Robustness in Dynamic Environments

Works effectively in environments where pod rescheduling and restarts are frequent, providing resilience against common disruptions in cloud-native architectures.

4.4 Adaptive Pod Similarity Matching

Utilizes machine learning techniques with encoded pod features to intelligently find the most appropriate replacement pod, ensuring continuity and context preservation.

4.5 Configurable Behavior

Allows users to enable or disable the auto-reconnect feature per deployment or at the kubectl client level, providing flexibility to match varied operational preferences.

4.6 Reduced Human Errors:

Minimizes the risk of connecting to the wrong pod or namespace, thereby reducing mistakes and improving security and operational accuracy.

4.7 Enhanced Productivity

Developers and administrators can focus on core tasks without being interrupted by connection drops, improving overall productivity and workflow.

5. Conclusion

This method maintain persistent shell connections in Kubernetes environments by automatically reconnecting to the most similar pod following pod restarts or failures. The system continuously collects and encodes critical pod and node metrics, employing machine learning driven similarity measures to identify an optimal replacement pod.

This approach significantly reduces operational overhead by automating the reconnection process, thereby enhancing continuity and reliability in dynamic cloud-native environments. The ability to toggle the feature at both the client and deployment levels provides practical flexibility, making the solution adaptable to varied user requirements.

Ultimately, it improves the robustness and usability of Kubernetes shell interactions, contributing to more seamless and efficient container orchestration management.