🛰️ Inside the Terminal Tunnel: The Deep-Dive Mechanics of kubectl exec

In my 12 years of architecting and debugging mission-critical infrastructure at scale, I have noticed that kubectl exec is arguably the most frequently run command that engineers completely take for granted. To a junior developer, it feels like an instant SSH tunnel directly into a container. But as a Lead SRE, I know that hitting enter on that command triggers a high-stakes, multi-component relay race across the API server, the worker node, and the Linux kernel.

If you don’t understand the complex plumbing : from multiplexed SPDY/WebSockets streaming to container runtime namespace attachments , you will find yourself completely paralyzed when your debugging tools hang during a production outage.

kubectl exec -it my-pod — /bin/bash is one of the most used commands in Kubernetes operations. It looks deceptively simple , you type it and a shell appears. But under the hood, this single command triggers a multi-component chain that spans your laptop, the cloud control plane, the worker node, the container runtime, and the Linux kernel. Understanding exactly how it works is essential for debugging connectivity issues, understanding security boundaries, and knowing why certain containers cannot be executed at all.

Dissecting the Command

Before tracing the internal journey, every part of the command has a specific meaning that influences the components involved downstream.

kubectl exec -it my-pod --namespace production -- /bin/bash

What About Container Selection?

If the pod has multiple containers, kubectl exec defaults to the first container listed in the pod spec. To target a specific container, use the -c flag:

kubectl exec -it my-pod -c sidecar-container -- /bin/sh

On GKE and EKS, pods commonly have multiple containers — application container, logging sidecar (Fluentd/Fluent Bit), service mesh sidecar (Envoy), or Calico network policy enforcement sidecar. The -c flag is critical in these environments.

What kubectl Sends to the API Server

kubectl exec does not open a direct connection to the node or the pod. It connects exclusively to the Kubernetes API server — the same HTTPS endpoint used for every other kubectl operation. This is important: even in cloud environments where worker nodes may be in private subnets unreachable from your laptop (which is standard on both GKE and EKS), kubectl exec works because it only needs to reach the API server’s public or private endpoint.

The HTTP POST to the exec Subresource

kubectl constructs an HTTP POST request to the pod’s exec subresource. The URL structure is:

POST /api/v1/namespaces/{namespace}/pods/{name}/exec

The query parameters carry the exec configuration:

POST /api/v1/namespaces/production/pods/my-pod/exec
  ?command=/bin/bash
  &stdin=true
  &stdout=true
  &stderr=true
  &tty=true
  &container=app

This request is sent over HTTPS using the credentials and server address from your kubeconfig file (~/.kube/config on Linux/Mac, %USERPROFILE%.kube\config on Windows). On GKE, the server address is the GKE control plane endpoint, accessible via Google’s managed control plane network. On EKS, it is the EKS cluster API server endpoint, accessible over the internet or via VPC private endpoint depending on your cluster’s endpointPublicAccess and endpointPrivateAccess settings.

How kubectl Reads Your kubeconfig on GKE and EKS

API Server Receives the exec Request

The exec request enters the same authentication and authorization pipeline as every other Kubernetes API request. There are no shortcuts for exec — if anything, exec requires more careful RBAC configuration because it grants direct shell access to running containers.

Authentication

The API server validates the Bearer token. On GKE, it verifies the Google OAuth token against Google’s token introspection endpoint. On EKS, it verifies the AWS STS pre-signed URL and maps the IAM identity to a Kubernetes subject via the aws-auth ConfigMap or EKS access entries (the newer mechanism replacing aws-auth).

RBAC: The pods/exec Permission

After authentication, the API server evaluates RBAC. kubectl exec requires the exec verb on the pods/exec subresource — not just on pods. This is a separate permission. A role that allows get, list, watch on pods does NOT grant exec access. exec must be explicitly granted.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-exec-role
  namespace: production
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create"]

The verb for exec is create, not exec. This is because the exec subresource is a POST operation (HTTP POST = create in REST semantics). On GKE and EKS, engineers often make the mistake of granting pods/* thinking it covers exec — it does not. pods/exec must be listed explicitly.

Admission Control

After RBAC, the exec request passes through admission webhooks if any are configured. Organisations running GKE or EKS in regulated environments often install OPA Gatekeeper or Kyverno policies that restrict exec access further — for example, blocking exec into pods in the production namespace unless the request comes from a specific service account used by the break-glass incident response process.

The WebSocket / SPDY Protocol Upgrade

Once the API server has authenticated and authorised the request, it needs to respond in a way that allows bidirectional streaming — your stdin going to the container, and stdout/stderr coming back. A standard HTTP request-response cycle cannot do this. The connection must be upgraded to a streaming protocol.

The API server responds with HTTP 101 Switching Protocols and upgrades the connection to one of two protocols:

The API Server Forwards to the kubelet

The API server knows which node the target pod is running on because the pod object’s spec.nodeName field is set when the pod is scheduled. The API server opens a connection to that node’s kubelet HTTPS endpoint and forwards the exec request, including the streaming connection.

kubelet Network Accessibility on GKE and EKS

kubelet’s exec Handler

The kubelet exposes an /exec HTTP endpoint on its HTTPS server (port 10250). When the forwarded exec request arrives, the kubelet:

1. Validates that the named pod is running on this node.
2. Validates that the specified container name exists in the pod.
3. Looks up the container’s container ID from its local pod cache.
4. Calls the CRI (Container Runtime Interface) to execute the command in the specified container.
5. Sets up the streaming channels (stdin/stdout/stderr) between the WebSocket/SPDY connection from the API server and the process it is about to start.

The kubelet does not start the process itself — it delegates to the container runtime via CRI. But it manages the streaming connection: it is the kubelet’s job to bridge the network stream coming from the API server with the I/O of the newly created process.

The Container Runtime Interface

The kubelet communicates with the container runtime via CRI — a gRPC API that standardises how Kubernetes interacts with different container runtimes. On GKE, the runtime is containerd. On EKS, it is also containerd (Docker was deprecated as the Kubernetes runtime and removed in Kubernetes 1.24).

The CRI ExecSync and Exec gRPC Calls

For kubectl exec, the kubelet calls the runtime’s Exec or ExecSync RPC. The difference:

For kubectl exec -it with a shell, the kubelet uses the Exec RPC. containerd returns a URL for a streaming server it manages, and the kubelet connects to this URL to relay the I/O between the streaming channel from the API server and the process containerd starts.

What Happens Inside containerd

containerd receives the Exec request with the container ID and the command to run (/bin/bash). It performs the following:

1. Looks up the container’s running state using the container ID. The container must be in Running state — exec does not work on stopped or completed containers.
2. Calls the OCI runtime (runc or crun) to create a new process inside the container’s existing set of Linux namespaces. This is the key distinction from starting a new container.
3. If TTY was requested, containerd calls openpty() to allocate a pseudo-terminal pair. The master end is held by containerd (and streamed to the client), the slave end is set as the controlling terminal of the new process.
4. The new process (/bin/bash) is started with its stdin, stdout, and stderr connected to the streaming channel (or the PTY slave device if TTY is enabled).

Linux Namespaces: What the Process Joins

This is the most important technical point about kubectl exec: the /bin/bash process is NOT a new container. It is a new process that joins the existing namespaces of the target container. Specifically:

What -i and -t Actually Do

The -i and -t flags look like simple switches but they control the entire interactive experience of the exec session. Understanding them prevents the most common kubectl exec frustrations.

The -i Flag: Keeping stdin Open

Without -i, kubectl closes its end of the stdin stream immediately after starting the exec. The container process receives EOF on its stdin. For a shell like bash, receiving EOF on stdin causes it to exit immediately. The exec session terminates in milliseconds.

With -i, kubectl keeps the stdin stream open and forwards your keyboard input through the WebSocket/SPDY channel to the container process’s stdin. You can type commands and they are received.

The -t Flag: Pseudo-Terminal Allocation

Without -t, the exec runs without a TTY. The container process has a non-terminal stdin. Bash in this mode behaves differently: no PS1 prompt is printed, tab completion does not work, command line editing (cursor movement, history) does not work, programs that require a terminal (vi, top, less) fail with ‘not a terminal’ errors, and output from commands like ls does not use colour formatting.

With -t, containerd calls openpty() to create a pseudo-terminal pair. The slave end is attached to the process’s stdin/stdout/stderr. The master end is streamed to kubectl, which sets its own terminal to raw mode and forwards bytes directly. The result is a fully interactive terminal experience: coloured output, tab completion, readline editing, terminal-aware programs.

Why /bin/bash — and Why It Sometimes Fails

The command after — is executed directly by the container runtime as a new process. It is not passed to a shell for interpretation. This means the binary must exist at the exact path specified in the container’s filesystem.

Container Images and Shell Availability

Why Bash Specifically?

bash (Bourne Again Shell) is chosen over other shells when available because:

• Tab completion: bash supports programmable tab completion for commands, file paths, and even kubectl resources (with bash-completion installed).
• History: bash maintains command history in memory and can persist it to ~/.bash_history.
• Readline editing: cursor movement, Ctrl+R reverse search, and all standard readline shortcuts work in bash.
• Arrays and associative arrays: bash 4+ supports these, making it more powerful for ad-hoc scripting inside a container during debugging.
• PS1 customisation: coloured, informative prompts are easier to configure in bash.

However, sh is usually the safer default because it exists in nearly every Linux container image. When writing scripts or documentation meant to work across different container images, /bin/sh is more portable than /bin/bash.

Debugging Distroless Containers on GKE and EKS

Google’s distroless images (used extensively in GKE environments) contain no shell, no package manager, and only the application binary and its runtime dependencies. kubectl exec — /bin/bash fails immediately. The modern solution is ephemeral debug containers via kubectl debug:

kubectl debug -it my-pod - image=busybox - target=app-container

This injects a temporary busybox container (which has a shell) that shares the process namespace of the target container. You can inspect files, check network connectivity, and run commands against the running process — all without requiring a shell in the production image. On GKE with Autopilot, ephemeral containers are supported as of Kubernetes 1.25+.

kubectl exec is a High-Privilege Operation

Granting pods/exec permission is equivalent to granting direct shell access to a running container. In production environments on GKE and EKS, exec access should be treated with the same level of caution as SSH access to a server. This section covers the security controls that should be in place.

RBAC: Least-Privilege exec Access

The pods/exec subresource should never be granted cluster-wide unless absolutely necessary. Instead:

• Grant exec only to specific namespaces (never cluster-wide).
• Bind exec permissions to specific service accounts used for incident response, not to regular developer roles.
• Use ResourceNames in the RBAC rule to restrict exec to specific pod names if the set of debuggable pods is known in advance.
• On EKS, map specific IAM roles to restricted Kubernetes groups via the aws-auth ConfigMap or EKS access entries — developers get read-only, SREs get exec.
• On GKE, use IAM conditions to restrict container.pods.exec permission to specific time windows or specific requestor identities.

Audit Logging

Every kubectl exec call is recorded in the Kubernetes audit log. On GKE, audit logs are written to Cloud Logging automatically. On EKS, audit logs must be enabled explicitly in the cluster’s logging configuration and are written to CloudWatch Logs. The audit log entry includes:

• The user identity (Google account on GKE, IAM ARN on EKS).
• The exact command executed (the — /bin/bash part).
• The target pod name, namespace, and container.
• The source IP of the kubectl client.
• The timestamp.

These logs are the forensic record for any incident involving interactive access to production pods. Ensure they are retained and monitored.

Calico NetworkPolicy and exec

Calico NetworkPolicy does not block kubectl exec. exec traffic travels from your laptop to the API server and from the API server to the kubelet — it does not traverse the pod’s network namespace directly. NetworkPolicy controls what the pod’s application network traffic can do, not what commands can be run inside the pod via the control plane path.

However, what you do inside an exec session is subject to the pod’s network policies. If you run curl http://internal-service from inside the exec’d shell, that curl request goes through the pod’s eth0 interface and is evaluated by Calico’s NetworkPolicy rules exactly as if the application had made the same request.

Every Step from Typing the Command to Getting a Shell

🏁 Final Thought: The Illusion of Proximity

In my 12 years of experience, I’ve realized that a senior engineer never forgets the abstractions supporting their tools. kubectl exec creates the seamless illusion that you are sitting directly inside the container, but it is actually a highly sophisticated, multi-hop stream. When you understand the handshake between the API server, the kubelet, and the Linux kernel namespaces, you stop looking at the terminal as a simple prompt and start respecting it as a masterfully engineered data pipeline.

Level Up Your Knowledge! 🚀

• 🎥 Watch: My “Latency Audit” Live Stream. I find 300ms of hidden lag in a production GKE cluster. [decodewithvivek]
• 📸 Instagram Tech Bites: 60-second Reels on Redis Caching and gRPC vs REST. [@decodewithvivek]
• ✍️ Master the Game: For deeper mentorship on acing your next System Design interview, let’s talk. 👉 **Connect with Vivek on Topmate**

If you found this helpful, feel free to leave a clap 👏 , highlight key points, drop a comment, or follow me for more insights. Let’s grow together — knowledge shared is progress multiplied!

💬 Enjoyed this post or found it helpful? **you can buy me a coffee here** ☕️