Kubernetes etcd Troubleshooting

Run etcdctl via kubectl debug without SSH

No SSH? No etcdctl? Run etcdctl Instantly in Kubernetes

In many Kubernetes environments (especially managed clusters), you may not have SSH access to control-plane nodes or local access to the **etcdctl binary**. This makes etcd Troubleshooting, snapshot validation, and cluster health checks difficult.

What is etcd?

etcd is a strongly consistent, distributed key-value store used as the primary, persistent backing store for all **Kubernetes** cluster data, including configuration, state, and metadata.

What is etcdctl?

etcdctlis the primary command-line client for interacting with etcd over a network. It is used for day-to-day operations such as managing keys and values, administering the cluster, health checks, member management, key reads, snapshots saves.

What is etcd-utils?

What if you don’t have SSH access to the node, or the etcdctl binary is missing? Just use **etcd-utils .**

🐳 etcdct-utils Docker repo here

etcd-utils is a Docker image designed to run as a debug container for etcd operations, i.e. etcdctl endpoint status --cluster -w table

kubectl debug nodes/NODE_NAME -it \
  --profile=sysadmin \
  --image=dejanualex/etcd-utils:v1.0.6 \
  --image-pull-policy=Always \
  -- etcdctl endpoint health --cluster -w table

What makes etcd-utils special?

Using etcdctl in Kubernetes, usually means dealing with TLS flags and cert paths (--cacert, --cert, --key) just to run a command.

etcd-utils removes that friction:

• Cert paths are baked into the image
• Works out of the box with k3s & k8s
• Run etcdctl commands directly—no extra flags
• Use it as a debug container → faster, no SSH needed

etcdctl endpoint health — cluster -w table

📓 etcdctl commands list here

etcdctl endpoint status --cluster -w table
etcdctl endpoint health --cluster -w table
etcdctl member list -w table

Troubleshooting

If you are seeing connection refused or context deadline exceeded errors, the debug container is most likely running on a worker node instead of a control-plane node.

The etcd-utils debug pod must run on a control-plane node, because those nodes run the local etcd instance and have the required etcd TLS certificates available on disk.

⚠️ kubectl debugcreates a debug pod with a name derived from the node name, so remember to delete the pod once you’re done debugging.

NAME                               READY   STATUS      RESTARTS   AGE
node-debugger-controlplane-7w84c   0/1     Completed   0          19s

etcdutl vs. etcdctl

**etcdutl is an administration utility designed to operate directly on etcd data files, including migrating data between etcd versions, defragmenting the database, restoring snapshots, and validating data consistency. For operations over a network, use `etcdctl`**.

etcdctl vs etcdutl

Sources

• understanding-etcdctl-and-etcdutl
• using etcdctl
• kubectl debug container