Remote Cluster Access by Kubernetes Controllers Using ~/.kube/config

Overview

Kubernetes controllers use a REST configuration to communicate with the Kubernetes API server over HTTP/HTTPS. As outlined in the Kubernetes documentation, there are multiple ways to create a REST client object. One approach is to set the KUBECONFIG environment variable to a file path, such as ../.kube/config.yaml. This file should include connection details such as the cluster URL, cluster name, user credentials, and authentication token, etc.

ConfigMap named “custom-config” having an example of config.yaml file

kind: ConfigMap
metadata:
  name: custom-config
data:
  config.yaml: |
    apiVersion: v1
    kind: Config
    clusters:
    - name: my-cluster
      cluster:
        server: https://remote-k8s-cluster:6443
        insecure-skip-tls-verify: true
    contexts:
    - name: my-context
      context:
        cluster: my-cluster
        user: my-user
    current-context: my-context
    users:
    - name: my-user
      user:
        token: abc...xyz

Now lets consider a Kubernetes controller named “custom-controller”, which supports the KUBECONFIG environment variable and runs as a Kubernetes pod. By mounting the above defined ConfigMap as a volume and setting KUBECONFIG as an environment variable, the controller’s default connection (https://kubernetes.default.svc or any other url) will be overridden, allowing it to connect to https://remote-k8s-cluster:6443 instead.

Patching the custom-controller Deployment/statefulset to Use a Custom Kubeconfig

To configure the custom-controller to use a specific KUBECONFIG, lets patch its Deployment using kubectl patch. This approach ensures that the controller connects to a remote Kubernetes cluster instead of the default in-cluster API server.

The following JSON patch:

1. Adds a volume that mounts a ConfigMap (my-kubeconfig) containing the config.yaml file.
2. Mounts the volume inside the container at /var/run/secrets/kubernetes.io/.kube.
3. Sets the KUBECONFIG environment variable to point to the mounted config file.

Command to Apply the Patch

kubectl patch deployment custom-controller --type='json' -p='[
  {
    "op": "add",
    "path": "/spec/template/spec/volumes/-",
    "value": {
      "name": "kubeconfig-volume",
      "configMap": {
        "name": "my-kubeconfig",
        "items": [
          {
            "key": "config.yaml",
            "path": "config"
          }
        ]
      }
    }
  },
  {
    "op": "add",
    "path": "/spec/template/spec/containers/0/volumeMounts/-",
    "value": {
      "name": "kubeconfig-volume",
      "mountPath": "/var/run/secrets/kubernetes.io/.kube",
      "readOnly": true
    }
  },
  {
    "op": "add",
    "path": "/spec/template/spec/containers/0/env/-",
    "value": {
      "name": "KUBECONFIG",
      "value": "/var/run/secrets/kubernetes.io/.kube/config"
    }
  }
]'

The above patch would work for a statefulset by changing resource kubectl patch deployment ..to kubectl patch statefulset ..

Once the above patch is applied, then the pod running the custom-controllerin the host k8s cluster would finally start communicating with remote k8s cluster https://remote-k8s-cluster:6443