Kubernetes Registry Mirror Authentication: The Secret Weapon for Multi-Tenant Security

Stop storing registry credentials at the node level. Here’s how.

You’ve set up a private registry mirror to save bandwidth and speed up deployments. There’s just one problem: it needs credentials. And now those credentials are hardcoded into every node in your cluster.

🔥 Top Tech Jobs Are Hiring NOW — Don’t Miss Out.

🚀 Multiple Roles Available 👉 Apply & Secure Your Job

If you’ve been there, you know the pain. Getting registry mirror authentication right in Kubernetes is one of those problems that sounds simple but hides a surprisingly nasty security trap.

In this guide, we’ll walk through a production-grade solution: using Kubernetes-native Secrets and the CRI-O credential provider to authenticate private registry mirrors — with full namespace-level isolation. No more cluster-wide credential leaks.

The Problem with “Classic” Registry Mirror Auth

Traditional container registry authentication in Kubernetes has a fundamental limitation when working with private registry mirrors. The kubelet has no knowledge of mirror configuration. Mirrors are configured at the container runtime level through files like /etc/containers/registries.conf. There are currently no Kubernetes enhancement proposals to change this architecture, so mirror configuration remains outside the Kubernetes API.

This creates a technical trap: while you can use namespace-scoped secrets with imagePullSecrets for pulling from source registries, this doesn't work when you want to use private mirrors or pull-through caches. Mirrors require node-level configuration, forcing you to use global credentials.

The result? Three serious problems:

1. Security isolation is broken. Node-level credentials are accessible across all namespaces. A compromised pod in namespace A can potentially access credentials intended for namespace B.
2. Operational complexity increases. Platform teams must manage credentials centrally. Individual dev teams lose autonomy over their own registry secrets.
3. Compliance concerns arise. In regulated environments, credential sharing across project boundaries violates security policies. Audit trails become murky.

Enter the CRI-O Credential Provider

The [crio-credential-provider](https://github.com/cri-o/crio-credential-provider) solves this by plugging into the Kubelet Credential Provider Plugin API (stable since Kubernetes 1.26). Instead of static node-level auth files, the credential provider:

1. Parses the JWT service account token to extract the pod’s namespace
2. Discovers configured mirrors from /etc/containers/registries.conf
3. Queries the Kubernetes API for all dockerconfigjson Secrets in that namespace only
4. Generates a short-lived auth file at /etc/crio/auth/<namespace>-<sha256(image)>.json
5. Returns an empty success response to the kubelet — CRI-O discovers and uses the auth file, then cleans it up after the pull

This means Team A’s secret stays in Team A’s namespace. Team B gets nothing.

Why imagePullSecrets doesn't help here: imagePullSecrets is used by the kubelet to pull from the source registry. It has no effect on mirror registry authentication, which happens at the CRI-O layer. The credential provider bridges this gap.

The Key Feature Gate

The magic powering this is the KubeletServiceAccountTokenForCredentialProviders feature gate, available in Kubernetes 1.33. It allows the credential provider to use Service Account tokens to call the Kubernetes API — securely, without separate service accounts or kubeconfig files on every node.

Architecture Overview

The fundamental security property: the service account token is namespace-scoped, and the Kubernetes API enforces RBAC when the credential provider queries for secrets. Cross-namespace access is impossible by design.

Prerequisites

Before we start, make sure your environment meets these requirements:

• Kubernetes: 1.33+ (for KubeletServiceAccountTokenForCredentialProviders feature gate)
• CRI-O: 1.34+(minimum for namespace-scoped auth file support)
• kubectl: Latest
• crio-credential-provider(Binary): Latest from GitHub releases

Step 1: Install the CRI-O Credential Provider Binary

Download the binary and install it on all worker nodes in a dedicated directory:

# Download the latest release binary
REPO="cri-o/crio-credential-provider"
LATEST=$(curl -s https://api.github.com/repos/$REPO/releases/latest | jq -r '.tag_name')

wget "https://github.com/$REPO/releases/download/$LATEST/crio-credential-provider-linux-amd64" \
  -O /usr/libexec/kubernetes/credential-providers/crio-credential-provider

chmod +x /usr/libexec/kubernetes/credential-providers/crio-credential-provider

The binary directory must match the --image-credential-provider-bin-dir kubelet flag you'll set next.

Step 2: Configure the Kubelet

Create the credential provider configuration file. The matchImages field tells the kubelet which source registry domains trigger this provider. Because CRI-O resolves mirrors from registries.conf, you specify the source registry (e.g. docker.io), not the mirror hostname:

# /etc/kubernetes/credential-providers/config.yaml
apiVersion: kubelet.config.k8s.io/v1
kind: CredentialProviderConfig
providers:
  - name: crio-credential-provider
    matchImages:
      - docker.io    # Source registry — mirrors are resolved by registries.conf
      - quay.io      # Add any other source registries you mirror
    defaultCacheDuration: "1s"  # Required by API; caching is not actually used
    apiVersion: credentialprovider.kubelet.k8s.io/v1
    tokenAttributes:
      serviceAccountTokenAudience: https://kubernetes.default.svc
    cacheType: "Token"
    requireServiceAccount: false

Note: defaultCacheDuration and cacheType are required fields by the API, but the CRI-O credential provider does not cache credentials. A fresh auth file is generated for every image pull.

Now add the following kubelet startup flags (method varies by your distribution — edit /etc/systemd/system/kubelet.service.d/ or equivalent):

--image-credential-provider-config=/etc/kubernetes/credential-providers/config.yaml \
--image-credential-provider-bin-dir=/usr/libexec/kubernetes/credential-providers \
--feature-gates=KubeletServiceAccountTokenForCredentialProviders=true

Reload and restart the kubelet:

systemctl daemon-reload
systemctl restart kubelet

Step 3: Set Up the Private Registry Mirror

For testing, start an authenticated local registry mirror on a worker node:

# Create htpasswd credentials (username: myuser, password: mypassword)
mkdir -p /tmp/registry/auth
podman run --rm --entrypoint htpasswd httpd:2 -Bbn myuser mypassword \
  > /tmp/registry/auth/htpasswd

# Start the authenticated registry mirror
podman run -d -p 5000:5000 \
  --name registry \
  -v /tmp/registry/auth:/auth \
  -e REGISTRY_AUTH=htpasswd \
  -e REGISTRY_AUTH_HTPASSWD_REALM="Registry Realm" \
  -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
  docker.io/library/registry:2

# Seed it with a test image
podman login localhost:5000 -u myuser -p mypassword
podman pull docker.io/library/nginx:latest
podman tag docker.io/library/nginx:latest localhost:5000/library/nginx:latest
podman push localhost:5000/library/nginx:latest

Step 4: Configure the Registry Mirror in CRI-O

Configure CRI-O to redirect pulls from docker.io to your mirror via /etc/containers/registries.conf:

# /etc/containers/registries.conf
unqualified-search-registries = ["docker.io"]

[[registry]]
location = "docker.io"

  [[registry.mirror]]
  location = "localhost:5000"
  insecure = true   # Set false with TLS in production

CRI-O reads this file to know that any pull for docker.io/* should first attempt localhost:5000/*.

Step 5: Configure RBAC

RBAC is split into two layers: one for the node, one for namespace-level secret access.

Cluster-Level RBAC (for Nodes)

# cluster-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-credential-providers
rules:
  - apiGroups: [""]
    resources: ["serviceaccounts"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["*"]
    verbs: ["request-serviceaccounts-token-audience"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: node-credential-providers-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: node-credential-providers
subjects:
  # Add one subject per node — must match exact node identity
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: system:node:your-node-name

You need a ClusterRoleBinding subject for each node, or add multiple subjects to a single binding.

Namespace-Level RBAC (for Service Accounts)

# namespace-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secrets-role
  namespace: default
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: secrets-role-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: secrets-role
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: system:serviceaccount:default:default

Apply it

kubectl apply -f cluster-rbac.yaml
kubectl apply -f namespace-rbac.yaml

Step 6: Create the Namespace Registry Secret

Create a docker-registry Secret in the namespace where your pods will run. The credential provider will look here automatically:

# registry-secret.yaml
apiVersion: v1
kind: Secret
type: kubernetes.io/dockerconfigjson
metadata:
  name: my-secret
  namespace: default
data:
  # Decodes to: {"auths": {"http://localhost:5000": {"username": "myuser", "password": "mypassword", "auth": "bXl1c2VyOm15cGFzc3dvcmQ="}}}
  .dockerconfigjson: eyJhdXRocyI6eyJodHRwOi8vbG9jYWxob3N0OjUwMDAiOnsidXNlcm5hbWUiOiJteXVzZXIiLCJwYXNzd29yZCI6Im15cGFzc3dvcmQiLCJhdXRoIjoiYlhsMWMyVnlPbTE1Y0dGemMzZHZjbVE9In19fQo=

Or imperatively:

kubectl create secret docker-registry my-secret \
  --docker-server=http://localhost:5000 \
  --docker-username=myuser \
  --docker-password=mypassword \
  --namespace=default

Step 7: Test It — Deploy a Pod

Deploy a pod that pulls from docker.io (CRI-O will transparently redirect to the mirror):

# pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  namespace: default
spec:
  containers:
    - name: nginx
      image: docker.io/nginx   # No imagePullSecrets needed!

Apply it with kubectl

kubectl apply -f pod.yaml
kubectl get pod nginx
# NAME    READY   STATUS    RESTARTS   AGE
# nginx   1/1     Running   0          12s

Check the credential provider logs to see the full flow:

[crio-credential] Running credential provider
[crio-credential] Parsed credential provider request for image "docker.io/library/nginx"
[crio-credential] Got mirror(s) for "docker.io/library/nginx": "localhost:5000"
[crio-credential] Getting secrets from namespace: default
[crio-credential] Got 1 secret(s)
[crio-credential] Found docker config JSON auth in secret "my-secret" for "http://localhost:5000"
[crio-credential] Wrote auth file to /etc/crio/auth/default-7e59ad...fad5.json

Namespace isolation confirmed: any pod in a namespace without a matching secret will fail to pull from the mirror — without affecting other namespaces at all.

Security Deep Dive

The security model relies on multiple enforcement layers:

Even if an attacker could invoke the credential provider directly, it would only ever retrieve secrets from the compromised pod’s own namespace. Defense in depth.

What’s Next: OpenShift Native Integration

For OpenShift users, the CRI-O credential provider ships natively:

OpenShift 4.21 — configure via MachineConfig (Butane → Machine Config Operator):

# machine-config.bu (Butane format)
variant: openshift
version: 4.20.0
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-worker-crio-credential-provider-config
storage:
  files:
    - path: /etc/kubernetes/credential-providers/ecr-credential-provider.yaml
      mode: 0644
      overwrite: true
      contents:
        inline: |
          apiVersion: kubelet.config.k8s.io/v1
          kind: CredentialProviderConfig
          providers:
            - name: crio-credential-provider
              matchImages:
                - docker.io
              defaultCacheDuration: "1s"
              apiVersion: credentialprovider.kubelet.k8s.io/v1
              tokenAttributes:
                serviceAccountTokenAudience: https://kubernetes.default.svc
              cacheType: "Token"
              requireServiceAccount: false

Compile and apply:

podman run -it -v $(pwd):/w -w /w quay.io/coreos/butane:release machine-config.bu -o machine-config.yml
kubectl apply -f machine-config.yml
# Monitor rollout:
oc get machineconfigpool worker -w

OpenShift 4.22+ introduces the CRIOCredentialProviderConfig CRD — fully declarative, zero MachineConfig needed:

apiVersion: config.openshift.io/v1alpha1
kind: CRIOCredentialProviderConfig
metadata:
  name: cluster
spec:
  matchImages:
    - docker.io
    - quay.io

The Machine Config Operator handles all node configuration, kubelet restarts, and rolling updates automatically.

Summary

The result: a namespace-isolated registry mirror authentication system where:

• ✅ No credentials hardcoded on nodes
• ✅ No imagePullSecrets needed on pods
• ✅ Each team rotates their own registry secrets independently
• ✅ Auth files are ephemeral — created and deleted per pull
• ✅ Natural fit for air-gapped and compliance-heavy environments

Further Reading

• CRI-O Credential Provider GitHub
• CNCF Blog: Part I — Architecture and Implementation
• CNCF Blog: Part II — OpenShift Platform Integration
• Kubernetes Docs: Kubelet Credential Provider
• containers/registries.conf reference

✔️ Found this useful? Drop a few 👏 claps

✔️ Follow for more DevOps, K8s, and AWS insights

✔️ Connect with me on LinkedIn

Thank you for being a part of the community

Before you go:

👉 Be sure to clap and follow the writer ️👏️️

👉 Follow us: **Linkedin| [Medium](https://medium.com/codetodeploy)**

👉 CodeToDeploy Tech Community is live on Discord — **Join now!**

Disclosure: This post includes affiliate or partnership links.