Implementing a Kubernetes Mutating Webhook to Sync Namespace Labels to Pods

I created a mutating admission webhook that automatically copies labels from a namespace into the Deployment’s pod specification. Whenever a Deployment is created, the webhook intercepts the request, reads the labels defined on the namespace, and injects them into the pod template. As a result, every pod automatically inherits the correct team and environment labels without developers needing to add or manage them manually in their manifests.

This is the approach my company followed to handle workload scheduling at scale. The infrastructure and number of deployments were massive, so a dedicated team was responsible only for creating namespaces and adding the required labels to them. This was necessary because there could be thousands or even lakhs of namespaces, along with hundreds of thousands of Deployments created by many different teams. All these microservices together formed a large application, like Flipkart. Expecting every team to manually manage labels in every Deployment was not practical and led to mistakes. To solve this, a mutating webhook was introduced to automatically copy labels from the namespace into the pod spec, ensuring consistent labeling and correct scheduling across the cluster without relying on developers to remember or manage it themselves.

Kubernetes labels are powerful. They drive scheduling, monitoring, cost allocation, security policies, and operational automation. But there is a subtle problem many teams face in production:

Namespace labels do not automatically propagate to Pods.

If you rely on namespace-level labels like env=prod, team=platform, or cost-center=123, you quickly realize that:

• Deployments don’t inherit them
• Pods don’t reflect them
• Restarting workloads doesn’t help

In large clusters, this becomes a consistency and governance nightmare.

Github Repo

[embed]GitHub - Manohar-1305/mutating-webhook Contribute to Manohar-1305/mutating-webhook development by creating an account on GitHub.github.com

In this blog, we’ll build a production-safe Mutating Admission Webhook that automatically copies namespace labels to Pods at creation time, without:

• Infinite pod creation
• ReplicaSet churn
• Controller loops
• Manual intervention

Why a Mutating Webhook?

Kubernetes gives us admission webhooks to intercept API requests before objects are persisted.

A MutatingAdmissionWebhook can:

• Inspect incoming objects (Pods)
• Modify them safely
• Return a JSONPatch
• Let the API server continue

This makes it the only correct place to inject namespace labels into Pods automatically.

Design Principles (Critical)

Before writing code, we lock down three non-negotiable rules:

1. Mutate only on Pod CREATE
2. Be idempotent (never re-apply the same labels)
3. Never touch Deployment templates

Breaking any of these leads to:

• Endless ReplicaSet creation
• Hundreds of Pending Pods
• Broken clusters

[embed]

🔑 Enabling Mutating Admission Webhooks in kube-apiserver

You edited:

/etc/kubernetes/manifests/kube-apiserver.yaml

and changed:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.0.100:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.0.100
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    image: registry.k8s.io/kube-apiserver:v1.29.15
    imagePullPolicy: IfNotPresent

Change this parameter.

[root@master01 mutating-webhook]# cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep enable-admission-plugins
    - --enable-admission-plugins=NodeRestriction

with this:

--enable-admission-plugins=NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook

❗ Why this is REQUIRED (not optional)

Kubernetes does NOT guarantee that all admission plugins are enabled by default in kubeadm clusters.

If MutatingAdmissionWebhook is not enabled:

• Your MutatingWebhookConfiguration exists
• Your webhook Pod runs
• Your Service works
• BUT the webhook is NEVER CALLED

No logs No errors No mutation Pods are created unchanged

This causes silent failure, which is the worst kind.

🧠 What each plugin does

NodeRestriction

Security plugin Restricts kubelets from modifying things they shouldn’t Already present (keep it)

MutatingAdmissionWebhook

THIS IS THE ONE YOU NEED

• Allows the API server to:
• Call external HTTPS webhooks
• Modify incoming objects (Pods, Deployments, etc.)
• Without this:
• mutate() is never invoked
• Namespace labels are never copied

ValidatingAdmissionWebhook

Not used by your current code, but:

• Required for validation webhooks
• Best practice to enable alongside mutating
• Many clusters assume both exist

🔁 What happens after editing this file

This file is a static Pod manifest.

The moment you save it:

1. kubelet detects the file change
2. kubelet restarts kube-apiserver automatically
3. API server comes back with webhook support enable

No manual restart needed.

🧪 How you KNOW it’s working

After enabling this:

• Your webhook Pod logs show /mutate requests
• Pod creation pauses briefly (webhook timeout applies)
• Namespace labels are read
• Pods get mutated (once)

Before enabling this:

• Zero webhook traffic
• Zero mutations
• No errors

⚠️ One critical warning

If your webhook is misconfigured (TLS, RBAC, DNS):

• Pod creation can:
• Hang
• Retry
• Create many ReplicaSet attempts
• This is expected behavior when admission fails

That’s why:

• TLS
• SAN
• RBAC
• Service must all be correct before enabling this plugin in production.

✅ Final Truth

This line is non-negotiable:

--enable-admission-plugins=NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook

Without it:

• Admission webhooks are dead code

With it:

• Your webhook finally becomes part of Kubernetes’ object lifecycle

This is the switch that makes everything you built actually matter.

Architecture Overview

Flow:

1. Pod creation request reaches API server
2. API server calls our webhook
3. Webhook:

• Reads namespace labels
• Adds missing labels to Pod metadata

1. API server persists the Pod
2. Controllers proceed normally

Scope control:

• Webhook only runs in namespaces labeled with ns-label-sync=enabled

import json
import base64
from flask import Flask, request, jsonify
from kubernetes import client, config
app = Flask(__name__)
config.load_incluster_config()
def esc(k):
    return k.replace("~", "~0").replace("/", "~1")
@app.route("/mutate", methods=["POST"])
def mutate():
    review = request.get_json(silent=True)
    if not review or "request" not in review:
        return allow()
    req = review["request"]
    uid = req.get("uid")
    namespace = req.get("namespace")
    kind = req.get("kind", {}).get("kind")
    pod = req.get("object", {})
    if kind != "Pod" or not namespace:
        return allow(uid)
    pod_labels = pod.get("metadata", {}).get("labels", {}) or {}
    v1 = client.CoreV1Api()
    ns_labels = v1.read_namespace(namespace).metadata.labels or {}
    patch = []
    for k, v in ns_labels.items():
        if k in pod_labels:
            continue
        patch.append({
            "op": "add",
            "path": "/metadata/labels/" + esc(k),
            "value": v
        })
    if not patch:
        return allow(uid)
    return jsonify({
        "apiVersion": "admission.k8s.io/v1",
        "kind": "AdmissionReview",
        "response": {
            "uid": uid,
            "allowed": True,
            "patchType": "JSONPatch",
            "patch": base64.b64encode(json.dumps(patch).encode()).decode()
        }
    })
def allow(uid=None):
    return jsonify({
        "apiVersion": "admission.k8s.io/v1",
        "kind": "AdmissionReview",
        "response": {
            "uid": uid,
            "allowed": True
        }
    })
if __name__ == "__main__":
    app.run(
        host="0.0.0.0",
        port=8443,
        ssl_context=("/tls/tls.crt", "/tls/tls.key")
    )

2️⃣ Containerizing the Webhook

Dockerfile

FROM python:3.11-slim
WORKDIR /app
RUN pip install flask kubernetes
COPY app.py /app/app.py
EXPOSE 8443
CMD ["python", "/app/app.py"]

3️⃣ BUILD NEW IMAGE : Recommended to give Tag

docker build -t manoharshetty507/webhook:v1 .

If using containerd:

ctr -n k8s.io images import <(docker save manoharshetty507/webhook:v2)

4️⃣ CREATE webhook-system NAMESPACE

kubectl create namespace webhook-system

5️⃣ TLS: Why SAN Certificates Matter (and Why Your Webhook Broke Without Them)

Admission webhooks sit on a critical security boundary in Kubernetes. Every request the API server sends to your webhook is made over mutual trust, and that trust is enforced strictly through TLS certificate validation.

Older Kubernetes versions were lenient. Newer ones are not.

❌ Why CN-only certificates no longer work

In the past, it was common to generate certificates like this:

CN = webhook-service.webhook-system.svc

And assume Kubernetes would accept it.

That assumption is now invalid.

Modern Kubernetes follows current TLS standards (RFC 6125), which clearly state:

The Common Name (CN) field must not be used for hostname verification. Subject Alternative Names (SANs) are mandatory.

As a result, when the API server connects to a webhook using a CN-only certificate, you see this error:

x509: certificate relies on legacy Common Name field, use SANs instead

This is not a warning. This is a hard failure — the webhook is rejected and Pod creation fails.

✅ How Kubernetes actually connects to your webhook

When you define this in your MutatingWebhookConfiguration:

clientConfig:
  service:
    name: webhook-service
    namespace: webhook-system
    path: /mutate
    port: 443

Kubernetes resolves and connects to the webhook using the Service DNS name, typically:

webhook-service.webhook-system.svc

During the TLS handshake, the API server verifies:

1. The certificate is signed by the CA in caBundle
2. The certificate contains this DNS name in SAN
3. The certificate is valid for serverAuth

If any one of these fails, the webhook call is rejected.

✅ Correct approach: SAN-based certificates

To satisfy Kubernetes, the certificate must explicitly list all Service DNS variants in the SAN section.

That’s why we generate a certificate with:

• DNS.1 = webhook-service
• DNS.2 = webhook-service.webhook-system
• DNS.3 = webhook-service.webhook-system.svc

This covers:

• Short service name (inside namespace)
• Fully qualified service name
• Exact name used by the API server

SAN configuration file (san.cnf)

cat > san.cnf <<EOF
[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no

[req_distinguished_name]
CN = webhook-service.webhook-system.svc

[v3_req]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1 = webhook-service
DNS.2 = webhook-service.webhook-system
DNS.3 = webhook-service.webhook-system.svc
EOF

Important detail: The CN is still present, but Kubernetes ignores it. Only the SAN entries matter.

Certificate generation

openssl genrsa -out tls.key 2048
openssl req -new -key tls.key -out tls.csr -config san.cnf
openssl x509 -req -in tls.csr -signkey tls.key \
  -out tls.crt -days 365 \
  -extensions v3_req -extfile san.cnf

At this point:

• tls.crt contains SANs
• tls.key is the private key
• The certificate is suitable for a webhook server

Storing TLS material securely

Kubernetes expects webhook servers to load TLS material from a Secret.

kubectl create secret tls webhook-tls \
  --cert=tls.crt \
  --key=tls.key \
  -n webhook-system

This Secret is then:

• Mounted into the webhook Pod
• Used by Flask (or any webhook server) for HTTPS
• Trusted by the API server via caBundle

Why this matters in production

Without SAN certificates:

• Pods fail to create
• Deployments loop endlessly
• ReplicaSets explode
• The cluster appears “unstable”

With SAN certificates:

• Webhook communication is deterministic
• TLS validation is clean
• Admission control works reliably
• Your cluster behaves normally

Is the CN required?

No. Kubernetes does not use the Common Name (CN) at all for TLS verification.

Kubernetes (and Go’s TLS stack, which Kubernetes uses) completely ignores CN for hostname verification. Only Subject Alternative Names (SANs) are checked.

So this works:

• Certificate has correct SANs → ✅ webhook works
• Certificate has only CN → ❌ webhook fails
• Certificate has CN + SANs → ✅ webhook works (CN ignored)

Then why is the CN still present?

Because of OpenSSL, not Kubernetes.

When you run:

openssl req -new -key tls.key -out tls.csr

OpenSSL expects a Distinguished Name (DN). The DN historically contains fields like:

• CN (Common Name)
• O (Organization)
• C (Country)

If you don’t supply one, OpenSSL

• Prompts interactively, or
• Errors out (depending on flags)

So we include this block:

[req_distinguished_name]
CN = webhook-service.webhook-system.svc

This:

• Satisfies OpenSSL’s requirement
• Makes the certificate look “complete”
• Has zero effect on Kubernetes behavior

How is the CN generated?

It’s generated explicitly from this line in san.cnf:

[req_distinguished_name]
CN = webhook-service.webhook-system.svc

When you run:

openssl req -new -key tls.key -out tls.csr -config san.cnf

OpenSSL

1. Reads CN from req_distinguished_name
2. Embeds it in the certificate’s Subject field
3. Also embeds SANs from subjectAltName

That’s it. No magic, no Kubernetes involvement.

What Kubernetes actually validates (important)

When the API server connects to:

https://webhook-service.webhook-system.svc:443

It checks only:

• Does the certificate SAN contain webhook-service.webhook-system.svc? ✅
• Is the cert signed by the CA in caBundle? ✅
• Is extendedKeyUsage = serverAuth? ✅

It does not check:

• CN
• Organization
• Country
• Anything else in Subject

Can we remove CN completely?

In theory: yes. In practice: pointless pain.

Removing CN requires:

• Custom OpenSSL flags
• Non-standard configs
• Zero benefit

So the industry-standard approach is:

Keep CN for OpenSSL compatibility. Rely on SAN for real security.

One-line takeaway

CN exists only to keep OpenSSL happy. SAN exists to keep Kubernetes working.

6️⃣ SERVICE ACCOUNT + RBAC (MANDATORY)

Why RBAC Is Required

The webhook reads namespace labels, so it must have permission to:

GET namespaces

We create:

• ServiceAccount
• ClusterRole
• ClusterRoleBinding

This is mandatory. Without it, the webhook fails with 403 Forbidden.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: webhook-sa
  namespace: webhook-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: webhook-ns-reader
rules:
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: webhook-ns-reader-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: webhook-ns-reader
subjects:
- kind: ServiceAccount
  name: webhook-sa
  namespace: webhook-system

kubectl apply -f rbac.yaml

Why RBAC Is Required

This mutating webhook reads labels from the Namespace of the Pod being created.

Inside the webhook code, this line is critical:

v1.read_namespace(namespace)

That call:

• Talks to the Kubernetes API server
• Uses in-cluster authentication
• Runs as the ServiceAccount attached to the webhook Pod

By default, Pods run as:

system:serviceaccount:<namespace>:default

The default ServiceAccount does NOT have permission to read namespaces.

Result without RBAC:

403 Forbidden
User "system:serviceaccount:webhook-system:default"
cannot get resource "namespaces"

When this happens:

• The webhook crashes or returns HTTP 500
• The API server retries
• Deployments create many pending Pods
• The cluster appears “broken”

So RBAC is not optional. It is mandatory.

What Permissions Are Needed

The webhook only needs read access to namespaces.

Specifically:

• API group: core ("")
• Resource: namespaces
• Verb: get

Nothing more.

What We Create

We create three objects, each with a specific purpose.

1️⃣ ServiceAccount

apiVersion: v1
kind: ServiceAccount
metadata:
  name: webhook-sa
  namespace: webhook-system

Purpose

• Identity for the webhook Pod
• Replaces the default ServiceAccount
• Used when calling the Kubernetes API from inside the Pod

The webhook Deployment must reference this ServiceAccount:

serviceAccountName: webhook-sa

2️⃣ ClusterRole

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: webhook-ns-reader
rules:
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get"]

Purpose

• Defines what actions are allowed
• ClusterRole is required because:
• namespaces are cluster-scoped
• They do not belong to a single namespace

This role grants:

• Read-only access
• To namespaces
• Nothing else

3️⃣ ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: webhook-ns-reader-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: webhook-ns-reader
subjects:
- kind: ServiceAccount
  name: webhook-sa
  namespace: webhook-system

Purpose

• Connects the ServiceAccount to the ClusterRole
• Without this binding:
• The role exists
• The ServiceAccount exists
• But permissions are not applied

This binding says:

“Allow webhook-sa to use webhook-ns-reader permissions.”

Apply RBAC

kubectl apply -f rbac.yaml

Once applied:

• The webhook can successfully call read_namespace
• No more 403 errors
• No more endless Pod creation
• Mutations work exactly once per Pod creation

Final Reality Check

If any one of these is missing:

• ❌ ServiceAccount
• ❌ ClusterRole
• ❌ ClusterRoleBinding

Then:

• Webhook fails
• API server retries
• Deployments explode into many Pending Pods

This RBAC setup is mandatory, minimal, and correct.

This is not a workaround. This is how Kubernetes admission webhooks are supposed to run.

vi webhook-deployment.yaml

7️⃣ DEPLOYMENT + SERVICE

This section is what actually runs your webhook and makes it reachable by the Kubernetes API server.

An admission webhook is not magic. It is just an HTTPS server running in a Pod.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ns-label-webhook
  namespace: webhook-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: webhook
  template:
    metadata:
      labels:
        app: webhook
    spec:
      serviceAccountName: webhook-sa
      containers:
      - name: webhook
        image: manoharshetty507/webhook:v1
        ports:
        - containerPort: 8443
        volumeMounts:
        - name: tls
          mountPath: /tls
          readOnly: true
      volumes:
      - name: tls
        secret:
          secretName: webhook-tls
---
apiVersion: v1
kind: Service
metadata:
  name: webhook-service
  namespace: webhook-system
spec:
  ports:
  - port: 443
    targetPort: 8443
  selector:
    app: webhook

🔹 Explaining each component:

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ns-label-webhook
  namespace: webhook-system

What this does

• Creates a managed Pod
• Kubernetes keeps it running
• Restarts it if it crashes

A Deployment is required because:

• Webhooks must be always available
• A bare Pod would die and not restart

spec:
  replicas: 1

Why replicas = 1

• Admission webhooks must be deterministic
• Multiple replicas can cause:
• Debug confusion
• Multiple webhook hits during rollout
• One replica is the correct default

selector:
    matchLabels:
      app: webhook

Why this matters

• Tells the Deployment which Pods it owns
• Must match the Pod template labels exactly
• If this mismatches → Deployment never becomes Ready

template:
    metadata:
      labels:
        app: webhook

Why this label exists

• Used by:
• Deployment selector
• Service selector
• This label is the glue between Deployment and Service

spec:
      serviceAccountName: webhook-sa

Why this is critical

• This attaches the RBAC-enabled ServiceAccount
• Without this:
• Pod runs as default ServiceAccount
• Namespace reads fail with 403 Forbidden
• Webhook crashes
• Deployments create many Pods endlessly

This line activates the RBAC you created earlier.

containers:
      - name: webhook
        image: manoharshetty507/webhook:v1

What this does

• Runs your webhook image
• This image contains:
• Flask server
• /mutate endpoint
• Kubernetes client
• TLS enabled HTTPS server

Changing this tag = rolling out a new webhook version.

ports:
        - containerPort: 8443

Why 8443

• Webhook server listens on HTTPS
• Flask is started with:

port=8443

• This is internal only, not exposed externally

volumeMounts:
        - name: tls
          mountPath: /tls
          readOnly: true

Why this exists

• Mounts TLS certificates into the container
• Your Flask app uses:

ssl_context=("/tls/tls.crt", "/tls/tls.key")

Without this:

• HTTPS fails
• API server refuses connection
• Webhook never gets called

volumes:
      - name: tls
        secret:
          secretName: webhook-tls

What this does

• Pulls certificates from Kubernetes Secret
• Makes them available as files inside the Pod

This Secret must contain:

• tls.crt
• tls.key

🔹 Service

apiVersion: v1
kind: Service
metadata:
  name: webhook-service
  namespace: webhook-system

Why a Service is required

• Kubernetes API server cannot talk directly to Pods
• Webhooks must be addressed via a Service DNS name
• The MutatingWebhookConfiguration references this Service

spec:
  ports:
  - port: 443
    targetPort: 8443

What this means

• API server connects on port 443
• Traffic is forwarded to container port 8443

Why this matters:

• Admission webhooks must be HTTPS
• Port 443 is the expected standard
• Internal mapping keeps your container clean

selector:
    app: webhook

Why this selector matters

• Connects Service → webhook Pod
• Must match Pod label exactly:

app: webhook

If this is wrong:

• Service has no endpoints
• Webhook calls fail with connection errors

🔥 End-to-End Flow (What Actually Happens)

1. Pod is created in a namespace
2. API server sees a matching MutatingWebhookConfiguration
3. API server calls:

https://webhook-service.webhook-system.svc/mutate

1. Service routes traffic to webhook Pod
2. TLS cert is validated (SAN required)
3. Flask webhook runs
4. Namespace labels are read
5. Pod is mutated once
6. Pod is created successfully

✅ Why This Works

• Deployment keeps webhook alive
• Service exposes it safely
• TLS secures communication
• ServiceAccount + RBAC authorize API access
• Single replica avoids chaos

This section is not optional boilerplate. Every line exists because Kubernetes demands it.

This is the correct, production-safe way to run an admission webhook.

Apply the webhook-deployment file

kubectl apply -f webhook-deployment.yaml

Check pod status

[root@master01 mutating-webhook]# k get pods -n webhook-system
NAME                                READY   STATUS    RESTARTS   AGE
ns-label-webhook-7bbb6dc87b-5rtsh   1/1     Running   0          23m

8️⃣ MUTATING WEBHOOK CONFIGURATION

MutatingWebhookConfiguration Key points:

• failurePolicy: Ignore → prevents cluster lockups
• sideEffects: None
• reinvocationPolicy: Never
• namespaceSelector limits scope

This keeps the webhook safe for production.

8. MutatingWebhookConfiguration (this is where most people fail)

A webhook pod without a MutatingWebhookConfiguration is just a lonely Flask app doing yoga.

Get CA bundle:

base64 -w0 tls.crt

<PASTE_CA_BUNDLE_HERE> with above generated CA cert

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: ns-label-webhook
webhooks:
- name: ns-label-webhook.webhook-system.svc
  admissionReviewVersions: ["v1"]
  sideEffects: None
  failurePolicy: Ignore
  timeoutSeconds: 5
  clientConfig:
    service:
      name: webhook-service
      namespace: webhook-system
      path: /mutate
      port: 443
    caBundle: <PASTE_CA_BUNDLE_HERE>
  rules:
  - operations: ["CREATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
  namespaceSelector:
    matchExpressions:
    - key: ns-label-sync
      operator: In
      values: ["enabled"]

Apply the webhook-configuration

kubectl apply -f webhook-configuration.yaml
kubectl get mutatingwebhookconfiguration

9️⃣ TEST (ONLY VALID TEST)

Enabling the Feature (Opt-In)

Namespaces must be explicitly labeled:

kubectl label namespace test ns-label-sync=enabled env=from-ns
namespace/test created
namespace/test labeled

This prevents accidental mutation across the cluster.

Create a deployment

kubectl create deployment nginx --image=nginx -n test

Check the labels

[root@master01 mutating-webhook]# kubectl get pods -n test --show-labels
NAME                     READY   STATUS    RESTARTS   AGE   LABELS
nginx-7854ff8877-jhpw6   1/1     Running   0          3s    app=nginx,env=from-ns,kubernetes.io/metadata.name=test,ns-label-sync=enabled,pod-template-hash=7854ff8877

✅ EXPECTED

env=from-ns
ns-label-sync=enabled

Why This Solution Is Safe in Production

• ✅ No controller loops
• ✅ No ReplicaSet churn
• ✅ No pod storms
• ✅ Explicit namespace opt-in
• ✅ Idempotent logic
• ✅ Admission best practices followed

This is the same pattern used by mature platform teams running multi-tenant Kubernetes clusters.

Conclusion

Namespace labels are foundational, but Kubernetes doesn’t propagate them by default. Using a carefully designed Mutating Admission Webhook, we can enforce label consistency without breaking the cluster.

The key lesson is not the webhook itself — it’s discipline:

• Mutate only what you must
• Never mutate repeatedly
• Respect controller behavior

Once done correctly, this becomes an invisible, reliable part of your platform — exactly how good infrastructure should behave.