Validating Webhooks: Preventing Misconfigured Workloads with Kubernetes

Kubernetes gives teams enormous freedom. That freedom is also how clusters slowly drift into chaos.

In most organizations, the problems don’t start with outages. They start quietly: pods without owners, workloads deployed to production without environment tags, cost reports that don’t add up, security teams asking uncomfortable questions no one can answer confidently.

The common root cause is simple: there is no enforcement at the API boundary.

This blog walks through a real production scenario where a Validating Admission Webhook is used to stop bad workloads before they ever enter the cluster. No controllers. No cron jobs. No retroactive cleanup. Just real-time policy enforcement at the Kubernetes API server.

Github Repo

[embed]GitHub - Manohar-1305/validating-webhook Contribute to Manohar-1305/validating-webhook development by creating an account on GitHub.github.com

The real-world problem

In many teams:

• Developers deploy pods directly.
• CI systems deploy manifests dynamically.
• Labels like team and environment are “recommended”, not enforced.

What happens next is predictable:

• Pods exist with no ownership.
• Cost allocation breaks.
• Security and compliance checks become manual.
• Production incidents take longer because no one knows who owns what.

The key insight is this:

Once a bad resource is created, you’re already late.

The only place where enforcement is guaranteed is admission time — when the API server decides whether a request is allowed.

Implemenatation Mutating Webhook

[embed]

What this webhook does (and what it does not)

This webhook:

• Intercepts pod creation requests
• Checks for mandatory labels (team, environment)
• Rejects the request if they’re missing

It does not:

• Modify objects
• Call the Kubernetes API
• Maintain state
• Act like a controller

This is pure validation. No mutation. No side effects. Exactly how validating webhooks are meant to be used.

Project layout

A minimal, production-friendly structure:

validating-webhook/
├── app.py
├── Dockerfile
├── requirements.txt
├── san.cnf
├── manifests/
│   ├── deployment.yaml
│   └── validating-webhook.yaml

Each file exists for a reason. Nothing here is accidental.

2. Flask application (validating logic)

app.py

from flask import Flask, request, jsonify

app = Flask(__name__)

REQUIRED_LABELS = ["team", "environment"]

def is_privileged(pod_spec):
    containers = pod_spec.get("containers", [])
    for c in containers:
        sec = c.get("securityContext", {})
        if sec.get("privileged", False) is True:
            return True
    return False

@app.route("/validate", methods=["POST"])
def validate():
    review = request.get_json()
    req = review["request"]

    uid = req["uid"]
    obj = req["object"]

    metadata = obj.get("metadata", {})
    spec = obj.get("spec", {})


    labels = metadata.get("labels", {})
    missing = [l for l in REQUIRED_LABELS if l not in labels]
    if missing:
        return jsonify({
            "apiVersion": "admission.k8s.io/v1",
            "kind": "AdmissionReview",
            "response": {
                "uid": uid,
                "allowed": False,
                "status": {
                    "message": f"Missing required labels: {missing}"
                }
            }
        })


    if is_privileged(spec):
        return jsonify({
            "apiVersion": "admission.k8s.io/v1",
            "kind": "AdmissionReview",
            "response": {
                "uid": uid,
                "allowed": False,
                "status": {
                    "message": "Privileged containers are not allowed"
                }
            }
        })

    return jsonify({
        "apiVersion": "admission.k8s.io/v1",
        "kind": "AdmissionReview",
        "response": {
            "uid": uid,
            "allowed": True
        }
    })

if __name__ == "__main__":
    app.run(
        host="0.0.0.0",
        port=8443,
        ssl_context=("/certs/tls.crt", "/certs/tls.key")
    )

This is pure validation. No mutation. No side effects. Exactly what a validating webhook should do.

3. Python dependencies

requirements.txt

flask==3.0.0

Nothing else. Keep webhooks lean.

4. Docker image (production-acceptable)

Dockerfile

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .
EXPOSE 8443
CMD ["python", "app.py"]

5. TLS with SAN (this is mandatory)

san.cnf

cat > san.cnf <<EOF
[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no

[req_distinguished_name]
CN = validating-webhook.webhook.svc

[v3_req]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1 = validating-webhook
DNS.2 = validating-webhook.webhook
DNS.3 = validating-webhook.webhook.svc
DNS.4 = validating-webhook.webhook.svc.cluster.local
EOF

Certificates are generated once and stored as a Kubernetes TLS secret.

Generate certs

openssl genrsa -out ca.key 2048
openssl req -x509 -new -key ca.key \
  -out ca.crt -days 365 \
  -subj "/CN=validating-webhook-ca"

openssl genrsa -out tls.key 2048
openssl req -new -key tls.key -out tls.csr -config san.cnf

openssl x509 -req -in tls.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out tls.crt -days 365 \
  -extensions v3_req -extfile san.cnf

• ca.crt → goes into **caBundle**
• tls.crt + tls.key → go into Kubernetes TLS secret

6. TLS Secret using the certificates

Once the TLS certificate and key are generated, they must be stored securely inside the cluster. Kubernetes expects the webhook server to present these certificates during the TLS handshake, so we create a TLS secret in the same namespace where the webhook runs.

kubectl create secret tls validating-webhook-tls \
  --cert=tls.crt \
  --key=tls.key \
  -n webhook

This secret is later mounted into the webhook pod as a read-only volume and used by Flask to serve HTTPS traffic. Without this step, the API server will refuse to connect to the webhook, and admission requests will fail.

7. Build the image

Option A: Docker (local)

docker build -t manoharshetty507/webhook-validating:v1 .

Check docker images

[root@master01 validating-webhook]# docker images
                                                                                                                     i Info →   U  In Use
IMAGE                                    ID             DISK USAGE   CONTENT SIZE   EXTRA
manoharshetty507/validating-webhook:v1   9f6bc5f7c915        206MB         51.1MB

Option B: Import into containerd

ctr -n k8s.io images import <(docker save manoharshetty507/webhook-validating:v1)

Verify on node:

ctr -n k8s.io images ls | grep validating-webhook

Option C: Push to registry

docker tag vmanoharshetty507/webhook:v1 manoharshetty507/webhook:v1
docker push manoharshetty507/webhook:v1

This works because containerd and Docker share the same image format. The API server doesn’t care how the image arrived — only that it exists.

8. Kubernetes manifests

Kubernetes deployment

The webhook runs like any other workload.

• Dedicated namespace
• Two replicas (admission paths must be HA)
• TLS mounted read-only
• Service exposed internally

Once the ValidatingWebhookConfiguration is applied, the API server starts calling the webhook synchronously for every matching request.

manifests/namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: webhook

Apply it:

kubectl apply -f manifests/namespace.yaml

9. Deployment

manifests/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: validating-webhook
  namespace: webhook
spec:
  replicas: 2
  selector:
    matchLabels:
      app: validating-webhook
  template:
    metadata:
      labels:
        app: validating-webhook
    spec:
      containers:
      - name: webhook
        image: validating-webhook:v1
        ports:
        - containerPort: 8443
        volumeMounts:
        - name: tls
          mountPath: /certs
          readOnly: true
      volumes:
      - name: tls
        secret:
          secretName: validating-webhook-tls
---
apiVersion: v1
kind: Service
metadata:
  name: validating-webhook
  namespace: webhook
spec:
  selector:
    app: validating-webhook
  ports:
  - port: 443
    targetPort: 8443

What actually happens at runtime

1. A user submits a Pod manifest.
2. The API server pauses admission.
3. The webhook is called over HTTPS.
4. The webhook evaluates the object.
5. The API server allows or rejects the request

If the webhook is unavailable and failurePolicy: Fail is set, the request is denied. This is enforcement, not best effort.

Check webhook pod status

[root@master01 validating-webhook]# k get pods -n webhook
NAME                                  READY   STATUS    RESTARTS   AGE
validating-webhook-78cd596567-9n46s   1/1     Running   0          106s
validating-webhook-78cd596567-wxlnd   1/1     Running   0          104s

Check logs of the webhook pod

k logs -f validating-webhook-78cd596567-9n46s -n webhook

10. ValidatingWebhookConfiguration (v1)

First, encode CA bundle:

base64 -w0 tls.crt

manifests/validating-webhook.yaml

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: validate-required-labels
webhooks:
- name: labels.webhook.example.com
  admissionReviewVersions: ["v1"]
  sideEffects: None
  failurePolicy: Fail
  rules:
  - apiGroups: [""]
    apiVersions: ["v1"]
    operations: ["CREATE"]
    resources: ["pods"]
  - apiGroups: ["apps"]
    apiVersions: ["v1"]
    operations: ["CREATE"]
    resources: ["deployments"]
  clientConfig:
    service:
      name: validating-webhook
      namespace: webhook
      path: /validate
      port: 443
    caBundle: <BASE64_CA_CERT>

Apply everything:

kubectl apply -f manifests/

Check the status of validating webhooks

[root@master01 validating-webhook]# kubectl get validatingwebhookconfiguration
NAME                       WEBHOOKS   AGE
validate-required-labels   1          8h

12. Verify behavior (this matters)

❌ Rejected pod

apiVersion: v1
kind: Pod
metadata:
  name: bad-pod
spec:
  containers:
  - name: nginx
    image: nginx

Observable behavior

A pod without required labels is rejected immediately:

Error from server: Missing required labels: ['team', 'environment']

A properly labeled pod proceeds normally and schedules as expected.

Nothing else in the cluster changes. No cleanup jobs. No drift correction. Just prevention.

✅ Accepted pod

apiVersion: v1
kind: Pod
metadata:
  name: good-pod
  labels:
    team: platform
    environment: prod
spec:
  containers:
  - name: nginx
    image: nginx

Pod schedules normally.

Deployment: NO labels + privileged container (DENIED)

• ❌ Labels are NOT present
• ❌ Privileged container IS enabled
• Result: Admission must be denied (label check will hit first)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deny-no-labels-privileged
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        securityContext:
          privileged: true

Expected result

Error from server: admission webhook "labels.webhook.example.com" denied the request:
Missing required labels: ['team', 'environment']

Control test (both rules satisfied)

Manifest: labels present, privileged = false

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-valid
  labels:
    team: platform
    environment: prod
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
        team: platform
        environment: prod
    spec:
      containers:
      - name: nginx
        image: nginx

Expected result

[root@master01 validating-webhook]# k get pods
NAME                          READY   STATUS    RESTARTS   AGE
test-valid-664cd6c54c-dvq9v   1/1     Running   0          5m29s

13. Production rules you must follow

• Keep webhook stateless
• Never call external services
• Use replicas ≥ 2
• failurePolicy: Fail for enforcement
• Timeouts kill clusters — keep it fast
• Validating webhooks block the API server

This is the same pattern used by:

• security teams
• platform engineering
• governance / compliance layers

Next escalation paths are obvious:

• validate Deployments / StatefulSets
• namespace-aware policies
• audit-only → enforce rollout
• chaining mutating + validating webhooks

Conclusion

Validating Admission Webhooks are not an advanced feature. They are a missing safety rail in many clusters.

If your organization relies on:

• conventions
• documentation
• reviews
• “please remember to add labels”

then you are trusting humans where Kubernetes already offers enforcement.

This webhook is small by design, but its impact is large: it turns tribal knowledge into guaranteed behavior.

Once you understand this pattern, the next steps are obvious:

• extend validation to Deployments and StatefulSets
• enforce namespace-specific rules
• introduce audit-only modes
• chain mutating and validating webhooks deliberately

At that point, you’re no longer “using Kubernetes”.

You’re operating a platform.