Traefik on Kubernetes: Building a Real Zero-Trust Ingress Layer

Lessons learned integrating JWT authentication, hybrid networking, and enterprise APIs

In the previous article, we talked about the broader modernization journey of a large enterprise integration platform.

This time, let’s zoom into one specific part of the architecture that turned out to be far more interesting than expected:

the ingress layer.

More specifically:

• Traefik
• Kubernetes
• JWT validation
• OAuth2 Proxy
• internal vs external routing
• hybrid networking
• and why DNS eventually ruins everybody’s day

This is not a beginner tutorial.

There are already thousands of articles showing how to deploy Traefik with a whoami container.

This is about what happens when:

• traffic is business critical
• security requirements are real
• legacy systems still exist
• and your cluster needs to communicate with both cloud-native services and systems that predate Kubernetes by a decade

Why Traefik Ended Up Being the Right Choice

Initially, Traefik was not the obvious winner.

Like most teams evaluating ingress architectures, we considered:

• NGINX Ingress
• Kong
• Istio
• API Gateway solutions
• and briefly, the terrible idea of writing custom middleware

What made Traefik particularly attractive was its balance between:

• simplicity
• flexibility
• Kubernetes-native integration
• middleware-driven architecture

We specifically wanted:

• reusable authentication layers
• clean separation between routing and auth logic
• minimal custom code
• dynamic configuration
• lower operational overhead than a full service mesh

Traefik hit the sweet spot.

Especially once we started using CRDs instead of standard Kubernetes Ingress objects.

Why Standard Ingress Became Limiting Very Quickly

Standard Kubernetes Ingress resources are fine.

Until they aren’t.

For simple routing:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: simple-api
spec:
  rules:
    - host: api.company.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: api-service
                port:
                  number: 8080

Perfectly acceptable.

But the moment requirements become enterprise-shaped, things get messy quickly.

We needed:

• reusable middleware
• centralized authentication
• different policies per route
• traffic manipulation
• internal/external route separation
• authentication chains
• cleaner configuration ownership

Trying to scale that cleanly with standard Ingress resources becomes painful.

That’s where Traefik CRDs become significantly more powerful.

Why We Switched to IngressRoute

IngressRoute gave us something extremely important:

modularity.

Instead of mixing everything together inside giant Ingress manifests, we could separate:

• routing
• middleware
• TLS
• authentication
• services

Example:

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: external-api
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`api.company.com`)
      kind: Rule
      middlewares:
        - name: jwt-auth
      services:
        - name: api-service
          port: 8080

The important part here is not the YAML itself.

The important part is the architecture.

Authentication becomes reusable infrastructure.

Applications stop caring about JWT validation.

Which massively simplifies backend services.

The JWT Validation Problem

At first, we considered validating JWT tokens directly inside applications.

This is usually how distributed authentication chaos begins.

Because suddenly:

• every service validates tokens differently
• claim validation becomes inconsistent
• JWKS caching behaves differently everywhere
• security updates become painful
• debugging becomes miserable

We wanted authentication centralized.

Not duplicated across twenty services.

Initially, we also evaluated building a dedicated custom auth service.

Technically possible.

Operationally dangerous.

Authentication middleware sounds fun until you become responsible for maintaining it forever.

Instead, we decided to leverage:

• Traefik ForwardAuth
• OAuth2 Proxy
• existing OIDC infrastructure

That combination dramatically reduced complexity.

And reduced the number of places capable of breaking authentication.

ForwardAuth Was the Real Game Changer

Traefik’s ForwardAuth middleware turned out to be one of the cleanest solutions in the entire architecture.

The idea is simple:

1. request reaches Traefik
2. Traefik forwards auth validation to an external service
3. external service validates JWT
4. request proceeds only if authentication succeeds

A simplified middleware looked like this:

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: jwt-auth
spec:
  forwardAuth:
    address: http://oauth2-proxy.auth.svc.cluster.local/auth
    trustForwardHeader: true
    authResponseHeaders:
      - X-Forwarded-User
      - X-Forwarded-Email
      - Authorization

Which meant backend applications received already-authenticated traffic.

Exactly what we wanted.

No JWT parsing inside business services. No duplicated validation logic. No custom middleware everywhere.

Why OAuth2 Proxy Was Surprisingly Useful

OAuth2 Proxy solved several problems at once.

It handled:

• OIDC integration
• token validation
• session handling
• provider communication
• identity propagation

Without requiring us to write custom auth code.

And honestly?

Every line of custom authentication code you avoid writing is usually a long-term win.

Especially in enterprise environments where security requirements evolve constantly.

Internal APIs vs External APIs

One of the most important decisions was separating ingress traffic into two categories.

External APIs

These APIs were:

• internet-facing
• JWT protected
• routed through external load balancers
• exposed via public DNS
• tightly monitored

Internal APIs

These APIs remained:

• private
• accessible only through VPN/internal routing
• behind internal load balancers
• exposed only via private DNS

This separation simplified:

• governance
• security reviews
• operational troubleshooting
• firewall policies
• audit requirements

And prevented one of the most dangerous anti-patterns in cloud infrastructure:

“We’ll expose it publicly just temporarily.”

Those words should trigger automatic incident response.

The DNS Problem Nobody Warns You About

Every hybrid Kubernetes project eventually becomes a DNS project.

This is unavoidable.

The cluster needed to resolve:

• Kubernetes services
• internal corporate domains
• private APIs
• on-premise middleware
• cloud-native services

Which introduced:

• forwarding zones
• split-horizon DNS
• conditional forwarding
• multiple authoritative zones

At one point we had:

• Kubernetes DNS
• cloud DNS
• enterprise DNS
• forwarding resolvers
• private zones

all participating in the same request flow.

This is effectively distributed systems comedy.

The final solution relied heavily on forwarding DNS requests toward authoritative on-premise DNS servers.

Which sounds simple.

Until one subnet resolves correctly and another subnet behaves like DNS has stopped existing entirely.

Hybrid Networking Was More Difficult Than Kubernetes

Ironically, Kubernetes itself was rarely the difficult part.

The difficult part was hybrid connectivity.

The cluster still needed access to:

• legacy middleware
• internal databases
• enterprise APIs
• monitoring systems
• private services

Which meant dealing with:

• VPN tunnels
• internal routing
• firewall policies
• asymmetric traffic
• DNS forwarding
• overlapping network assumptions

Cloud-native architecture diagrams rarely show this part.

Mostly because nobody wants to admit how much time gets spent debugging routes.

The Reverse Proxy Stack Became Slightly Ridiculous

At some point, requests looked like this:

Client
  -> External Load Balancer
      -> Traefik
          -> OAuth2 Proxy
              -> Internal Gateway
                  -> Legacy Middleware
                      -> Actual Service

Technically valid.

Psychologically aggressive.

Every additional proxy layer introduces:

• timeout problems
• header propagation issues
• TLS complications
• debugging complexity

We spent a surprising amount of time ensuring headers survived the entire request chain correctly:

• Authorization
• X-Forwarded-For
• X-Forwarded-Proto
• X-Real-IP

Because once identity propagation breaks somewhere in the middle, debugging becomes extremely unpleasant.

Observability Became Essential

Once you combine:

• Traefik
• Kubernetes
• OAuth
• VPN networking
• hybrid DNS
• internal routing

traditional troubleshooting stops working.

Observability becomes mandatory.

Traefik metrics and logs ended up being incredibly valuable.

Especially during authentication debugging.

There’s something uniquely frustrating about discovering:

“The token is valid, but the audience claim is wrong.”

three hours into a production investigation.

Metrics reduce that from a crisis into a quick fix.

What We’d Do Differently

Standardize Middleware Earlier

Once multiple teams create different authentication patterns, chaos arrives quickly.

Centralize auth middleware early.

Document DNS Thoroughly

Nobody documents DNS enough.

Until the one person who understands the forwarding rules disappears for vacation.

Avoid Custom Auth Logic

Seriously.

Use standards. Use existing tooling. Avoid inventing authentication systems.

Treat Hybrid Networking as a First-Class Concern

Because it absolutely is.

Final Thoughts

Traefik ended up being an excellent fit for this architecture.

Not because it magically solved every problem.

But because it allowed us to:

• centralize authentication
• simplify application logic
• separate routing concerns
• modernize incrementally
• integrate cleanly with Kubernetes
• avoid unnecessary custom code

And honestly, that’s what successful platform modernization usually looks like.

Not revolutionary rewrites.

Careful evolution.

With slightly more YAML than any human should reasonably consume.