Photo by Tim Foster on Unsplash

Headlamp on VKS: OIDC, Security Plugins, and Lessons Learned

Introduction

Headlamp is a CNCF Sandbox Kubernetes web UI. It runs in-cluster or as a desktop app, offers a plugin system to extend its functionality, and natively supports OIDC authentication. Unlike heavier dashboard options, Headlamp remains minimal at its core and lets you add features via plugins. I've been working with Headlamp a lot recently and am really impressed by some of its outstanding features.

This post explains how to deploy Headlamp on a vSphere VKS cluster using Google OIDC authentication, install security and observability plugins (Kubescape, Cert-manager, Prometheus), and discusses the networking issues encountered with WebSockets along the way.

All manifests mentioned and used in this post can be found in the companion repository: papivot/Headlamp-OIDC-plugins

Prerequisites and Cluster Configuration

The environment used for this setup:

• Cluster: VKS running Kubernetes v1.35 with a custom ClusterClass that patches the API server to mount an external AuthenticationConfiguration for OIDC
• Ingress Controller: Contour (with Envoy as the data plane), exposed via a vSphere LoadBalancer
• Certificate Management: Cert-manager
• Monitoring: Prometheus

The cluster-level OIDC configuration and ClusterClass patches are in the [vks-cluster-config/](https://github.com/papivot/Headlamp-OIDC-plugins/tree/main/vks-cluster-config) directory of the repo. The custom ClusterClass custom-v3.6.0-external-oidcuses jsonPatches to inject the AuthenticationConfiguration file into the control plane nodes and add the — authentication-config flag to the API server.

Installing Headlamp with Helm

Add the Helm repo and install:

helm repo add headlamp https://headlamp-k8s.github.io/headlamp/
helm repo update
helm install headlamp headlamp/headlamp \
 - namespace headlamp \
 - create-namespace \
 - version 0.40.0 \
-f headlamp-values.yaml

Pin the chart version Chart version 0.40.1 introduced a -session-ttlflag that the v0.40.0 binary does not recognize, causing a CrashLoopBackOff. Always pass — version 0.40.0 when upgrading too.

The full values file is at [headlamp-values.yaml](https://github.com/papivot/Headlamp-OIDC-plugins/blob/main/headlamp-values.yaml) Here is a breakdown of its key sections.

In-cluster mode and OIDC

config:
  inCluster: true
  inClusterContextName: "workload-vsphere-vks2"
  watchPlugins: true
  oidc:
    clientID: "566831913343-fj7vvdk43eithm6o750of9v84qg1b11b.apps.googleusercontent.com"
    clientSecret: "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"
    issuerURL: "https://accounts.google.com"
    scopes: "openid,email,profile"
    callbackURL: "https://headlamp.10.138.169.33.sslip.io/oidc-callback"
    usePKCE: true
  # The Prometheus plugin ships inside the Headlamp image at
  # /headlamp/static-plugins/ but isn't loaded because -plugins-dir only
  # points to /headlamp/plugins/.  Adding -user-plugins-dir makes the
  # server also serve plugins from the static directory.

inCluster: true tells Headlamp to use the in-cluster service account for discovery.

inClusterContextName controls the display name in the sidebar. Without it, the cluster shows up as “main”.

usePKCE enables Proof Key for Code Exchange, which binds the authorization code to the originating client session and prevents interception attacks.

callbackURL must be an FQDN. Google’s OAuth rejects bare IP addresses. We used sslip.io for DNS resolution.

Enabling the Prometheus plugin

extraArgs:
    - -user-plugins-dir=/headlamp/static-plugins

The Prometheus plugin ships inside the Headlamp container image at /headlamp/static-plugins/prometheus/, but the server only loads plugins from -plugins-dir (which points to /headlamp/plugins/). Adding -user-plugins-dir=/headlamp/static-plugins tells the server to also serve plugins from the static directory. This is necessary because the Prometheus plugin is not published on ArtifactHub and cannot be installed through the Plugin Manager. Worse, the Plugin Manager actively removes any plugin in /headlamp/plugins/ that is not in its config, so copying the files there does not survive a plugin sync cycle.

Plugin Manager

pluginsManager:
  enabled: true
  configContent: |
    plugins:
      - name: kubescape
        source: https://artifacthub.io/packages/headlamp/kubescape-headlamp-plugin/headlamp_kubescape
        version: 0.10.5
      - name: cert-manager
        source: https://artifacthub.io/packages/headlamp/headlamp-plugins/headlamp_cert-manager
        version: 0.1.0
    installOptions:
      parallel: true
      maxConcurrent: 2

The Plugin Manager runs as a sidecar container. It downloads and installs plugins from ArtifactHub at startup and watches for config changes. Plugins are version-pinned to prevent unexpected upgrades.

Resource limits

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 512Mi

OIDC Authentication Flow

Headlamp and the Kubernetes API server are two independent OIDC consumers. Headlamp handles token acquisition as an OAuth Relying Party, while the API server manages token validation as the Resource Server. Both must trust the same OIDC provider and client ID, but they serve complementary roles. The entire authentication and authorization flow is discussed in detail in the [README](https://github.com/papivot/Headlamp-OIDC-plugins/blob/main/README.md).md.

VKS Cluster API server OIDC configuration

The K8s API server must be configured to trust Google as a token issuer. This is done through an AuthenticationConfiguration resource mounted into the control plane (see [vks-cluster-config/cluster-oidc-secret.yaml](https://github.com/papivot/Headlamp-OIDC-plugins/blob/main/vks-cluster-config/cluster-oidc-secret.yaml))

Without this configuration, the API server rejects every OIDC token — it would have no way to know that Google is a trusted issuer, which client ID is valid, or how to extract a Kubernetes username from the JWT claims.

The OIDC-derived username (google:user@gmail.com) is then bound to a ClusterRole via a standard ClusterRoleBinding (see [vks-cluster-config/clusterrolebinding.yaml](https://github.com/papivot/Headlamp-OIDC-plugins/blob/main/vks-cluster-config/clusterrolebinding.yaml))

HTTPProxy Instead of Ingress

This was a frustrating issue that took some time to resolve! Headlamp uses WebSocket connections for terminal features such as kubectl exec, kubectl attach, and ephemeral debug containers. Understanding how traffic flows through the network stack was necessary to get WebSockets working.

What failed

Direct LoadBalancer (ServiceType: LoadBalancer): In our environment, the LoadBalancer performs L7 HTTP inspection on port 80. It apparently strips the Connection: Upgrade and Upgrade: websocket headers, breaking WebSocket handshakes. Every attempt to exec into a pod or attach a debug container failed silently.

Standard Kubernetes Ingress: Switching to an Ingress resource through Contour did not help. Contour’s Ingress controller does not enable WebSocket support by default on standard Ingress resources. There is no annotation to toggle it.

What worked

Contour’s HTTPProxy CRD has an explicit enableWebsockets field per route. Combined with TLS, this resolved the issue completely.

apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: headlamp
  namespace: headlamp
spec:
  virtualhost:
    fqdn: headlamp.10.138.169.33.sslip.io
    tls:
      secretName: headlamp-tls
  routes:
    - conditions:
        - prefix: /
      services:
        - name: headlamp
          port: 80
      enableWebsockets: true

The TLS certificate is generated by cert-manager using a self-signed issuer.

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: headlamp-selfsigned
  namespace: headlamp
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: headlamp-tls
  namespace: headlamp
spec:
  secretName: headlamp-tls
  dnsNames:
    - headlamp.10.138.169.33.sslip.io
  issuerRef:
    name: headlamp-selfsigned
    kind: Issuer

Why TLS is required

TLS is not just about encryption here. The current LoadBalancer behaves differently on port 443 vs port 80:

• Port 80: L7 HTTP inspection — the LB parses HTTP headers and strips the Upgrade header
• Port 443: L4 TCP passthrough — the LB forwards raw TCP, preserving all headers including Upgrade

Without TLS, even HTTPProxy with enableWebsockets: true failed because the Upgrade header was stripped before reaching Envoy. With TLS on port 443, the LB performs TCP passthrough, the Upgrade header reaches Envoy intact, and Envoy (configured by HTTPProxy) properly handles the WebSocket upgrade.

Plugins: Kubescape

Kubescape is a Kubernetes security platform that provides vulnerability scanning, compliance checking, runtime threat detection, and more. It deploys as an operator on the cluster. The Headlamp Kubescape plugin (v0.10.5) surfaces all of this data directly in the Headlamp UI.

Operator installation

helm repo add kubescape https://kubescape.github.io/helm-charts/
helm repo update
helm install kubescape kubescape/kubescape-operator \
 - namespace kubescape \
 - create-namespace \
-f kubescape-operator-values.yaml

The values file ([kubescape-operator-values.yaml](https://github.com/papivot/Headlamp-OIDC-plugins/blob/main/kubescape-operator-values.yaml) ) provides a comprehensive set of security capabilities, explained in detail in the inline comments in the file. Please refer to the values file for additional information.

What the Headlamp plugin surfaces

The Kubescape plugin adds several views to the Headlamp UI:

• Vulnerability scans with per-image CVE details, severity ratings, and fix versions
• Compliance frameworks, including NSA-CISA, MITRE ATTACK, and CIS benchmarks
• Runtime detection alerts for anomalous behavior caught by the eBPF-based node-agent
• Network policy suggestions generated from observed traffic patterns (my favorite!)
• Seccomp profiles are auto-generated per workload

Known issue

The Policy Playground in the Kubescape plugin crashes with a TypeError: Cannot read properties of undefined (reading ‘name’) when no ValidatingAdmissionPolicy resources exist in the cluster. This may be an existing bug.

Plugins: cert-manager and Prometheus

cert-manager plugin

The cert-manager plugin (v0.1.0) is installed via the Plugin Manager from ArtifactHub.

It adds views for:

• Certificates and their current status (Ready, Renewing, Failed)
• Issuers and ClusterIssuers
• Certificate renewal timelines
• Certificate request history

This is useful for quick visibility into TLS certificate health across the cluster without switching to kubectl or a separate tool.

Prometheus plugin

The Prometheus plugin is a different story. It adds context to the pods and provides fine-grained metrics on pod usage.

Configuring the Prometheus endpoint

Once the plugin is loaded using the Helm chart (see above), configure it in Headlamp Settings > Prometheus. The expected format is:

namespace/service-name:port

For most VKS clusters, it will be something similar to tanzu-system-monitoring/prometheus-server:80

Customizations and Tips

Cluster display name: Set config.inClusterContextName in the Helm values. Without it, the cluster appears as main in the sidebar.

Node Shell and Pod Debug images: Configurable in the Headlamp Settings UI. This is useful in air-gapped environments where the default images may not be reachable. For example, you can set both to mirror.gcr.io/library/busybox:latest.

Plugin Manager: Supports declarative plugin installation with version pinning and parallel downloads. Plugins are specified by ArtifactHub URL and version in the pluginsManager.configContent block of the Helm values.

Chart version pinning: Always pass the version with helm upgrade. Helm repos can publish new patch versions at any time, and even a minor change to a chart (such as adding an unsupported server flag) can cause pod crashes.

Conclusion

Headlamp works well as a lightweight, OIDC-aware Kubernetes UI. Its plugin system lets you add security scanning (Kubescape), certificate management, and Prometheus metrics without deploying separate dashboards. The OIDC integration is straightforward once you understand the dual-consumer model.