Cracking the Grafana MCP Authentication Dilemma: Implementing Zero-Trust OAuth for Self-Hosted Community Deployments

Photo by Shane on Unsplash

Introduction

As Anthropic’s Model Context Protocol (MCP) sweeps through the AI development ecosystem, enabling local AI assistants (like Claude Code, Cursor, and other MCP-compatible agents) to directly query monitoring and logging data has become a standard requirement for Platform Engineering teams focused on Developer Experience (DevEx).

However, if you attempt to roll out MCP within an enterprise running a self-hosted, open-source (Community Version) Grafana deployment, you will immediately run into a major compliance roadblock.

In the official grafana/mcp-grafana upstream repository, GitHub Issue #284 highlights a painful reality for the community: "Support OAuth / SSO authentication for self-hosted Grafana MCP servers."

Currently, the upstream open-source MCP server only supports static Service Account Tokens or Basic Authentication. In a modern enterprise environment, Basic Auth is heavily deprecated, and storing long-lived, plaintext tokens inside engineers’ local ~/.claude/.mcp.json files violates fundamental Zero-Trust security principles. As project maintainers note, while Grafana Cloud provides managed OAuth solutions, the self-hosted community variant lacks this native capability due to core architectural limitations.

To bridge this gap for the global open-source community, our Platform Engineering team designed a production-ready proxy and network encapsulation layer. This post details how we solved Issue #284, securing our self-hosted infrastructure around two core pillars: Human-to-Machine OAuth Integration and a Machine-to-Machine eBPF Network Defense.

Pillar 1: Solving Issue #284 — The Human-to-Machine OAuth Proxy Architecture

Since the upstream MCP server cannot natively negotiate authentication flows with an enterprise Identity Provider (IdP like Okta, Google Auth, or Azure AD), our architectural solution was to deploy a lightweight, multi-tenant security proxy layer between the local AI tools and the upstream MCP component.

This proxy is deployed inside our Kubernetes cluster, restricted strictly to our corporate VPN.

The Request Lifecycle

When a human developer executes a query via their local AI assistant, the traffic splits across secure boundaries as follows:

[ Engineer's Laptop (On Corporate VPN) ]
          |
          | (1) HTTPS POST /mcp + Short-lived Bearer JWT (via Browser SSO)
          v
[ Ingress Controller (VPN-Only / Private Class) ]
          |
          v
+-----------------------------------------------------------------------+
|  Kubernetes Pod (One dedicated siloed Pod per isolated Grafana Org)    |
|                                                                       |
|  +------------------------------------+                               |
|  |  1. Security Proxy Container       | <-- Only container exposed     |
|  |     (mcp-auth-proxy)               |                                |
|  |  - Triggers browser IdP SSO flow   |                                |
|  |  - Issues short-lived RS256 JWT    |                                |
|  |  - Strips Auth Header before proxy |                                |
|  +-----------------|------------------+                               |
|                    | (2) localhost:8000 only (Completely hidden from network)
|                    v                                                  |
|  +------------------------------------+                               |
|  |  2. Core MCP Container             |                                |
|  |     (Upstream mcp-grafana)         |                                |
|  |  - Dynamically injects Org-scoped  |                                |
|  |    Viewer token from Cloud Secrets |                                |
|  +-----------------|------------------+                               |
+--------------------|--------------------------------------------------+
                     | (3) Scoped API Request
                     v
       [ Self-Hosted Grafana Enterprise ] (Basic Auth disabled, accepts tokens)

Key Technical Implementations

• Token-Less Local Configurations: The configuration file on the developer’s laptop remains entirely secret-free, containing only the designated routing URL. On first use, the proxy orchestrates a standard OIDC/OAuth 2.1 authorization code flow in the user’s browser. The resulting cryptographic JWT is saved securely in the operating system’s native keychain (via the IDE’s secure credential store).
• Deep Multi-Tenant Isolation: Self-hosted Grafana instances typically host data across multiple isolated teams/organizations. We enforce strict boundary alignment by verifying IdP group memberships via regular expressions during the token exchange step. Furthermore, every organizational slice maps to a completely distinct Kubernetes Pod with isolated networking.
• Dynamic Credential Ingestion: The upstream mcp-grafana core container never faces the network directly. Instead, the Pod uses cloud-native identities (such as AWS IRSA or Vault) to dynamically mount an organization-scoped, strictly Read-Only (Viewer) token from a centralized secrets manager. This completely mitigates the risk of credential leakage while fulfilling the missing OAuth gap highlighted in Issue #284.

Pillar 2: Machine-to-Machine Workarounds — Defending Routing Bypasses with Cilium eBPF

After securing the human workflow, we faced a secondary challenge native to open-source architectures: internal automation systems (such as Slack Support Bots and Incident Response Scripts) also needed to call the MCP servers, but the authentication proxy was strictly optimized for human interactive browser logins (authorization_code flow).

Until Phase 2 introduces programmatic Machine-to-Machine (M2M) client_credentials support to the proxy sidecar, we needed a secure way to let verified bots query metrics without exposing a giant perimeter hole.

The Solution: A Controlled Network Bypass Injected at the Linux Kernel Level

We established dedicated internal Kubernetes Services that target the core mcp-grafana server container (Port 8000) directly, bypassing the authentication sidecar entirely for intra-cluster traffic.

Exposing an unauthenticated backend port within a shared cluster is traditionally a severe security risk; any compromised pod in the service mesh could perform lateral movement to scrape monitoring data. To nullify this risk completely, we introduced a strict CiliumNetworkPolicy (CNP) driven by eBPF ingressDeny rules.

Here is our declarative Infrastructure-as-Code policy:

YAML

apiVersion: "cilium.io/v2"
kind: "CiliumNetworkPolicy"
metadata:
  name: "mcp-grafana-sa-access-allowlist"
  namespace: "mcp-monitoring"
spec:
  endpointSelector:
    matchLabels:
      "io.kubernetes.pod.namespace": "mcp-monitoring"
  ingressDeny:
    - toPorts:
        - ports:
            - port: "8000"
              protocol: "TCP"
      fromEndpoints:
        - matchExpressions:
            # Intercept all internal traffic traversing our interconnected Multi-Cluster Mesh
            - key: "io.cilium.k8s.policy.cluster"
              operator: "Exists"
            # [CRITICAL DEFENSE]: Explicitly DENY all namespaces EXCEPT our specific automation bots
            - key: "k8s:io.kubernetes.pod.namespace"
              operator: "NotIn"
              values:
                - "platform-support-bot"
                - "oncall-incident-bot"

Why This Network Defense is Ironclad

• Deny Precedence over Allow: Under Cilium’s advanced security model, ingressDeny evaluations supersede any standard global or broad namespace allow configurations. Port 8000 remains strictly locked down by default.
• Kernel-Level, Zero-Overhead Dropping: Because Cilium runs at the Linux kernel layer via eBPF, unauthorized pods attempting to probe or scan Port 8000 have their packets dropped instantly at the virtual ethernet pair (veth). The traffic never reaches the user space or the application stack, blocking lateral movement at the absolute lowest architectural level.
• Elegant Technical Debt Decommissioning: This approach treats the exception as an audited, explicit configuration. Once M2M capabilities are introduced to the proxy in Phase 2, we can deprecate the bypass services and dismantle this policy simply by removing the NotIn array.

Conclusion and Community Outlook

The beauty of open-source engineering lies in building reliable abstractions around upstream software constraints when community feature requests are stalled by architectural friction.

By designing this zero-trust wrapper architecture for a self-hosted Grafana MCP environment, we proved that:

• For Human Engineers: You don’t have to wait for native upstream support. Abstract infrastructure secrets completely off local developer machines by elevating authentication to the browser and the OS keychain using lightweight sidecar proxies.
• For Automated Machines: When temporary architectural workarounds are necessary, leverage the full force of cloud-native eBPF networking (like Cilium) to erect fortress-like, declarative firewalls around your technical debt.

This model lets our engineering teams unlock the massive speed gains of contextual AI troubleshooting while ensuring our underlying monitoring datasets remain completely fortified. If you are struggling with the constraints highlighted in Issue #284, we hope this framework provides a practical roadmap for your SecOps and Platform teams!