Building a Production-Grade Multi-Tenant Self Managed Kubernetes Platform on Azure — Without AKS

The Business Problem

Most platform engineering teams eventually hit the same inflection point.

You have three product teams — let’s call them payments, analytics, and dev. Each needs compute, networking, and Kubernetes. Running three separate clusters is operationally expensive and wastes cloud spend. Running one shared cluster is efficient, but creates a different problem: how do you guarantee that a misconfigured workload in dev can't exhaust payments' CPU, reach its network endpoints, read its secrets, or pull an untrusted container image?

The naive answer is: use a managed Kubernetes service, apply some namespaces, and hope for the best. That’s not isolation — that’s a shared blast radius with extra steps.

The real answer requires thinking about isolation at every layer of the stack: compute scheduling, kernel-level networking, API server access, and admission control. Each layer must be independently enforceable, because relying on a single control means a single failure mode.

This article documents a self-managed, multi-tenant Kubernetes platform I built from scratch on Azure — no AKS, no managed control plane, no shortcuts. The goal was to understand and own every layer: from raw Azure infrastructure through to GitOps-driven policy enforcement.

Why Not AKS?

AKS is a perfectly reasonable production choice. But it abstracts away the exact things worth understanding deeply: how kubeadm initialises a HA control plane, how the CNI wires pod networking into the kernel, how the API server actually enforces RBAC, and what happens at the node level when a pod is scheduled. If you only ever operate managed Kubernetes, you borrow that understanding from your cloud vendor. I wanted to own it.

There is also a practical argument: in regulated industries, organisations sometimes cannot use managed control planes due to data residency, compliance scope, or air-gap requirements. Knowing how to stand this up from scratch is a distinct and marketable capability.

Architecture

┌─ Azure ─────────────────────────────────────────────────────────┐
│  3 × Control Plane VMs   (Standard_D2s_v3, private IPs only)    │
│  3 × Worker VMSS          (1 per tenant, isolated node pools)   │
│  Standard Internal LB     (API server VIP :6443)                │
│  VNet 10.0.0.0/16         (CP: 10.0.1.0/24 / Workers: 10.0.2.0/24) │
│  Azure Bastion            (no public IPs on any VM)             │
│  NAT Gateway              (outbound internet, private subnets)  │
└─────────────────────────────────────────────────────────────────┘

Isolation model (defence in depth):
 Compute   — per-tenant VMSS node pools, taints + node affinity
 Network   — Cilium eBPF, zero-trust NetworkPolicies per namespace
 API       — RBAC Roles scoped to namespace, Azure AD group bindings
 Admission — Kyverno ClusterPolicies: validate+mutate at deploy time

The entire platform is declarative, version-controlled, and reproducible. Deployment from a clean Azure subscription to a fully operational multi-tenant cluster takes three commands.

1. Infrastructure — Terraform with a Single Source of Truth

The Azure estate is provisioned through a composable, modular Terraform codebase with four reusable modules: network, compute, load-balancer, and bastion. A single prod.yaml file is the source of truth for every value — VNet CIDRs, VM sizes, tenant names, LB private IP, pod CIDR, service CIDR, admin IP. Nothing is duplicated between Terraform, bootstrap scripts, or Kubernetes manifests.

# config/prod.yaml — one file governs everything
location: swedencentral
network:
  address_space: ["10.0.0.0/16"]
  subnets:
    control_plane: "10.0.1.0/24"
    workers: "10.0.2.0/24"
tenants:
  - name: payments
  - name: analytics
  - name: dev

Terraform reads this file via yamldecode() at plan time. terraform output -json exposes the resolved values as a structured contract that the bootstrap script consumes. This means changing the pod CIDR in one YAML file propagates correctly into Terraform, kubeadm init config, and Cilium Helm values — with zero manual synchronisation.

Notable Engineering Decisions

Per-tenant VMSS node pools. Rather than a single worker pool, each tenant gets its own azurerm_linux_virtual_machine_scale_set, dynamically generated via for_each over the tenant list. Adding a fourth tenant means adding one line to prod.yaml and running terraform apply.

Cloud-init as the provisioning contract. All VM prerequisites — kernel modules (overlay, br_netfilter), sysctl parameters for Kubernetes networking, containerd with systemd cgroup driver, kubeadm/kubelet/kubectl v1.32 — are baked into cloud-init templates. Nodes are cluster-joinable the moment they finish first boot. No Ansible, no configuration management, no drift.

Azure ILB hairpin workaround. Azure’s Standard Internal Load Balancer has a well-documented limitation: a VM in the backend pool cannot reach the ILB’s own VIP (hairpin/loopback is blocked). Since kubeadm writes all kubeconfigs pointing at the LB VIP, control-plane nodes cannot reach their own API server through the LB. The solution — a systemd oneshot unit (fix-kubelet-ilb.service) injected via cloud-init — waits for kubeadm to finish writing its conf files, then patches every /etc/kubernetes/*.conf to use the node's own IP rather than the VIP. This is the kind of problem you only encounter when you're not using a managed service that handles it silently.

Terraform remote state. A separate, independently-bootstrapped Terraform root module provisions the Azure Storage Account and private container used as the Terraform backend. State locking is handled natively by Azure Blob Storage lease mechanism.

2. Cluster Bootstrap — Automated HA kubeadm Without SSH

The bootstrap engine (bootstrap.ps1) orchestrates the full kubeadm lifecycle across six Azure VMs using **az vm run-command** — no SSH, no bastion hop, no jump host required during bootstrap. It:

1. Reads all configuration from terraform output -json — zero hardcoded values
2. Runs kubeadm init on cp-0 with the ILB VIP as controlPlaneEndpoint, generating the certificate key and join tokens
3. Joins cp-1 and cp-2 as additional control-plane members (kubeadm join --control-plane --certificate-key)
4. Joins all worker VMSS instances via az vmss run-command
5. Installs Cilium via Helm with kubeProxyReplacement: true (kube-proxy is skipped at kubeadm init time via --skip-phases=addon/kube-proxy)
6. Applies tenant=<name>:NoSchedule taints and tenant=<name> labels to each worker node pool
7. Installs ArgoCD and seeds the App of Apps root application

The script is fully idempotent: re-execution detects already-completed phases and skips them safely. This matters for partial failure recovery — if a worker join fails at step 4, re-running the script picks up exactly where it left off.

3. Networking — Cilium eBPF, kube-proxy Replacement

The platform uses Cilium as the CNI, configured with kubeProxyReplacement: true. This means:

• All service routing (ClusterIP, NodePort, LoadBalancer) is handled by Cilium’s eBPF dataplane rather than iptables rule chains
• Network policy enforcement happens at the kernel level — packets are evaluated in the eBPF program before they ever reach userspace
• The kube-proxy DaemonSet is never installed; kubeadm init is run with --skip-phases=addon/kube-proxy

IPAM is configured in cluster-pool mode: Cilium allocates per-node pod CIDRs from the global pod CIDR (192.168.0.0/16), without requiring Azure CNI or any cloud-provider networking plugin.

Cilium’s observability primitives (Hubble) are available as a natural future addition for L7 flow visibility per namespace.

4. Multi-Tenant Isolation — Four Independent Boundaries

This is the architectural centrepiece. Isolation is implemented as four independent, overlapping layers. A failure at one layer does not propagate through the others.

Layer 1 — Compute: VMSS Node Pools + Kyverno Scheduling Injection

Each tenant owns a dedicated VMSS. Worker nodes are tainted at bootstrap time:

kubectl taint node <node> tenant=payments:NoSchedule
kubectl label node <node> tenant=payments

A Kyverno ClusterPolicy with a mutate rule intercepts every incoming Pod at admission time. It reads the tenant label from the pod's namespace via a live API call and injects:

• A toleration matching the node taint (tenant=<name>:NoSchedule)
• A required nodeAffinity pinning the pod to nodes labelled tenant=<name>

context:
  - name: tenantName
    apiCall:
      urlPath: "/api/v1/namespaces/{{request.namespace}}"
      jmesPath: "metadata.labels.tenant"
mutate:
  patchStrategicMerge:
    spec:
      tolerations:
        - key: "tenant"
          operator: "Equal"
          value: "{{tenantName}}"
          effect: "NoSchedule"
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: "tenant"
                    operator: "In"
                    values: ["{{tenantName}}"]

No individual Deployment manifest needs to declare node affinity. Adding a new tenant automatically inherits the policy. VMSS scale-out nodes join pre-labelled via cloud-init --node-labels, so new nodes are immediately correct.

Layer 2 — Network: Zero-Trust NetworkPolicies

Every tenant namespace starts with a default-deny-all NetworkPolicy blocking all ingress and egress. Explicit policies then carve out only what is required:

• Intra-namespace pod-to-pod communication
• DNS egress to kube-system port 53 (UDP + TCP)
• Any cross-namespace flows must be explicitly declared

Since Cilium enforces these policies in eBPF rather than iptables, enforcement is both more performant and more reliable — there is no rule ordering concern, and rules are evaluated atomically.

Layer 3 — API: Namespace-Scoped RBAC

Each tenant gets a Kubernetes Role granting full workload management (pods, deployments, services, secrets, configmaps, jobs, cronjobs) scoped exclusively to their namespace, bound to an Azure AD group via RoleBinding:

subjects:
  - kind: Group
    name: payments-team   # Azure AD group or OIDC group claim
    apiGroup: rbac.authorization.k8s.io

No ClusterRole bindings are issued. A payments-team member issuing kubectl get pods -n analytics receives a 403 from the API server before the request touches any workload.

Layer 4 — Admission: Kyverno Policy Engine

Five ClusterPolicy resources govern all tenant namespaces (scoped via namespaceSelector: tenant: Exists, leaving kube-system, argocd, and kyverno unaffected):

Policy: disallow-root-containers Type: Validate/Enforce Prevents: UID 0 in containers and init containers

Policy: require-resource-limits Type: Validate/Enforce Prevents: Missing CPU/memory limits (noisy-neighbour exhaustion)

Policy: restrict-image-registries Type: Validate/Enforce Prevents: Images from non-approved registries (supply-chain attacks)

Policy: require-labels Type: Validate/Enforce Prevents: Pods missing mandatory observability/cost-attribution metadata

Policy: inject-tenant-scheduling Type: Mutate Prevents: Scheduling escapes — injects toleration + nodeAffinity on every pod

The registry allowlist lives in a Kyverno-watched ConfigMap. Updating it requires only a git push — no Kyverno restart, no policy redeploy. The admission webhook re-reads the ConfigMap on every request.

5. GitOps — App of Apps

The platform is fully Git-driven using ArgoCD.

Post-bootstrap, the cluster is entirely Git-driven via the App of Apps pattern:

• platform-root watches platform → continuously reconciles namespaces, RBAC, NetworkPolicies, Kyverno policies
• tenants-root watches tenants → manages tenant workload applications

Both applications run with automated.prune: true and automated.selfHeal: true. Any out-of-band kubectl apply or kubectl delete is reverted within the next sync cycle. There are no post-bootstrap manual cluster operations in the normal operating model.

6. From Zero to Platform

az login          # authenticate to Azure

make all          # terraform apply (full Azure estate)
                  # + automated kubeadm HA cluster bootstrap
                  # + Cilium CNI + ArgoCD install

git push          # ArgoCD activates — namespaces, RBAC, network
                  # policies, and Kyverno go live automatically

That is the complete deployment runbook. No portal clicks, no manual kubectl apply sequences, no undocumented steps. Tearing down and reprovisioning for DR testing is terraform destroy && make all. For Windows environments, make.ps1 is a drop-in with identical targets — no GNU make dependency.

What’s Next

The platform is a production-ready foundation. These additions are architecturally pre-positioned and require no structural changes to implement:

Kata Containers — hardware VM isolation per pod. The current model trusts the host kernel. Kata Containers runs each pod inside a lightweight hardware-virtualised VM (Cloud Hypervisor or QEMU), so a container escape lands in a disposable VM rather than on the shared host. A Kyverno policy would enforce runtimeClassName: kata on high-sensitivity namespaces like payments. Combined with per-tenant VMSS pools, this achieves nested virtualisation-based isolation — the strongest model short of dedicated physical hardware.

Kubernetes Gateway API. Replaces Ingress with a role-oriented traffic management model. The platform team owns the Gateway; tenant teams bind HTTPRoute resources from their own namespaces without cluster-admin access. Cilium's native Gateway API support enforces routes in eBPF — no sidecar proxy required.

External Secrets Operator + Azure Key Vault. Secrets would never touch etcd in plaintext. Per-tenant SecretStore resources authenticate to dedicated Key Vault instances via Azure AD Federated Identity (workload identity). A Kyverno policy would deny any Secret not originating from ESO.

Observability stack. observability is already scaffolded. Planned: Prometheus + kube-state-metrics for metrics, Loki for logs, Tempo for distributed traces, and Grafana with namespace-scoped RBAC dashboards — each tenant sees only their own data.

Cluster Autoscaler + KEDA. Worker VMSS pools are already structured correctly for the Cluster Autoscaler — nodes join pre-tainted and pre-labelled, so scale-out nodes are immediately correct without intervention. KEDA adds event-driven HPA on custom metrics (queue depth, HTTP RPS) per tenant.

Falco — runtime threat detection. Kyverno operates at admission time. Falco complements it with kernel syscall monitoring via eBPF, alerting on anomalous runtime behaviour — privilege escalation, unexpected outbound connections, sensitive file reads — feeding into the Alertmanager stack.

OPA/Conftest in CI. Policy violations currently surface at admission time. Shifting left with Conftest means a developer attempting to push a root container or a non-allowlisted image gets a pipeline failure at git push, not a Kubernetes webhook rejection. Feedback in seconds rather than minutes.

Why This Matters

Most Kubernetes experience is experience with managed services — which is valuable, but bounded. This project demonstrates something different:

• Kubernetes internals ownership: kubeadm HA topology, etcd clustering, control plane certificate SANs, CNI wiring at the kernel level
• Multi-layer security architecture: each isolation boundary is independently enforceable, independently auditable, and independently explainable to a compliance team
• Platform engineering thinking: the platform is a product — tenants onboard with zero manual intervention, policies are automatically inherited, and operational state is entirely in Git
• Real-world edge cases solved: Azure ILB hairpin, VMSS scale-out label propagation, cloud-init idempotency, Kyverno live ConfigMap reload, kube-proxy replacement with eBPF
• Operational simplicity at scale: three commands from zero to production. That is the standard a platform team should be held to.

Full implementation: https://github.com/jaeveloper/k8s-multitenancy-platform