Running Multi-Tenant Kubernetes Clusters at Scale: Our Production Toolbox

When operating Kubernetes clusters in production—particularly in multi-tenant environments—There are several critical concerns to think about: Tenant isolation, Automation, Resource efficiency, Observability, Security, and Reliability.

Balancing these areas effectively requires a robust toolkit. Over the years, we've built up a battle-tested arsenal of open-source and in-house tools to keep our clusters secure, scalable, observable, and maintainable with minimal hassle.

In this post, I'll share the tools we rely on daily, categorized by their primary function. Whether you're beginning your Kubernetes journey or scaling across teams and complex workloads, I hope this serves as a useful reference.

⚙️ 1. Automation & GitOps

Automation is essential for reliable and repeatable operations, minimizing manual errors and speeding up deployments through declarative infrastructure and GitOps practices.

• **ArgoCD**: Declarative continuous delivery and GitOps workflows. This is the main tool for managing all of the components we install into our cluster.
• ArgoCD ApplicationSet: Manages applications across multiple clusters or environments at scale. We used ApplicationSet on top of the ArgoCD with progressive roll-out to carry out changes for multiple clusters we managed in the fleet. I’ve covered in detail how we use it here.
• **KubeJanitor**: Cleans up unused Kubernetes resources like completed jobs and stale PVCs. This is offered to our user as well so they can take care of the unused resources in their own namespaces.
• **Pluto**: Detects Kubernetes APIs that are deprecated, ensuring smooth upgrades. We used it to establish metrics that will be used during Kubernetes version upgrades to ensure nothing breaks and smooth transitioning.
• **Stakater Reloader**: The popular operator that automatically triggers pod restarts upon changes in ConfigMaps and Secrets. We used this a lot since we have a lot of dynamic configurations that got changed from different trigger points. so, the reloader comes in to rollout those changes without any manual intervention.
• **Renovate CronJob**: Keeps Helm chart versions and Kubernetes manifests up to date automatically. As we are maintaining a lot of components, having an automated way to keep track of their newer versions when there’s a new patch, minor or major is very helpful to maintain them.
• **KRO**: Useful for defining higher-level platform APIs and composing Kubernetes resources into reusable abstractions. since we are running platform and defining a proper interfaces are key to keep things rolling without frictions. KRO relieves us from the hard work of maintaining a cluster operator just to provide an abstraction, allowing us to iterate easily on newer CRDs.
• **Cloud Custodian**: Automates cloud governance across accounts. Mangaging more than 10 AWS acccounts come with tons of toil. In our case, it can monitor service quota usage and automatically open increase requests before quota exhaustion causes scaling failures.

📦 2. Autoscaling & Resource Management

Efficient resource management and automatic scaling are critical for performance and cost optimization. It allows us to sleep better at night without waking up just to add another replica of pod or add another node just to satisfy the alert.

• HPA / VPA / KEDA: Covers horizontal, vertical, and event-driven autoscaling. All of these have their own specific case. HPA for most of the stateless application scaling, VPA for stateful and DaemonSet and lastly KEDA for application-specific behavior such as scaling ingress based on request per seconds metrics
• **Cluster Autoscaler**: Adjusts node group capacity based on unschedulable workloads. Since we already moved to Karpenter this is kept only for the controller nodes where we run certain things in segregation such as Karpenter controller itself.
• **Karpenter**: Provisions right-sized nodes dynamically based on workload requirements. Karpenter allows us to provide capacity to run workloads in a very flexible fashion.
• **Kube-downscaler**: Reduces resource usage by scaling down workloads outside business hours. We used it to scale down development environment at night to save on cost.
• Metrics-server: Provides CPU and memory metrics required by HPA and other components.
• HPA Exporter: Exposes HPA state and behavior as metrics, useful for understanding scaling decisions.
• **Noe**: Our custom-developed operator that automatically applies node selectors and taints based on container image architecture. It’s very useful to run a mix of ARM and X86 images especially during a transition period.

📊 3. Observability, Metrics & Synthetic Checks

Metrics and logs are key to our platform for making all the needed decision and also provide them to our users as well.

• **Prometheus Operator**: Manages Prometheus instances and alerting resources through Kubernetes CRDs. We use it as a base for our customer operator that create one Prometheus instance per customer.
• **Observability operator**: Manages observability metrics/logs/traces for the users running in our platform. It takes care of provisioning Prometheus instance for our tenants and does all the wiring so the logs/metrics/traces end up in the right place whether it be Grafana cloud stack or custom endpoints.
• Fluent Bit: Lightweight log collector and forwarder run as DaemonSet to collect all the logs from our nodes.
• Fluentd: Log processing and aggregation layer for more complex routing or transformation, we use fluentd in conjunction with the Adevinta OSS observability operator to orchestrate sending logs to different managed log servers.
• **Kubernetes Event Exporter**: Exports Kubernetes events as structured logs into centralized logging, making debugging and correlation much easier.
• Kube-state-metrics: Exposes Kubernetes object state as Prometheus metrics. We use this to provide many core metrics, but at the same time, use it to collect metrics on custom CRDs.
• Prometheus Node Exporter: Provides node-level system metrics.
• Prometheus Blackbox Exporter: Probes DNS, HTTP, TCP, ICMP, and other external signals used in SLI calculations.
• Prometheus Conntrack Exporter: Exposes connection tracking metrics, useful for detecting networking pressure or exhaustion.
• Prometheus AWS Exporter / YACE: Pulls CloudWatch metrics into Prometheus. We used this to monitor our AWS resource usage including EBS metrics and such.
• Custom Canary Application (custom operator): Runs multi-tenancy checks such as ingress reachability, network policy enforcement, and permission boundaries. This component is key as it’s used as the main source of our alerting system to ping us to check when things go wrong, and also being used as gatekeeper to our cluster deployment automation.

4. Security, Policy & Secrets

• **Kyverno**: Kubernetes-native policy engine for validation, mutation, generation, and image verification. We used it for both enforcing multi-tenancy configuration and also facilitate configuration via mutating policies.
• External Secrets Operator: Syncs secrets from external secret stores into Kubernetes. This allows our user to bring their own secrets on top of our platform including Vault, AWS Parameter store and secret management.
• **Sealed Secrets**: Allows encrypted secrets to be stored safely in Git. We provide the endpoint for our users to sealed their secrets to only be used by their namespace in our platform. This needs a greater guarantee of namespace ownership to reduce attack surface.
• **Cert-manager**: Automates certificate issuance and renewal. Paired with LetsEncrypt and ZeroSSL providing seamless certificate lifecycle management for our users.

6. Networking, DNS & Traffic Management

• **ExternalDNS**: Automatically manages DNS records from Kubernetes resources to our Route53 records.
• **Traffic Controller**: Custom operator for registering traffic and managing weighted routing.

The two combos had been described in detail in this blogpost https://medium.com/adevinta-tech-blog/achieve-a-smaller-blast-radius-with-highly-available-kubernetes-clusters-96f9a475544b

• NodeLocal DNS Cache: Improves DNS performance and reduces load on CoreDNS and upstream resolvers.
• Retina: We used Retina to provide more visibility in the networking layer. Our CNI was still on AWS VPC CNI but we rely on Retina for a lot of metrics especially, cross-AZ network monitoring which was challenging to have at scale.
• Ingress-allow-listing: This is another Adevinta OSS project that we used for providing reusable CRD for the user to use on their ingress objects to provide allow-listing without repeating CIDRs.

7. Reliability, Recovery & Node Operations

• **Velero**: Backs up and restores Kubernetes resources and persistent volumes. Not only used for backup, but there were times where this became useful when debugging the previous state of Kubernetes objects.
• Node Problem Detector: Detects node-level issues and surfaces them as node conditions or events. Combined with a custom operator, it can trigger remediation actions such as cordoning, draining, or replacing unhealthy nodes.
• Custom Remediation Operator: We called this, housekeeper operator, it watches node events, pod events, conditions or even custom Prometheus metrics, and takes automated recovery actions.

Some of the most valuable tools in our production setup are not generic open-source components, but small custom operators and canaries built around our own failure modes.

Our custom canary application continuously validates multi-tenancy assumptions such as ingress availability, network policy enforcement, and permission boundaries. Our traffic controller manages weighted traffic registration, while node health automation reacts to node conditions emitted by node-problem-detector. Noe operator manage multi-architecture container images without customer’s hassle. These tools are custom but the pattern they solve is broadly useful: encode your platform assumptions as continuous checks, metrics, and automated remediation.

The real value of this toolbox is not any single component. It is the way these tools reinforce each other: GitOps gives us repeatability, observability gives us confidence, policy gives us guardrails, and automation lets us operate many clusters without turning every incident into manual toil.

Hopefully, this post gives you an idea and could help you fill some gaps that you might have missed.

Thank you and see you in another blog post.