How we seamlessly S(C)HIP metrics and logs for 1000+ developers — Part 1: Metrics

At Adevinta, our Platform team operates an internal Kubernetes as a Service platform called SCHIP.

SCHIP is a multi-tenant Kubernetes platform where engineers across our marketplaces can build, deploy, and operate microservices without worrying about the underlying infrastructure. SCHIP runs 30+ Kubernetes clusters and serves 100+ tenants, which involves 1000+ developers, each isolated at the namespace level.

Beyond just running workloads, SCHIP also provides a set of opinionated platform capabilities out of the box:

• Certificates & DNS
• Ingress & IAM
• And — what we’ll focus on in this article — metrics and logs

This post is Part 1, where we’ll deep-dive into how we s(c)hip metrics seamlessly to our users as part of the platform’s golden path.

The problem we wanted to solve

Our users interact with SCHIP primarily at the namespace level. We have given the user autonomy over their namespaces. They don’t manage clusters, nodes, or Prometheus instances — and we want to keep it that way.

Our goal was simple:

Users should be able to deploy their workloads, and metrics should “just work”.

No Prometheus setup. No scraping configuration. No worrying about cardinality, storage, or dashboards.

If a workload exposes Prometheus metrics, all the user needs is:

prometheus.io/scrape: "true"

…and magically, their metrics show up in their own Grafana Cloud stack, alongside all the infrastructure metrics they need to operate their service.

The golden path for metrics

We deliberately designed SCHIP’s observability around a golden path concept:

• Zero-config by default
• Opinionated, but extensible
• Safe for multi-tenancy
• Scales with cluster and tenant growth

Under the hood, we distinguish between two types of metrics.

Metrics we provide out of the box

1. System metrics/Infrastructure metrics

These are metrics that every service needs, regardless of what the application does. These are a few examples:

Ingress metrics

• Requests per minute
• Error rates
• Latency percentiles

Resource usage metrics

• CPU / memory requested vs used
• Network In/Out
• Useful for capacity planning and debugging performance issues

HPA metrics

• Scaling behavior
• Replica counts
• Scale-up / scale-down signals

Other platform metrics

• Certificate expiration
• Cost signals
• Platform-specific indicators

All of these are delivered automatically, without any user configuration.

Abstraction matters

We don’t expose raw infrastructure metrics directly. Instead, we provide them through a stable abstraction:

• Standardized metric names This avoids implementation leaks. If we change the underlying tooling such as cadvior version, users don’t need to update their queries, dashboards or alerts.
• Pre-aggregated metrics To control cardinality and volume, we expose aggregated views (for example, sum or avg over 30s) rather than raw series.

This keeps tenant stacks lightweight, predictable, and safe.

Dashboards as part of the platform

Because SCHIP owns these metrics, we can also ship curated dashboards right into the customer’s Grafana cloud stack:

• Consistent across tenants
• Automatically provisioned
• Based on stable metric contracts

From day one, users get meaningful dashboards without building anything themselves.

Curated dashboards for each tenant based on system metrics

2. Application metrics

These are metrics exposed by user applications/workloads.

If a pod exposes Prometheus metrics and is annotated correctly, SCHIP discovers it automatically, scrapes it, and ships it to the Grafana Cloud stack of the tenant along with the system metrics

From the user’s perspective:

Infrastructure metrics + application metrics are ready to query, alert on, and visualize without hassle.

No extra steps required.

Self-service: the only thing users need to do

The only prerequisite is that the user has a pre-provisioned Grafana Cloud stack. (If they prefer to be on the golden path, otherwise, we also support sending the metrics to different backend if preferred)

Then they simply annotate their namespace with their stack name:

From there, the platform takes over all the credentials management, remote write configuration, and dashboard provisioning. All handled automatically.

How did we actually build this?

Now let’s talk about how we make all of these effortless for our users.

High-level design

Architecture diagram of the orchestration

Let’s walk through each component

Cluster-level Prometheus: cluster-metrics

Each cluster runs a set of central Prometheus instances we call cluster-metrics.

These instances scrape key system metrics, including, but not limited to:

• kubelet metrics (CPU, memory, network, etc.)
• kube-state-metrics
• HPA exporter
• cert-manager
• All node-level exporters
• SCHIP’s platform components metrics

In this central Prometheus instance, it’s where we:

• Define aggregation rules
• Decide which metrics are safe to expose
• Enforce tenant isolation

Metrics are selected using:

• Namespace labels
• A federate=true label to explicitly mark exportable metrics. The rest will stay local to the instance for debugging high cardinality metrics in real time.

Sending everything downstream would:

• Explode cardinality
• Break isolation
• Overwhelm tenant stacks

Namespace-level Prometheus: one per tenant

Each tenant namespace gets its own Prometheus instance.

This tenant Prometheus, is responsible for:

1. Scrapes application metrics in its namespace via PodMonitor / ServiceMonitor
2. Scrapes filtered system metrics from cluster-metrics using federation

• federate=true
• namespace=<tenant> for isolation

At this point, the tenant Prometheus contains:

• Application metrics inside the namespace
• Relevant infrastructure metrics for the namespace

Why not let tenants query Prometheus directly?

We don’t. We want to limit the risk surface and limit the actions our users can do with the underlying components.

Instead, the tenant Prometheus remote-writes everything to Grafana Cloud.

Our user can focus on their metrics, and we will make sure that the metrics flow to their stacks without problems.

The glue that makes it all work

At this point, some of you would already have a few questions, such as:

• Who creates the tenant Prometheus?
• Who creates PodMonitor objects?
• Where do credentials come from?
• Who provisions dashboards?

No engineers in a basement doing manual work — we promise (we don’t do that here).

Introducing the Observability Operator

All of this is orchestrated by our open-source Kubernetes operator: https://github.com/adevinta/observability-operator (aka Grafana Cloud Operator)

The Observability Operator manages and orchestrates observability pipelines in multi-tenant Kubernetes clusters, requiring minimal configuration from users and providing sane defaults out of the box.

Observability operator

By leveraging Prometheus operator inside the cluster, and with its implementation of the Grafana Cloud API, and the controller-runtime, the observability operator can:

• Discover workloads
• Provision Prometheus instances
• Retrieve Grafana Cloud stack details
• Create the scraping configuration and the remote write configuration correctly

Workflow: from annotation to metrics

Let’s take a quick look at the overview of how the operator works from start until the metrics lie in the destination Grafana cloud stack

1. User deploys a workload

prometheus.io/scrape: "true"

2. PodReconciler └─ Creates PodMonitor for the workload └─ Inherits annotations from namespace

3. NamespaceReconciler └─ Syncs namespace annotations / exclude namespaces └─ Resolves Grafana Cloud stack name

4. PodMonitorReconciler

└─ Creates / updates Prometheus CR for tenant Prometheus └─ Calls Grafana Cloud API to obtain required information └─ Fetches Prom URL, Instance ID └─ Stores credentials in a Secret └─ Configures remote_write

5. Prometheus Operator └─ Creates and manages Prometheus StatefulSet └─ Scrapes application + federated metrics └─ Remote-writes to the Grafana Cloud stack

Overall workflow

The operator also supports

• Sending metrics to different Grafana cloud stacks at the same time supporting migration
• Sending metrics to different backends (VictoriaMetrics)

Please check out the repository and feel free to adopt or contribute to the operator.

Challenges we had to solve

Here, I want to take an opportunity to share some of the challenges that we need to tackle along the way.

Prometheus size & cardinality

Cluster-level metrics grow fast. It’s a central metrics storing metrics from all of the exporters in the cluster.

Ingress metrics, in particular, are high-volume and high-cardinality, so we:

• Shard them into a dedicated ingress-metrics Prometheus
• We configure the tenant Prometheus to extend the scrape to the ingress metrics

Rate limits & error visibility

Users don’t have access to their Prometheus instances, which makes failures harder to debug.

We addressed this by:

• Sensible default rate limits
• Allowing controlled tuning

• Providing Grafana Cloud dashboards showing: Ingestion rates, Errors, Limits

So the users can troubleshoot autonomously when their metrics do not arrive, and we will only be needed if the pipe is somehow broken.

Scale & resilience

With surging events, for example, pod churns, traffic spike, it’s highly possible that your metrics cardinality will explode and your Prometheus instance will likely suffer downtime.

We have invested heavily in:

• HA Prometheus sets up

so that when one instance is down or gets disrupted during maintenance, the metrics can still be served from another instance.

• Vertical Pod Autoscaling (VPA)

We fine-tune our Vertical Pod autoscaler, and with the help of the Karpenter flexible node type. We make it more robust to cope with dynamic change of metric volume. Also, with the help of a dedicated Karpenter node pool for Prometheus, things are more robust. We share more detail in: https://medium.com/adevinta-tech-blog/how-we-saved-5k-a-month-with-a-single-line-of-code-646311b063b4

• WAL snapshots on shutdown to reduce startup times

Another issue is that Prometheus attempts to rebuild the entire WAL on startup. With 30GB+ of data, this process takes an excessively long time and jeopardizes our HA setup. During an OOM event, one instance fails, and shortly after, the other instance is also at risk because the instance that receives the VPA update takes a very long time to restart.

We mitigate this by using the memory-snapshot-on-shutdown feature, which significantly reduces startup time https://prometheus.io/docs/prometheus/latest/feature_flags/#memory-snapshot-on-shutdown

And recently with Kubernetes now supports “In-Place Pod Resize”, along with newer VPA version. We are able to adjust the Prometheus instance in-place without disrupting the instance.

in-place support from VPA

• Careful handling of pod churn and metric explosions

We actively monitor metrics cardinality to ensure that there’s no sharp spike and if needed, aggregate or drop them.

What’s next?

This post covered metrics — how we make them invisible, scalable, and boring (in the best possible way).

In Part 2, we’ll cover logs, including:

• Shipping application logs
• Multi-tenant isolation
• Kubernetes events as logs
• And how the Observability operator helps keeping the same golden-path experience

Stay tuned 🚀