Taming the Prometheus Beast: How I Fixed a Metric Cardinality Explosion in Project HAMi.

If you’ve spent enough time working with Kubernetes, monitoring, and cloud-native infrastructure, you already know that Prometheus is an incredibly powerful tool. But if you aren’t careful with how you design your metrics, Prometheus can quickly turn from a helpful watchdog into a resource-devouring monster.

Recently, I encountered this exact scenario while contributing to Project HAMi (Heterogeneous AI Computing Virtualization), a massive open-source project used for managing GPU resources in Kubernetes. A bug in the metric design was causing a Prometheus cardinality explosion, bringing monitoring stacks like VictoriaMetrics to their knees.

Here is a deep dive into the issue, why it happened, and how I fixed it.

🧨 The Problem: What is a Cardinality Explosion?

To understand the bug, you have to understand how Prometheus stores data. Prometheus is a time-series database. Every unique combination of a metric name and its labels creates a brand-new, distinct time series.

For example, if you have a metric called http_requests_total and a label called status_code, you might generate a few time series:

• http_requests_total{status_code="200"}
• http_requests_total{status_code="404"}
• http_requests_total{status_code="500"}

Since HTTP status codes are bounded (there are only a handful of them), this is perfectly safe.

But what happens if you use a dynamic, unbounded value as a label? A cardinality explosion.

🔍 The Root Cause in Project HAMi (Issue #1623)

A user opened Issue #1623 reporting that the metric Device_memory_desc_of_container was causing a severe time series explosion.

I dove into the codebase (cmd/vGPUmonitor/metrics.go) and found the culprit immediately. Here is how the metric was originally defined:

Go

ctrDeviceMemorydesc = prometheus.NewDesc(
    "Device_memory_desc_of_container",
    "Container device memory description",
    []string{"podnamespace", "podname", "ctrname", "vdeviceid", "deviceuuid", "context", "module", "data", "offset"},
    nil,
)

Do you spot the issue? The labels context, module, data, and offset were storing actual dynamic memory values in bytes.

Memory usage changes constantly. Every time a container’s memory shifted by even a single byte, Prometheus created an entirely new time series:

• Device_memory_desc_of_container{data="2453799936", ...}
• Device_memory_desc_of_container{data="2453799937", ...}
• Device_memory_desc_of_container{data="2453799938", ...}

This unbounded label generation flooded the database, consuming massive amounts of RAM and CPU, eventually causing the monitoring stack to crash.

🛠️ The Fix: Moving Data from Labels to Values (PR #1628)

The golden rule of Prometheus is: Never use highly dynamic data (like IDs, timestamps, or raw byte counts) as labels. They must be metric values.

To fix this, I completely refactored how HAMi handles device memory telemetry.

Step 1: Deprecating the Bad Labels

First, I had to ensure backward compatibility while stopping the bleed. I updated the description of the old metric to warn users that these labels were deprecated, and I stopped passing the dynamic memory values into the label arrays.

Step 2: Creating Dedicated Metrics

Instead of packing memory data into strings as labels, I created brand-new, dedicated Prometheus metrics using GaugeValue (since memory can go up and down):

Go

ctrDeviceMemoryContextDesc = prometheus.NewDesc(
    "vGPU_device_memory_context_size_bytes",
    "Container device memory context size",
    []string{"podnamespace", "podname", "ctrname", "vdeviceid", "deviceuuid"}, nil,
)

ctrDeviceMemoryModuleDesc = prometheus.NewDesc(
    "vGPU_device_memory_module_size_bytes",
    "Container device memory module size",
    []string{"podnamespace", "podname", "ctrname", "vdeviceid", "deviceuuid"}, nil,
)

ctrDeviceMemoryBufferDesc = prometheus.NewDesc(
    "vGPU_device_memory_buffer_size_bytes",
    "Container device memory buffer size",
    []string{"podnamespace", "podname", "ctrname", "vdeviceid", "deviceuuid"}, nil,
)

Step 3: Pushing the Telemetry

Finally, I updated the collectContainerMetrics loop to push these values correctly. Instead of appending strings to a memoryLabels array, the system now safely records the byte size as the core float64 value of the new metrics:

Go

if err := sendMetric(ch, ctrDeviceMemoryContextDesc, prometheus.GaugeValue, float64(memoryContextSize), labels...); err != nil {
    klog.Errorf("Failed to send Device Memory context size metric: %v", err)
    return err
}
// Repeated for Module and Buffer size...

📈 The Result

After pushing the code and going through code review with the HAMi maintainers, PR #1628 was merged!

The impact:

• Zero Cardinality Explosions: By moving the memory bytes out of the labels and into the metric values, the number of generated time series dropped from potentially infinite to a fixed, manageable number per container.
• Better Data Typing: The metric types are now correctly defined as Gauges rather than generic counters or labels, allowing for accurate PromQL queries (like calculating memory spikes over time).
• Cleaner Logs: I also took the opportunity to clean up the error logs, making it easier to debug specific memory metrics in the future.

💡 Key Takeaway for Developers

If there is one thing you take away from this bug fix, it’s this: Prometheus labels are meant to identify the source of the data, not the data itself. If you find yourself putting things like user IDs, unique hash strings, or raw fluctuating numbers inside a []string{} label array, stop! Your monitoring stack will thank you.

Have you ever accidentally nuked a monitoring stack with bad metrics? Let me know in the comments below! And if you are interested in Kubernetes, Go, and open-source backend development, feel free to check out my GitHub at @maishivamhoo123.