Beyond OpenTSDB: How PhonePe Scaled to 10M+ Writes/Sec with VictoriaMetrics

The Next Evolution of Our 500 Billion Daily Metrics Journey

In our previous technical deep-dives (Managing 300 Billion Daily Metrics), we shared how PhonePe built a world-class monitoring stack on top of OpenTSDB. For four years, from 2021 to 2025, OpenTSDB was the bedrock of our observability, reliably handling a staggering 10 million writes per second.

However, as PhonePe’s ecosystem evolved — with more microservices, and deeper cardinality — we began to hit the physical and architectural limits of OpenTsdb. To power the next generation of financial services, we needed a modern engine.

Today, we are excited to share why and how we transitioned to VictoriaMetrics.

The OpenTSDB Era: Why 2025 demanded a change

OpenTSDB served us well, but it was designed for a different era of time-series data. As our scale grew, three primary bottlenecks emerged:

1. The Row Key Trap

OpenTSDB relies on a fixed Row Key design in HBase. While this is efficient for specific time-range lookups, it is rigid. When your labels change or cardinality spikes, you face high scan cost.

2. The Cost of “The Missing Index”

OpenTSDB lacks a global inverted index for label values. If a query doesn’t perfectly align with the UID-based prefix in the row key, the system is forced to perform massive table scans. At 10 million writes per second, “scanning the entire table” is no longer a viable query strategy — it is a recipe for dashboard timeouts and infrastructure strain.

3. The Lack of Isolation (The “Noisy Neighbour” Problem)

OpenTSDB is essentially a giant, global table. Within a massive organisation like PhonePe, with 30–40 different engineering teams, there was no native support for multi-tenancy. A single team accidentally pushing high-cardinality metrics (like unique user IDs in labels) could degrade the performance for every other team in the company.

Enter VictoriaMetrics: New Age Technology for New Age Scale

In our search for a successor, we looked for a “Shared-Nothing” architecture that could decouple ingestion from storage and querying. VictoriaMetrics emerged as the clear winner.

Unlike the complex Hadoop ecosystem required by OpenTSDB, VictoriaMetrics is a set of lightweight, highly optimized components:

• vminsert: Stateless ingestion proxies.
• vmstorage: Independent data nodes.
• vmselect: Query aggregators.

This separation allows us to scale our “write” capacity independently of our “read” capacity — a necessity when dealing with 10M+ metrics per second during peak hours.

The PhonePe Architecture: Reliability at the Core

To ensure this new stack could handle our 500M+ active time series, we laid out a robust, multi-tenant architecture designed for performance and containment.

Native Multi-Tenancy

We moved away from the “global table” approach. Using VictoriaMetrics’ AccountID feature, we logically isolated every team at the storage level. Using vmauth as our routing layer, we provide each team with their own token.

• Isolation: A cardinality explosion in the “Payments” team’s metrics stays in their index and never touches the “Accounting” team’s performance.
• Self-Service: Teams manage their own data retention and limits.

Our Architecture: High-Performance Compute Isolation & Storage Topologies

To seamlessly handle our massive metrics throughput, our core compute components — VMInsert and VMSelect — are fully containerized using Docker for maximum agility and rapid scaling. In contrast, the stateful VMStorage tier is deployed directly onto virtual machines pinned strictly to a single NUMA node on our bare-metal hardware. This isolation guarantees predictable memory latency and ensures the second NUMA node remains completely free for separate, CPU-intensive workloads.

Each VMStorage node drives all 10 local disks, which are carved into two independent 5-disk Zpools and aggregated using MergerFS to present a unified file system path to the storage application.

Our environment is divided into two distinct functional domains: a 15-node long-term cluster (001–015) and a highly dense 3-node short-term cluster (018–020). To safeguard against unexpected bare-metal failures, nodes 016 and 017 are maintained as completely unassigned hot-standbys. For disaster recovery, we execute automated daily snapshots using vmbackup targeting a remote Apache Ozone S3 cluster. Should any production node suffer a catastrophic hardware failure, we can instantly trigger a vmrestore onto standby node 016 or 017, seamlessly bringing a fully hydrated, healthy node back into active rotation without impacting cluster availability.

The Results: 5x Better Throughput

The results of our migration have been nothing short of transformative.

In our head-to-head load tests, we observed that VictoriaMetrics delivers nearly 5x more throughput per server compared to our optimized OpenTSDB/HBase setup.

• Efficiency: We have significantly reduced our hardware footprint while increasing our ingestion ceiling.
• Compression: Data that previously took up petabytes in HBase is now stored more compactly, thanks to VictoriaMetrics’ specialized compression algorithms for time-series data.
• Query Speed: Queries that previously required “Full Table Scans” in OpenTSDB now return in milliseconds, thanks to the Contiguous Indexing of labels.

Conclusion

The move from OpenTSDB to VictoriaMetrics isn’t just a change in database; it’s a shift in how PhonePe views observability. By moving to a modern, shared-nothing, multi-tenant architecture, we have built a platform that doesn’t just store metrics — it provides instant, actionable insights for hundreds of engineers.

As we continue to scale toward the next trillion metrics, VictoriaMetrics ensures that our observability stack remains as fast and reliable as the payments we process.