Real-Time Meets Real-Scale: The Tech Powering Angel One’s Insights Platform

How do you process billions of events daily without breaking a sweat?

India’s order volume is exploding in every sector , and the diversity in India makes the real time analytics very important. Many sectors including trading will even see volumes which will be comparable to the complete US population in the next 5–8 years. Hence, the Indian tech ecosystem needs to focus both in feature development and cost optimized platform development.

Angel One realtime streaming pipelines process > 10 Billion events per day. With the rapid growth of data volume across sectors, particularly in trading, building a robust real-time analytics platform is no longer optional — it’s essential. Here’s how we designed and implemented a centralized analytics platform to unify real-time business metrics, fraud detection, and user-facing insights.

Figure 1.1: Snapshot of a Prism page in QA

Motivation :

All our data platforms are domain oriented — we leveraged the same idea of datamesh principles to build this platform.

• Domain Ownership
• Data as a product
• Self-serve data platform

Every domain owns the data it ingests in the kafka topics, so building a multi-tenancy model solution on top of it becomes simple.

The solution has to support user facing analytics where the peak qps can go till ~50k. Also the platform needs to support business dashboard having complex queries and joins.

Pinot is not costly, its the use case that itself is costly pinot comes out to be one of the cheapest solutions for the use case.

Data and Volume :

Over 2 billion events must be processed daily, spanning trading activities, KYC activities, and clickstream interactions, all handled at a transaction rate of 30,000 per second.

Pinot builds segments using the data it consumes and it provides 70–80% of the actual data which makes it cost effective in this space when compared to other stacks.

We achieved an almost 90% reduction in data footprint, significantly optimizing storage efficiency and enabling the retention of substantial historical data on disk. Utilizing Amazon EBS gp3 volumes, which cost $0.08 per GB per month, this reduction not only minimizes storage costs but also enhances data accessibility for real-time and analytical workloads.

Note : Standard S3 cost is around ~0.02$ per GB per month.

Architecture and components :

Figure 2.2 Pinot components

Apache Pinot, initially developed at LinkedIn and later open-sourced, is a distributed OLAP datastore built with a strong emphasis on distributed systems principles and multi-tenancy. Designed for real-time and high-throughput analytical queries, Pinot powers low-latency user-facing analytics at scale.

Core Components of Pinot

Pinot’s architecture is modular, with specific components tailored to handle ingestion, query execution, and cluster management. Key components include:

Pinot Controller

• The controller acts as the administrative core of a Pinot cluster, orchestrating the lifecycle of tables and schemas.
• As an Apache Helix Controller, it is responsible for resource management and orchestrating interactions between external systems and cluster components.
• It also maintains global metadata, such as configuration and schema definitions, using ZooKeeper as the persistent metadata store.

Pinot Server

• The server handles data ingestion from real-time streams (e.g., Kafka) and offline sources (e.g., batch files).
• All data is ingested and stored as immutable segments, whose size is defined by table-level configurations.
• These servers form the backbone of Pinot’s columnar storage, enabling efficient data storage and high-performance queries.

Pinot Broker

• The broker serves as the query execution engine, responsible for distributing queries across Pinot servers.
• It leverages a scatter-gather architecture, distributing query execution across servers, aggregating the results, and delivering them to users.
• Brokers play a critical role in Pinot’s real-time analytics pipeline, ensuring low-latency response times even for complex analytical queries.

Minion (Optional)

• Pinot includes Minion tasks, which enable background compute intensive batch ingestion jobs and tasks. However, Minions are not covered in this discussion.

At Angel One, Navigator is our real-time, user-facing insights registry built on top of Apache Pinot. It is designed to deliver low-latency analytics for mission-critical services while addressing unique challenges such as multi-tenancy and query fingerprinting.

Key Features of Navigator

Seamless Integration with Pinot

• Navigator leverages Pinot’s real-time ingestion capabilities for analytics. It is integrated with Angel One’s in-house authentication library (Common-Auth) to ensure secure access.

Broker Multi-Tenancy

• Out-of-the-box, Pinot brokers do not support multi-tenancy on Application Load Balancers (ALB) in EKS (Elastic Kubernetes Service).
• Navigator enables true multi-tenancy by implementing table-to-tenant ALB routing, ensuring tenant isolation and efficient resource utilization across brokers.

Query Fingerprinting

• Navigator implements query fingerprinting by registering prepared SQL statements and generating unique MD5 hashes for each query pattern.

This helps in:

• Performance Optimization: Analyzing query patterns for latency and resource usage.
• Cluster Protection: Identifying and mitigating query abuse or bottlenecks.

Figure 2.3 Overall Angel One’s Pinot ecosystem

In a distributed microservices architecture, state changes within the system are streamed as events to Kafka or Kinesis in real-time, providing a reliable and auditable trail of activity. For example, in a trading platform, trade state changes are emitted by the trading service to Kafka. These events serve as the foundational data source for a real-time analytics pipeline.

Apache Pinot, designed for real-time OLAP workloads, integrates seamlessly with Kafka, leveraging its native real-time ingestion capabilities. Kafka functions as the event streaming backbone, while Pinot serves as the OLAP datastore for real-time data analysis.

Real-Time Analytics Pipeline Workflow

Event Stream Consumption:

• Pinot real-time servers ingest messages directly from Kafka topics in near real-time, adhering to the schema and table configuration defined within Pinot.
• The ingestion process is fault-tolerant and partition-aware, ensuring scalability, high throughput, and reliability under heavy workloads.

Segment Creation and Storage:

• Incoming data is first stored in in-memory segments, optimized for real-time processing. High-performance, low-latency queries are enabled by indexes, such as Roaring Bitmap, that are created during segment generation.
• Concurrently, immutable segments are generated and persisted to disk. These segments ensure data durability and form the foundation for efficient historical queries.
• Pinot employs a columnar storage format and advanced indexing techniques (e.g., forward indexes, inverted indexes, and star-tree indexes) to further accelerate query execution and minimize response times.

At the heart of our platform are Pinot tables, which store and query the vast amounts of data ingested in real time. Configuring these tables correctly is critical for balancing query performance, resource utilization, and cost efficiency. Let’s look at how we fine-tuned our table configurations to optimize performance.

Table Configuration :

Pinot have three types of tables -

Realtime table — Consumes data from realtime sources like kafka, kinesis.

Offline table — Stores batch data from offline sources like s3 , files, etc.

Hybrid table — Loads data from both realtime and offline sources.

In Apache Pinot, data within tables is partitioned into independent units called segments. The size of these segments is governed by table-specific configurations, such as the segment flush threshold or row limit per segment.

During performance testing, we observed that segment size directly influences query performance. Smaller segments tend to degrade query latency because the broker must spawn and manage a higher number of threads to handle the increased segment count. Conversely, overly large segments can negatively impact performance due to increased memory overhead and prolonged disk I/O operations during query execution.

Optimal segment sizing is thus critical to achieving a balance between query throughput and resource utilization.

Pinot supports upsets for real-time tables. This also makes pinot stand out compared to other stacks.

Every table belongs to a tenant. If no tenant details are provided then the table belongs to the default tenant.

Deep Store :

While table configurations govern real-time ingestion and query performance, ensuring data durability and fault tolerance requires a robust storage strategy. Here’s how we leveraged the deep store.

For real-time tables, the default behavior involves segment construction in-memory on the server. Once completed, the segment undergoes the Segment Completion Protocol (SCP), during which it is uploaded to the lead controller. The lead controller then writes the segment to the deep store, as illustrated in the accompanying diagram.

To mitigate potential bottlenecks at the controller level, we have enabled direct server-to-S3 pushes. This bypasses the controller for segment uploads, enhancing throughput and scalability. In scenarios where in-memory or disk-resident segments are lost, the system remains resilient as segments can be regenerated from the deep store backup.

With a scalable ingestion and storage pipeline in place, ensuring system reliability and performance becomes paramount. Monitoring and observability play a crucial role in identifying bottlenecks and maintaining operational excellence.

Monitoring and Observability :

Prometheus collects metrics from Pinot and other components in real-time, enabling detailed monitoring of system performance and resource utilization. It is configured to scrape Pinot’s metrics endpoints and store time-series data for analysis.

Grafana is used to visualize the metrics.

AngeOne’s Contribution :

As part of our ongoing commitment to enhancing open-source platforms, Angel One has contributed to Apache Pinot by addressing the handling of untagged servers in the controller’s availability check. This improvement ensures that servers without specific tags are appropriately managed, thereby enhancing the robustness and reliability of cluster operations.

Enhancement in server multi-tenancy :

https://github.com/apache/pinot/commit/601c5b73540b34dd159fc5c12521e37706c0d31e

TLS support [Note : Some more changes are required to make it work ]: https://github.com/apache/pinot/commit/21ff6bf3545e4fe15fcdee9cc3851a04f85cc08c

We have also contributed to the core data structure of bitmaps :

https://github.com/RoaringBitmap/RoaringBitmap/commit/d74b6ba39129695edd08b4768b02d0ee3190b543

Some more works are in the pipeline for both Pinot and Roaring Bitmaps.

Implementation :

One of the major wins of Pinot has been the clickstream data analysis. It helped us build real-time analytics dashboards on superset — Imagine getting the response within a second on 1.5 Billion records.

We are leveraging multiple indexes provided by pinot like inverted index , star tree index, forward-index , etc.

Conclusion and Future Works :

Pinot has been a game-changer in building our real-time analytics platform, staying true to distributed system principles like scalability and fault tolerance. For now, we’ve opted for a self-managed setup, giving us full control over customization and operations.

In the initial rollout, we focused on real-time streaming integrations, working extensively on connecting Kafka and Kinesis and with Pinot. This allowed us to process and query live data seamlessly.

Our next big milestone is to bring offline tables into the picture. We’re planning to implement batch ingestion from Amazon S3, enabling us to combine real-time and historical data for more comprehensive analytics.

There are on-going Evaluation and benchmarking activities going on to replace open-search with Pinot in many use cases.

DR strategy. (Right now , Pinot is not leveraged by business critical orgs)

https://pinot.apache.org/

https://startree.ai/

We are thankful for the support from the Pinot Slack community too.