Surviving 1 Million Requests Per Second: A Deep Dive into Extreme Scale Engineering

When we build web applications, the mantra “code that just works is good enough” often prevails. But what happens when you enter the extreme scale environment of companies like Uber, Netflix, or Google?

To put things in perspective, Amazon Web Services’ Identity and Access Management (IAM) service handles over 400 million requests per second globally. At this elite level of engineering, a simple sub-optimal algorithm isn’t a “bug” — it’s a catastrophic financial drain that can cost tens of thousands of dollars.

In this deep dive, I’m going to simulate an architecture that is capable of handling 1 million HTTP requests per second. We will explore the bottlenecks of CPU clustering, network bandwidth limits, database Disk I/O, and why your favorite tech stack might completely fail you.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Phase 1: The Framework Tax (Node.js)

Before scaling hardware, you must optimize software. Every web framework carries “overhead” — the extra CPU cycles required to process an HTTP request before your actual business logic even runs.

When benchmarking a simple GET /simple route using a multi-core machine (simulating traffic using the autocannon tool), the choice of framework drastically changes performance:

• Express.js: Handled roughly ~14,000 to 20,000 Requests Per Second (RPS).

• Fastify: Designed for speed, it averaged ~66,000 to 77,000 RPS.

• Cpeak (Custom/Raw Node): A zero-dependency framework built for this test achieved ~73,000 RPS.

Industry Insight: At high scales, Express.js is a massive liability. By simply switching to Fastify or a raw Node.js implementation, you can increase your throughput by over 300% without spending a single extra dollar on hardware.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Phase 2: The Network Bottleneck (Payload Size Matters)

To push beyond local testing, we move to AWS and provision a “Power Server” (c8i.32xlarge) boasting 128 CPU cores, 256 GB of RAM, and a 50 Gbps network card, costing around $5,000 a month.

While this server easily handled 6 million RPS for a simple “Hello World” JSON response, real-world APIs don’t just return “Hello World”.

When testing a PATCH request that returns a 30 KB JSON payload, the throughput plummeted to just 100,000 RPS. Why? The server's CPU was only at 50% utilization, but it was moving almost 6 Gigabytes of data per second.

6 GB/s * 8 bits = 48 Gbps

We completely maxed out the server’s 50 Gbps network card.

Industry Insight: If you have an endpoint transferring heavy payloads (like user profiles or complex dashboard configurations), your bottleneck will not be the CPU; it will be your network bandwidth. Shrinking the payload to 1 KB immediately allowed the server to hit 3 million RPS.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Phase 3: The Database Dilemma (Disk I/O vs. Scale)

Handling requests is one thing; persisting data is another. We introduced an AWS Aurora PostgreSQL database (db.m5.16xlarge) with 64 CPU cores, costing another $5,000/month.

The Write Bottleneck

When attempting simple INSERT operations (database writes), the system capped at a pathetic 35,000 RPS. Even after paying an extra $1,000/month to upgrade the storage disk to 12,000 IOPS (Input/Output Operations Per Second), throughput only reached 66,000 RPS.

The Read Bottleneck & The Big O Mistake

When attempting to read from a database with 10 million records, a query using ORDER BY random() LIMIT 1 took a staggering 40 seconds to return. Why? Because random() requires the database to scan the entire table—an O(N) time complexity operation. At extreme scale, O(N) algorithms will instantly crash your system. Changing the query to do a direct index lookup based on the Max ID (O(1) time complexity) dropped query time to milliseconds and pushed reads to 400,000 RPS.

Yet, we are still far short of 1 million RPS. Vertically scaling the database to handle 1M RPS would cost upwards of $20,000 to $30,000 per month.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Phase 4: The In-Memory Revolution (Redis Clustering)

Disk storage (SSDs) is thousands of times slower than RAM. To hit 1 million writes per second without bankrupting the company, we must abandon disk-based writing in the critical path and move to Redis, an in-memory data store.

However, a single Redis instance is single-threaded and caps at around 100,000 to 150,000 RPS. The solution is Redis Clustering.

Architecture Diagram: Redis Cluster Setup

              [ Incoming Traffic: 1 Million RPS ]
                        |
                        v
              +---------------------------------------------------+
              | Node.js Cluster (128 Processes running via PM2)   |
              +---------------------------------------------------+
                        | (Hashes Request ID to find correct Node)
                        v
              +---------------------------------------------------+
              |                 REDIS CLUSTER                     |
              |                                                   |
              |  [Master 1]  [Master 2] ... [Master 15]           |
              |      |           |               |                |
              |  [Replica 1] [Replica 2]... [Replica 15]          |
              +---------------------------------------------------+
                        | (Asynchronous Background Sync)
                        v
              +---------------------------------------------------+
              |          PostgreSQL (Persistent Storage)          |
              +---------------------------------------------------+

By deploying 30 Redis instances (15 Masters, 15 Replicas) across our server’s RAM, the Node.js application can instantly hash incoming data and distribute it across the masters. With this cluster, we successfully hit 1,000,000 database writes per second while utilizing 100% of the server’s CPU.

Real-World Industry Example (Uber): Imagine Uber tracking millions of driver locations. If the app wrote GPS pings directly to a PostgreSQL database every second, the disk would instantly fail. Instead, high-scale apps save these real-time locations to an in-memory Redis queue. A background worker process then reads this queue and executes bulk inserts (batch processing) to the persistent database overnight or during off-peak hours.

The UUID Optimization: To prevent hitting a centralized database to auto-increment user IDs (which causes race conditions and network bottlenecks), we switched to generating 122-bit UUIDs in the application layer. Because of the math behind the Birthday Paradox, processing 1 million UUIDs per second would take 86,000 years to reach a 50% chance of an ID collision.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Phase 5: Pushing Beyond Language Limits (C++ & Drogon)

We solved the database problem, but we still wanted to hit 1 million RPS on our heavy PATCH route that moves a large JSON payload.

We provisioned AWS “Beast Servers” (C8GN.48xlarge) equipped with 192 CPU Cores and an astonishing 600 Gbps network capacity. But Node.js failed us. It maxed out at around 500,000 RPS. The architectural overhead of PM2 receiving traffic on a parent process and distributing it to 180 child processes was simply too heavy. Languages like Python, Node.js, and Java begin to show severe friction at this extreme scale.

We rewrote the application in C++ using the Drogon web framework and the RapidJSON parser. C++ inherently handles multi-threading without the massive parent-child process overhead of Node’s cluster module.

The Result: The C++ server comfortably achieved 1.2 Million RPS, utilizing only 70% of the CPU, while Node.js struggled at half that speed using 100% of the CPU.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

The Final Epic Test: 2 Billion Requests

To fully saturate the C++ server and prove its resilience, we launched 60 individual tester instances (c8gn.2xlarge) to bombard the main server simultaneously.

• Concurrency: 60 machines * 400 connections = 120,000 concurrent active connections at any given millisecond.

• The Run: 30 continuous minutes.

• The Output: The C++ Beast Server successfully processed 2 Billion Requests and moved over 60 Terabytes of data. Out of 2 billion requests, there were only 40 timeouts.

An Expensive Lesson in AWS Load Balancers

In a real-world scenario, you wouldn’t use one supercomputer; you would put a Load Balancer in front of hundreds of smaller servers. However, when we attempted to use an AWS Network Load Balancer, performance degraded completely, capping at 5 Gbps. AWS Support confirmed that default Load Balancers have hard limits. To push 600 Gbps of traffic through a load balancer, you must contact AWS in advance to “pre-warm” and reserve load balancer capacity (LCUs).

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Conclusion

Handling 1 million requests per second requires a fundamental shift in how you view computing.

1. Code Efficiency Trumps Hardware: An O(N) algorithm will kill your system regardless of how big your server is. Dropping Express.js for Fastify/C++ can save you millions in compute costs.

2. Network is the hidden killer: Before your CPU maxes out, large JSON payloads will bottleneck your network interface cards. Use compression.

3. RAM > Disk: Disk I/O is too slow for extreme scale. Use Redis clusters to accept writes, and sync to persistent databases asynchronously.

4. Serverless is Expensive at Scale: Paying per-request for serverless infrastructure (like API Gateways or Cloudflare Workers) at 1 Million RPS would cost roughly $1 to $4 million per month. At this scale, managing your own bare-metal/EC2 infrastructure is a financial necessity.

The scale of modern titans like Uber and AWS is terrifying, complex, and incredibly fun to engineer. It forces you to stop being just a “programmer” and start thinking like a true systems engineer !!