Designing Scalable APIs: From 100 to 1 Million Requests

When an API is first built, it usually works well with a small number of users. Requests are fast, responses are clean, and everything feels stable.

However, as usage grows, the same API often starts slowing down. Response times increase, CPU usage spikes, and in some cases, the system becomes unresponsive.

Scaling an API is not just about adding more servers. It is about designing the system in a way that can handle increasing load efficiently.

This article focuses on practical concepts that help APIs scale from handling a few hundred requests to millions.

Understanding Where Things Break

Before scaling, it is important to understand why systems fail under load.

Some common bottlenecks include:

• CPU-heavy operations
• Slow database queries
• Too many concurrent requests
• Blocking operations in the application

In many Node.js applications, one of the biggest issues is not infrastructure — it is how the code interacts with the event loop.

The Event Loop: More Practical Than Theoretical

Most developers have heard that Node.js is “single-threaded” and uses an event loop. However, this knowledge is often limited to interview preparation.

In real applications, the event loop is the core reason why an API either scales well or fails under pressure.

The event loop is responsible for handling all incoming requests. If it gets blocked, the entire application becomes slow.

What Blocks the Event Loop?

• Heavy computations (large loops, data processing)
• Synchronous file operations
• Long-running tasks executed directly in request handlers

A simple example:

// Bad: blocks event loop
app.get('/process', (req, res) => {
  let sum = 0;
  for (let i = 0; i < 1e9; i++) {
    sum += i;
  }
  res.send("Done");
});

While this loop runs, no other request can be processed.

The Rule

A scalable Node.js API follows one simple rule:

Never block the event loop.

Using All CPU Cores (Node.js Limitation)

By default, Node.js runs on a single CPU core. Even if a machine has multiple cores, only one is used.

This becomes a major limitation under high load.

Solution: Cluster / Fork

Node.js provides a way to create multiple instances of the same server using the cluster module.

Each instance runs on a different CPU core.

const cluster = require('cluster');
const os = require('os');

if (cluster.isMaster) {
  const numCPUs = os.cpus().length;
  for (let i = 0; i < numCPUs; i++) {
    cluster.fork();
  }
} else {
  // start server
}

This allows the application to handle more requests in parallel.

Worker Threads vs Child Processes

Many developers are aware of these terms but are unsure when to use them.

Worker Threads

• Used for CPU-heavy tasks
• Runs in the same process but different thread
• Shares memory

Best for:

• Image processing
• Data transformations
• Complex calculations

Child Processes

• Runs a separate process
• Completely isolated
• Useful for running external scripts or services

Best for:

• Running shell commands
• Executing separate services

When Not to Use Them

If the task is already asynchronous (like database calls or API requests), there is usually no need for worker threads or child processes.

Horizontal Scaling

Scaling is not limited to a single machine.

To handle large traffic:

• Run multiple instances of the API
• Use a load balancer (NGINX, cloud load balancer)
• Keep APIs stateless

Stateless design ensures any request can be handled by any server.

Caching: Reducing Unnecessary Work

One of the easiest ways to scale is to avoid doing the same work repeatedly.

Caching helps reduce:

• Database load
• Response time

Common approaches:

• In-memory cache
• Redis
• CDN (for static content)

Database Optimization

Even a well-designed API can fail if the database is slow.

Important practices:

• Use indexes properly
• Avoid unnecessary queries
• Use pagination instead of loading large datasets
• Use connection pooling

Rate Limiting

If too many requests hit the API at once, even a good system can crash.

Rate limiting helps:

• Prevent abuse
• Protect resources
• Maintain stability

Monitoring and Observability

Scaling without visibility is risky.

It is important to track:

• Response times
• Error rates
• CPU and memory usage

Tools like logging systems and monitoring platforms help identify issues before they become critical.

Final Thoughts

Scaling an API is not achieved through a single change. It is the result of multiple small decisions:

• Writing non-blocking code
• Using system resources efficiently
• Distributing load properly
• Monitoring performance continuously

Many performance issues are not caused by infrastructure limitations, but by how the application is written.

Understanding concepts like the event loop, worker threads, and process management in a practical way makes a significant difference when building scalable systems.