Blog Thumbnail

Rate Limiting in System Design: The Invisible Guardrail That Keeps Your APIs Alive

Most applications work perfectly… until they don’t.

A product launches. Traffic jumps overnight. A bot starts sending thousands of requests every second. One customer accidentally creates an infinite retry loop. Another client ignores the API documentation and floods your servers.

Without any protection in place, even a well-designed backend begins to struggle.

This is exactly why rate limiting exists.

It isn’t just another interview topic in System Design. It is one of the first defensive mechanisms used by production systems to protect infrastructure, maintain fairness, and ensure a consistent experience for every user.

If you’re preparing for system design interviews or building APIs that might serve thousands — or millions — of requests, understanding rate limiting is essential.

What is Rate Limiting?

Rate limiting is a mechanism that restricts how many requests a client can make within a specific period of time.

Instead of allowing unlimited access, the server enforces predefined limits.

For example:

• 100 requests per minute
• 5 login attempts every 10 minutes
• 1,000 API calls per hour
• 10 file uploads each day

Once the limit is reached, the server rejects additional requests until the limit resets.

A typical response looks like this:

HTTP/1.1 429 Too Many Requests

Retry-After: 60

The 429 Too Many Requests status code tells clients that the request itself is valid, but the request frequency is not.

Why Do We Need Rate Limiting?

Imagine opening your application to the public.

Everything works.

Then one user starts making 50,000 requests every minute.

Should everyone else suffer because of one client?

Probably not.

Rate limiting solves several real-world problems at the same time.

Prevents Abuse

Public APIs often become targets for automated scripts.

Without restrictions, bots can continuously send requests and consume expensive server resources.

Rate limiting slows them down.

Protects Infrastructure

Every request costs something.

CPU cycles.

Memory.

Database queries.

Network bandwidth.

Even a small spike can become expensive if requests continue unchecked.

Ensures Fair Usage

Large platforms serve millions of users simultaneously.

If a handful of clients monopolize the system, everyone else experiences slower responses.

Rate limiting distributes resources more fairly.

Reduces Security Risks

Many attacks rely on repeated requests.

Examples include:

• Password brute-force attacks
• OTP guessing
• Credential stuffing
• API scraping

Limiting request frequency makes these attacks significantly harder.

Where is Rate Limiting Used?

You’ll find it almost everywhere.

Some common examples include:

Common Examples

Social media platforms, banking applications, cloud providers, and payment gateways all rely heavily on rate limiting.

Where Should Rate Limiting Be Applied?

There isn’t a single correct location.

Different architectures enforce limits at different layers.

API Gateway

This is the most common choice.

Every incoming request passes through the gateway before reaching backend services.

Client
   │
   ▼
API Gateway
   │
   ▼
Microservices

Advantages:

• Centralized enforcement
• Easier monitoring
• Consistent policies

Reverse Proxy

Tools like NGINX or Envoy can reject excessive traffic before it reaches the application.

Internet
    │
    ▼
NGINX
    │
    ▼
Backend

This reduces unnecessary load.

Application Layer

Sometimes limits depend on business rules.

For example:

• Premium users get 10,000 requests/day.
• Free users receive only 1,000 requests/day.

The application itself understands these rules better than infrastructure components.

Popular Rate Limiting Algorithms

Different situations require different strategies.

There isn’t a universal winner.

1. Fixed Window Counter

The simplest approach.

Suppose the limit is:

100 requests per minute

A counter starts at the beginning of every minute.

12:00 - Counter = 0

User sends requests...

Counter = 100

Next request → Rejected

12:01

Counter resets to 0

Advantages

• Easy to implement
• Minimal memory usage
• Fast lookups

Drawback

A user can exploit the window boundary.

Example:

100 requests at 12:00:59

100 requests at 12:01:01

Nearly 200 requests are processed within two seconds.

2. Sliding Window Log

Instead of resetting counters, the server stores timestamps for every request.

When a new request arrives:

• Remove old timestamps
• Count recent requests
• Decide whether to allow or reject

Example:

Current Time

12:10:30

Stored Requests

12:10:02
12:10:08
12:10:15
12:10:22
12:10:28

Only timestamps inside the configured window are considered.

Advantages

• Accurate
• Fair
• No boundary issue

Drawback

Storing every timestamp increases memory usage.

3. Sliding Window Counter

This combines Fixed Window and Sliding Window Log.

Instead of storing every request, it estimates traffic using adjacent windows.

It delivers better accuracy without excessive memory consumption.

Many production systems prefer this balance.

4. Leaky Bucket

Imagine pouring water into a bucket.

Water enters at varying speeds.

The bucket leaks at a constant rate.

Incoming Requests

██████████

      │

      ▼

+-----------+
|  Bucket   |
+-----------+

      │

Constant Output

If the bucket becomes full, new requests are discarded.

Benefits

• Smooth traffic
• Predictable request flow
• Prevents sudden bursts

5. Token Bucket

This is probably the most widely used algorithm today.

The server continuously generates tokens.

Each request consumes one token.

If tokens are available:

Allow Request

Otherwise:

Reject Request

Example:

Bucket Capacity = 10 Tokens

Refill Rate = 2 Tokens/Second

Suppose the bucket contains ten tokens.

A user suddenly sends eight requests.

Eight tokens disappear immediately.

Two remain.

After one second:

+2 Tokens

The bucket starts filling again.

Why is it popular?

Because occasional bursts are allowed without sacrificing long-term control.

Cloud providers frequently use this approach.

Comparing the Algorithms

Compare Algorithm

Distributed Rate Limiting

Things become interesting when multiple servers are involved.

Suppose you have five API servers.

Load Balancer

      │

 ┌────┼────┐

Server A
Server B
Server C
Server D
Server E

A user sends requests.

Some reach Server A.

Others go to Server C.

The rest land on Server E.

If every server maintains its own counter, each one thinks the user is below the limit.

Collectively, the user bypasses the restriction.

That’s a problem.

Using Redis for Distributed Rate Limiting

A shared datastore solves this.

Redis is a common choice because it supports:

• Extremely fast reads
• Atomic operations
• Automatic expiration
• High throughput

Architecture:

               Client

                  │

                  ▼

           Load Balancer

                  │

     ┌────────────┼────────────┐

     ▼            ▼            ▼

 Service A    Service B    Service C

      \           |           /

       \          |          /

             Redis

Every server updates the same counter.

No matter which backend processes the request, the limit stays consistent.

Common Rate Limiting Keys

Applications don’t always limit by IP address.

Different identifiers may be more appropriate.

Examples include:

User ID

API Key

Session ID

IP Address

Organization ID

JWT Subject

OAuth Client ID

The right choice depends on the product and authentication model.

Challenges in Rate Limiting

The concept sounds straightforward.

Implementing it at scale isn’t.

Some common challenges include:

• Distributed counters
• Clock synchronization
• Regional deployments
• Memory optimization
• False positives
• High Redis traffic
• Dynamic user plans
• Retry storms

These issues usually appear only after the system starts handling large volumes of traffic.

Best Practices

A few practical guidelines make rate limiting more effective.

• Return meaningful error messages.
• Include retry information.
• Monitor rejected requests.
• Separate limits for authenticated and anonymous users.
• Use different limits for different API endpoints.
• Avoid identical limits for free and premium customers.
• Log excessive traffic for security analysis.
• Choose algorithms based on traffic patterns rather than popularity.

Interview Questions You Should Expect

During a System Design interview, discussions often move beyond the basic definition.

Some common questions include:

• Which algorithm would you choose for a payment API?
• How would you implement distributed rate limiting?
• Why is Redis commonly used?
• How do you prevent users from bypassing limits?
• Should rate limiting happen before or after authentication?
• How would you handle millions of requests per second?
• How would you rate limit GraphQL APIs?
• What happens if Redis becomes unavailable?

Being able to explain the trade-offs matters more than memorizing algorithm names.

Final Thoughts

Rate limiting rarely receives the same attention as databases, caching, or load balancing. Yet, it quietly protects systems every second.

A thoughtfully designed rate limiter improves reliability, reduces operational costs, limits abuse, and keeps services responsive under pressure.

From a System Design interview perspective, understanding the available algorithms is only part of the story. The more valuable skill is recognizing when to use each approach and how to make it work in a distributed environment.

As systems grow, rate limiting shifts from being a useful feature to becoming a foundational requirement. Ignore it for too long, and even a well-architected application can struggle under traffic it was never meant to handle.