Rate Limiting Using Token Bucket Algorithm with Java & Spring Boot

Implementing production-grade request throttling for enterprise Spring Boot microservices.

Introduction

While working at Government tech platform, one of the backend engineering challenges we faced was controlling excessive API traffic from individual users.

The platform handled requests from:

• Students
• Institutions
• Government workflows
• External integrations

Without request throttling:

• APIs could receive excessive traffic from individual users
• Backend resources could become overloaded
• Database connections and thread pools could get exhausted
• One user could negatively impact overall platform stability

To solve this, I implemented API rate limiting using the Token Bucket Algorithm in Java & Spring Boot.

The implementation created a virtual bucket for every user:

• Every request consumed one token
• Tokens were replenished gradually over time
• If the bucket became empty, the API returned:

HTTP 429 Too Many Requests

The goal was not just request blocking.

The real objective was:

• Backend protection
• Fair resource allocation
• Traffic smoothing
• System resiliency

High-level architecture of API rate limiting using Token Bucket algorithm in Spring Boot microservices.

Why Rate Limiting Matters

In distributed backend systems, APIs must remain stable under high traffic.

Without proper throttling:

• APIs become vulnerable to abuse
• Sudden traffic spikes impact latency
• Databases receive excessive concurrent requests
• Thread pools become saturated

Rate limiting acts as a defensive layer between clients and backend services.

It improves:

• System stability
• Fair resource allocation
• Predictable performance
• Infrastructure protection

Why We Chose Token Bucket Algorithm

Initially, multiple approaches were evaluated:

• Fixed Window
• Sliding Window
• Leaky Bucket
• Token Bucket

The Token Bucket algorithm was selected because it provides:

• Burst traffic support
• Lightweight computation
• O(1) token validation
• Better traffic smoothing
• Predictable request control

Why We Did Not Use Existing Libraries

Libraries such as:

• Bucket4j
• Resilience4j

already provide rate limiting capabilities.

However, in our case, we implemented a custom solution because:

• lightweight internal requirements
• custom bucket behavior
• complete control over token lifecycle
• easier observability and debugging
• flexibility for future Redis-based distributed throttling

Building the implementation internally also helped us optimize the request lifecycle specifically for our Spring Boot microservice architecture.

Why We Did Not Use Fixed Window Rate Limiting

Initially, a fixed-window approach was considered.

Example:

• 100 requests per minute

However, fixed windows introduce burst traffic issues.

A user could:

• Send 100 requests at 12:00:59
• Then another 100 requests at 12:01:01

This creates sudden traffic spikes.

The Token Bucket algorithm solved this problem by:

• Allowing controlled bursts
• Smoothly throttling traffic
• Enforcing long-term request limits

This made it more suitable for enterprise API traffic patterns.

High-Level Architecture

Request lifecycle:

Client Request ↓ Spring Boot Filter ↓ User Bucket Lookup ↓ Token Validation ↓ Allow Request OR Return HTTP 429 ↓ Controller Layer

The rate limiter was implemented before the request reached the controller layer.

This ensured:

• Centralized request validation
• Minimal business logic changes
• Better performance
• Consistent throttling policies

Sequence Flow

Client ↓ Rate Limiting Filter ↓ Bucket Service ↓ Token Bucket Engine ↓ Allow / Reject Request ↓ Controller Layer

This layered approach improved maintainability and separation of concerns.

Core Architecture Components

The implementation consisted of four layers:

1. Rate Limiting Filter
2. Bucket Service
3. Token Bucket Engine
4. Token Refill Logic

1. Rate Limiting Filter

The filter intercepted every incoming request before business logic execution.

Responsibilities:

• Identify user
• Fetch user bucket
• Validate token availability
• Allow or reject request

Implementation:

@Component
public class RateLimitingFilter
        extends OncePerRequestFilter {
    @Autowired
    private BucketService bucketService;
    @Override
    protected void doFilterInternal(
            HttpServletRequest request,
            HttpServletResponse response,
            FilterChain filterChain)
            throws ServletException, IOException {
        String userId =
                request.getHeader("X-USER-ID");
        TokenBucket bucket =
                bucketService.getBucket(userId);
        boolean allowed =
                bucket.tryConsume();
        if (allowed) {
            filterChain.doFilter(
                    request,
                    response
            );
        } else {
            response.setStatus(429);
            response.getWriter().write(
                    "Too Many Requests"
            );
        }
    }
}

Why We Used Filter Instead of Controller

The rate limiter was implemented using Spring filters instead of controllers because:

• centralized request enforcement
• request rejection before business processing
• better performance
• reusable throttling logic
• reduced duplicate validation

This ensured all APIs followed consistent throttling policies.

2. Bucket Service Layer

Each authenticated user had an independent virtual bucket.

The bucket stored:

• Maximum capacity
• Available tokens
• Last refill timestamp

Implementation:

public class TokenBucket {
private final int capacity;
    private final int refillRate;
    private int availableTokens;
    private long lastRefillTimestamp;
    public TokenBucket(
            int capacity,
            int refillRate) {
        this.capacity = capacity;
        this.refillRate = refillRate;
        this.availableTokens = capacity;
        this.lastRefillTimestamp =
                System.currentTimeMillis();
    }
}

This ensured:

• User-level isolation
• Controlled request consumption
• Fair API usage

3. Token Consumption Logic

Every request consumed one token.

Flow:

1. Refill tokens
2. Validate token availability
3. Consume token
4. Reject if no tokens remain

Implementation:

public synchronized boolean tryConsume() {
refill();
    if (availableTokens > 0) {
        availableTokens--;
        return true;
    }
    return false;
}

The method was synchronized to prevent race conditions during concurrent requests.

4. Token Refill Logic

Tokens were replenished automatically over time.

Instead of using background schedulers, refill calculations were performed lazily during request execution.

Implementation:

private void refill() {
long currentTime =
            System.currentTimeMillis();
    long elapsedTime =
            currentTime - lastRefillTimestamp;
    int tokensToAdd =
            (int) (elapsedTime / 1000)
                    * refillRate;
    if (tokensToAdd > 0) {
        availableTokens =
                Math.min(
                        capacity,
                        availableTokens + tokensToAdd
                );
        lastRefillTimestamp =
                currentTime;
    }
}

Benefits:

• No scheduler overhead
• Better scalability
• Lower CPU utilization

Refill calculations were performed lazily during request execution

Configuration Example

Example configuration:

rate-limit:
  capacity: 10
  refill-rate: 2

Where:

• capacity → maximum tokens allowed
• refill-rate → tokens added per second

This made the implementation configurable across environments.

Example Request Flow

Example API request:

GET /api/user/profile
X-USER-ID: 1024

Suppose the user bucket state is:

PropertyValueCapacity10Available Tokens5Refill Rate2 tokens/sec

Request processing flow:

1. Request reaches Rate Limiting Filter
2. User bucket is fetched
3. Token availability is checked
4. One token is consumed
5. Remaining tokens become 4
6. Request is forwarded to controller

If available tokens become 0:

• request is rejected
• HTTP 429 is returned

This ensures controlled API consumption while protecting backend services.

Bucket Service Implementation

The bucket service handled:

• Bucket creation
• Bucket retrieval
• User-based isolation

Implementation:

@Service
public class BucketService {
private final ConcurrentHashMap<
            String,
            TokenBucket> buckets =
            new ConcurrentHashMap<>();
    public TokenBucket getBucket(
            String userId) {
        return buckets.computeIfAbsent(
                userId,
                id -> new TokenBucket(10, 2)
        );
    }
}

Using ConcurrentHashMap provided:

• O(1) lookup
• Thread-safe access
• Lightweight memory operations

Request Lifecycle

Complete request lifecycle:

Request Received ↓ User Identification ↓ Bucket Lookup ↓ Token Refill Calculation ↓ Token Consumption ↓ Allow OR Reject Request ↓ Controller Execution

HTTP 429 Handling

If tokens became exhausted:

HTTP 429 Too Many Requests

The request was rejected before reaching the controller layer.

This protected:

• APIs
• Thread pools
• Database connections
• Downstream services

Standard Error Response

Example rejection response:

{
  "timestamp": "2026-05-29T10:15:30",
  "status": 429,
  "error": "Too Many Requests",
  "message": "Rate limit exceeded",
  "path": "/api/user/profile"
}

Using standardized error responses improved:

• client-side debugging
• API consistency
• retry handling
• observability

Request Lifecycle Diagram

Production Challenge: Concurrent Requests

One major challenge was handling multiple simultaneous requests from the same user.

Without synchronization:

• Multiple threads could consume the same token
• Rate limits could become inconsistent

To solve this:

• synchronized token consumption was implemented
• bucket operations became atomic

This ensured consistent throttling behavior.

Production Challenge: Memory Cleanup

Since buckets were maintained per user:

• inactive users could increase memory usage over time

To solve this:

• idle bucket cleanup strategies were introduced
• expiration logic could remove inactive buckets periodically

This improved long-term scalability.

In-Memory Architecture

Initially, buckets were maintained using:

ConcurrentHashMap<String, TokenBucket>

Advantages:

• Fast bucket lookup
• Lightweight implementation
• Minimal infrastructure dependency

This worked efficiently for:

• Single-node deployments
• Moderate traffic systems

Distributed System Challenge

In multi-instance deployments:

Problem:

• Each application instance maintains separate memory
• Token states become inconsistent across nodes

Example:

• Request 1 hits Server A
• Request 2 hits Server B
• Separate bucket states exist

This can unintentionally bypass rate limiting.

Distributed Architecture Using Redis

For distributed environments, Redis-based bucket storage is preferred.

Architecture:

Client ↓ Spring Boot Service ↓ Redis Shared Bucket Store ↓ Token Validation ↓ Allow / Reject Request

Benefits:

• Shared token state
• Cross-instance consistency
• Atomic operations
• Horizontal scalability

Distributed rate limiting architecture using Redis shared token bucket storage.

Performance Characteristics

Complexity:

OperationComplexityBucket LookupO(1)Token ValidationO(1)Token RefillO(1)

The implementation introduced minimal request overhead and supported high-throughput traffic efficiently.

Monitoring & Observability

In production environments, monitoring rate limiting behavior became extremely important.

We monitored:

• Request rejection rate
• Token exhaustion frequency
• Most throttled APIs
• High-frequency users
• Average request latency
• Token refill patterns

This helped tune:

• bucket capacity
• refill rates
• throttling strategies

The metrics were integrated into centralized observability systems using:

• Micrometer
• Prometheus
• Grafana

This provided better operational visibility into API traffic behavior.

Security & Reliability Benefits

The implementation improved:

• Backend resiliency
• API stability
• Fair resource allocation
• Infrastructure protection

It also reduced:

• Excessive traffic spikes
• Resource exhaustion
• Uncontrolled API abuse

Architecture Evolution

The implementation evolved through multiple stages.

Phase 1

Initial implementation:

• Single-instance deployment
• In-memory token buckets
• ConcurrentHashMap storage

Phase 2

Improved concurrency handling:

• synchronized token consumption
• atomic bucket updates
• thread-safe request validation

Phase 3

Distributed architecture planning:

• Redis shared bucket storage
• cross-instance token consistency
• distributed throttling support

Phase 4

Future enterprise improvements:

• API Gateway-based throttling
• Kubernetes ingress rate limiting
• adaptive throttling policies
• dynamic user-based limits

This gradual evolution helped scale the platform reliably as traffic increased.

Future Improvements

Possible enterprise-scale improvements:

• Redis distributed throttling
• API Gateway rate limiting
• Spring Cloud Gateway integration
• Dynamic user-based throttling
• Monitoring dashboards
• Adaptive traffic shaping

Why This Architecture Worked Well

The implementation worked well because it was:

• Lightweight
• Easy to integrate
• Computationally efficient
• Scalable
• Framework-independent

Most importantly, the rate limiter operated before business logic execution, which reduced unnecessary backend processing and protected downstream services early in the request lifecycle.

Final Thoughts

Rate limiting is one of the most important resiliency patterns in scalable backend systems.

By implementing the Token Bucket algorithm using Java & Spring Boot, we were able to:

• Protect backend services
• Smooth traffic spikes
• Enforce fair API usage
• Improve platform stability
• Prevent infrastructure overload

The implementation remained lightweight while still supporting enterprise scalability patterns such as distributed throttling and shared token management.

In distributed systems, protecting infrastructure is just as important as building features.

Rate limiting is not merely traffic control — it is a foundational resiliency pattern for scalable backend architectures.

Engineering Insight

In distributed systems, the hardest part of rate limiting is not rejecting requests — it is maintaining consistent token state across multiple application instances.

This becomes especially important in horizontally scaled microservice architectures where requests can hit different servers on every call.

Distributed token storage using Redis helps solve this problem efficiently through shared state management and atomic operations.