The Sliding Log Rate Limiter — Accurate, Expensive, and When It’s Worth It

Fintech needed exact “10 auths per 60 seconds” — token bucket wasn’t enough. The sliding log algorithm + Redis Lua atomic version.

Photo by David Valentine on Unsplash

Here’s a scenario where standard rate limiting isn’t enough.

A fintech company processes payment authorizations for merchants. Their policy: no merchant can attempt more than 10 authorizations per minute on cards flagged as high-risk. The rule exists for regulatory reasons — regulators specifically care that the limit is 10 in any 60-second window, not “10 per minute” with fuzzy boundaries.

The team’s first implementation uses a fixed-window counter — count requests per calendar minute, reset at the top of each minute. Simple, fast, standard. Then compliance points out the problem: a merchant could make 10 authorizations at 12:00:59, then 10 more at 12:01:00. That’s 20 authorizations in under a second, but the counter shows 10 for each minute-bucket. The regulator would say the policy was violated. The compliance team asks for something more accurate.

The team upgrades to a sliding window counter — a well-known algorithm that weighs the previous window and current window based on how far into the current window you are. Better. But still approximate. In some traffic patterns, it can allow 12–13 requests in a true 60-second span. Compliance says “close, but not exact.” They want exact.

The team implements a sliding log rate limiter. Now the count is provably accurate — the limit of 10-per-60-seconds is enforced to the millisecond. Compliance is satisfied. Then the ops team notices Redis memory has jumped by 40%.

This article is about that trade-off. The sliding log rate limiter is the most accurate rate limiting algorithm available, and also the most expensive. Sometimes the accuracy is worth the cost. Sometimes it isn’t. Knowing when to reach for it — and when a cheaper approximation is fine — is a judgment call that comes up more often than you’d expect. Verified throughout against Redis 7 and PHP 8.3.6.

TL;DR Speedrun

• Rate limiting algorithms trade accuracy for efficiency. Fixed window is fastest but has boundary bursts. Sliding window is a good approximation. Token bucket handles bursts naturally but doesn’t enforce exact windows. Sliding log stores every request timestamp and gives exact enforcement — at the cost of storing every timestamp.
• The sliding log stores individual request timestamps in a sorted structure (Redis sorted set). On each request, prune timestamps older than the window, count what remains, allow if under the limit, add the current timestamp if allowed. Every operation is exact.
• Verified memory cost: with Redis 7, 100 timestamps per user occupies about 3 KB; 1,000 timestamps takes 100 KB; 5,000 timestamps takes 587 KB. At scale (100k users × 100 requests per window), that’s about 500 MB for the rate limiter alone. Compare to token bucket at roughly 3 MB for the same user count — about 150× more memory.
• Correctness is the strong argument. No boundary bursts. No approximations. If you say “10 requests per 60 seconds,” the algorithm enforces exactly that. For regulated systems, audit-critical operations, and anti-abuse where false negatives are expensive, the accuracy is worth the memory cost.
• The atomic version uses a Lua script. Without atomicity, a race condition can let concurrent requests both pass the count check before either has added its timestamp. The Lua script executes prune, count, and add in a single atomic operation. Verified working against Redis 7.
• Practical use cases: authentication attempts (regulatory compliance), payment endpoints (fraud prevention), premium API tiers (small user counts, high accuracy needs), any operation where “close enough” isn’t enough. Not for typical public APIs with millions of users hitting normal endpoints.
• The hybrid pattern: use sliding log for critical operations, cheaper algorithms (token bucket, sliding window counter) for everything else. Most applications don’t need exact accuracy for every endpoint. Reserve the expensive algorithm for the cases that actually benefit.

What You’ll Learn

• How sliding log works, with verified Redis code
• The atomic Lua script version and why it matters
• Real memory measurements (not theoretical estimates)
• When to use it vs cheaper alternatives
• How to integrate it with Symfony’s rate limiter component

A Quick Rate Limiting Primer

If you already know the common rate limiting algorithms, skip ahead. Otherwise, here’s the compact version.

Rate limiting means restricting how often a client can perform an action — API requests, login attempts, message sends. The goal is preventing abuse, managing capacity, or enforcing tier limits.

The typical shape: “no more than N actions per T time.” Every algorithm answers this differently.

Fixed window counter — count requests per calendar-aligned window (each minute, each hour). Reset at window boundary. Simple, fast, one integer per user. Problem: a burst at the end of one window plus a burst at the start of the next allows nearly 2× the intended rate in a short span.

Sliding window counter — weigh the previous window and current window based on how far through the current window you are. If you’re 30 seconds into the current minute and the limit is 10/minute, the effective count is current + 0.5 × previous. Approximation, but a good one. Two integers per user.

Token bucket — each user has a “bucket” that refills at a fixed rate up to a max capacity. Each request consumes one token. When the bucket is empty, requests are denied. Naturally handles bursts. About 32 bytes per user (last-refill timestamp + current tokens).

Leaky bucket — like token bucket but requests drip out at a fixed rate. Smooths traffic rather than allowing bursts.

Sliding log — store the exact timestamp of every request. On each new request, remove timestamps older than the window and count what remains. If under the limit, add the new timestamp; otherwise deny. Exact. But storage is O(N) per user where N is requests-per-window.

The four counter-based algorithms are approximations that trade some accuracy for constant-space storage. Sliding log gives up that constant space to be exact.

What the Sliding Log Actually Stores

The mental picture is straightforward. For each user (or IP, or API key, whatever the rate limiting scope is), maintain a sorted collection of timestamps. Every request goes in. Every request older than the window comes out.

In Redis, sorted sets are the natural data structure. Redis sorted sets (ZSET) store members with numeric scores, sorted by score. Use the timestamp as both the score and (with some uniqueness suffix) the member.

The core operations are three Redis commands:

• ZREMRANGEBYSCORE key -inf (now - window) — remove entries with score older than window
• ZCARD key — count remaining entries
• ZADD key now (now-with-unique-suffix) — add current timestamp

Here’s a naive PHP implementation (not atomic — we’ll fix that later):

<?php
declare(strict_types=1);
function checkRateLimit(
    Redis $redis, 
    string $userId, 
    int $limit, 
    int $windowSeconds
): array {
    $key = "ratelimit:{$userId}";
    $now = microtime(true);
    $windowStart = $now - $windowSeconds;

    // 1. Remove entries older than the window
    $redis->zRemRangeByScore($key, '-inf', (string)$windowStart);

    // 2. Count entries currently in the window
    $count = $redis->zCard($key);

    // 3. Check if we're under the limit
    if ($count >= $limit) {
        return [
            'allowed' => false, 
            'count'   => $count, 
            'limit'   => $limit,
        ];
    }

    // 4. Add the current timestamp
    $redis->zAdd($key, $now, (string)$now);

    // 5. Set TTL so the key expires when unused
    $redis->expire($key, $windowSeconds + 5);

    return [
        'allowed' => true, 
        'count'   => $count + 1, 
        'limit'   => $limit,
    ];
}

Walk through each step:

• Line 8: build a namespaced Redis key. In production, include additional context like endpoint name or bucket type.
• Line 9: get the current time as a float (microtime(true) gives sub-second precision, which matters for high-throughput scenarios).
• Line 10: compute the earliest timestamp still within the window.
• Line 13: zRemRangeByScore removes members whose scores fall in the given range. -inf to windowStart means "everything older than the window."
• Line 16: zCard returns the count of members in the sorted set — the current request rate.
• Line 19–23: if we’re at the limit, refuse and return the current state.
• Line 26: add the current timestamp. Note the member string includes the timestamp value — sorted sets require unique members, so members with the same score need distinct values. Using the timestamp itself works if timestamps are unique; for higher throughput you’d add a random suffix.
• Line 29: set an expiration on the entire key. If the user stops making requests, the key naturally disappears after the window plus a small buffer.

Verified test — 5 requests with a limit of 3 per second:

Request 1: ALLOWED (count: 1/3)
Request 2: ALLOWED (count: 2/3)
Request 3: ALLOWED (count: 3/3)
Request 4: DENIED (count: 3/3)
Request 5: DENIED (count: 3/3)

After waiting 1.1 seconds:

Request 1: ALLOWED (count: 1/3)  ← counter fully reset
Request 2: ALLOWED (count: 2/3)
Request 3: ALLOWED (count: 3/3)

The Race Condition (And Why Atomicity Matters)

The naive implementation has a subtle bug. Consider two requests arriving simultaneously from the same user, hitting different PHP processes:

• Process A at t=0: runs zRemRangeByScore and zCard → gets count = 2 (below limit of 3)
• Process B at t=0: runs zRemRangeByScore and zCard → also gets count = 2 (before A has added)
• Process A: adds its timestamp → count becomes 3
• Process B: adds its timestamp → count becomes 4

Now there are 4 timestamps in the window, but the limit is 3. Both requests were allowed because both saw count=2 before either had a chance to write.

This isn’t a hypothetical — under load, race conditions between the read and write phases are common. Under high load with many concurrent requests, the effective limit can be significantly higher than advertised.

The fix is to make the whole check-and-add operation atomic. Redis supports this via Lua scripts — Redis executes the entire script as a single atomic operation, so no other commands can run in between.

<?php
$luaScript = <<<'LUA'
local key = KEYS[1]
local now = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local limit = tonumber(ARGV[3])
-- Remove entries older than the window
redis.call('ZREMRANGEBYSCORE', key, '-inf', now - window)
-- Count current entries
local count = redis.call('ZCARD', key)
if count >= limit then
    return {0, count, limit}
end
-- Add current timestamp (with random suffix for uniqueness)
redis.call('ZADD', key, now, now .. '-' .. math.random(100000))
redis.call('EXPIRE', key, window + 5)
return {1, count + 1, limit}
LUA;
function checkRateLimitAtomic(
    Redis $redis, 
    string $userId, 
    int $limit, 
    int $windowSeconds,
    string $script
): array {
    $now = microtime(true);
    $result = $redis->eval(
        $script, 
        ["ratelimit:{$userId}", $now, $windowSeconds, $limit], 
        1  // number of keys (KEYS[1] = ratelimit:...)
    );

    return [
        'allowed' => (bool)$result[0],
        'count'   => (int)$result[1],
        'limit'   => (int)$result[2],
    ];
}

Walk through the Lua script:

• Line 2–5: parse arguments. Lua strings need to be converted to numbers with tonumber().
• Line 8: prune old entries — same operation as before, just inside the script.
• Line 11: count remaining entries.
• Line 13–15: if at or over limit, return {0, count, limit} (0 means denied).
• Line 18: add current timestamp with a random suffix. The suffix ensures unique members even when multiple requests arrive at the same microtime.
• Line 19: set expiration.
• Line 21: return {1, count+1, limit} (1 means allowed).

The critical property: all these Redis commands run atomically. No other client can interleave commands between them. The race condition is eliminated.

Verified test result:

Request 1: allowed=1 count=1 limit=3
Request 2: allowed=1 count=2 limit=3
Request 3: allowed=1 count=3 limit=3
Request 4: allowed=0 count=3 limit=3
Request 5: allowed=0 count=3 limit=3

Performance note: Lua scripts add a small overhead vs raw commands, but for rate limiting it’s negligible. The atomicity is worth the microseconds.

The Real Memory Cost

The classic complaint about sliding log is memory. Let’s verify with actual measurements. Verified with Redis 7.4 (via MEMORY USAGE):

Entries per user Memory used Per-entry cost 10 376 bytes 37.6 bytes 50 1,592 bytes 31.8 bytes 100 3,128 bytes 31.3 bytes 500 52,096 bytes 104.2 bytes 1,000 100,192 bytes 100.2 bytes 5,000 587,104 bytes 117.4 bytes

Notice the jump between 100 and 500 entries. Redis internally switches sorted set representation from a compact listpack (small sets) to a full skiplist (larger sets) at a configurable threshold. Small sets are more memory-efficient per entry; larger sets pay more per entry but scale better.

Let’s project to production:

100,000 active users × average 100 requests per window:

• Sliding log: 100k × 3 KB per user ≈ 300–500 MB (depending on distribution)
• Token bucket: 100k × 32 bytes ≈ 3.2 MB

That’s roughly 150× more memory for sliding log. For a small application, that’s fine — 500 MB of Redis is affordable. For a large public API, it’s significant. For a system with millions of active users, it’s prohibitive.

The other cost is per-request CPU. Sorted set operations are O(log N) where N is set size. In practice, this means a few microseconds per operation even for large sets. Not a bottleneck at typical rate limiting throughput, but worth knowing.

When It’s Worth It

The sliding log’s exact accuracy is a real advantage in specific scenarios:

Regulatory compliance where “close” isn’t good enough. Payment authorization limits, KYC verification attempts, PII access rate limits — situations where auditors or regulators specifically require exact enforcement of a window-based rule. Approximation may be technically non-compliant.

Anti-abuse where false positives are cheap and false negatives are expensive. For login attempt limits, letting one extra brute-force attempt through per window is genuinely worse than being slightly slow. The accuracy protects against the specific attack of timing bursts around window boundaries.

Small user counts with high per-user precision. Internal tools, B2B APIs with dozens of clients, admin operations. Memory cost is per-user; with 100 users, even generous per-user allocations are trivially small.

Payment or financial operations. Rate limiting on payment endpoints often has both regulatory and fraud-prevention reasons for exact enforcement. Memory cost is usually acceptable given payment volumes are modest per user.

Fair-use enforcement on premium tiers. If your paid customers have “1000 requests per hour” as part of their contract, they expect exact accounting. An approximation that occasionally allows 1010 or 990 might be fine internally but can create billing disputes.

Audit trail requirements. If you need to prove “user X made exactly N requests in the last hour,” sliding log naturally provides that data. Other algorithms give you counts but not individual timestamps.

When Cheaper Algorithms Are Better

For the majority of rate limiting needs, an approximation is fine. Reach for sliding window counter or token bucket when:

High-scale public APIs. Millions of users, memory efficiency matters. Sliding window counter provides ~99% accuracy for typical traffic patterns at a fraction of the cost.

General API rate limiting. Standard “100 requests per minute” style limits where the exact boundary doesn’t matter. Users don’t complain if their 100th request arrives 60.5 seconds after the first — the limit is a general throttle, not a hard boundary.

High-throughput per-user rate limiting. If a single user might make thousands of requests per window, storing every timestamp is expensive. Token bucket or leaky bucket handle this naturally.

Burst-tolerant scenarios. Token bucket explicitly allows bursts up to the bucket capacity. Some workloads benefit from this — occasional bursts should succeed, then rate drops naturally.

Distributed systems where consistency matters less than availability. Sliding log needs a consistent view of timestamps; sharding across multiple Redis instances requires care. Simpler algorithms are easier to distribute.

The default recommendation: use sliding window counter for public APIs, token bucket for user tiers, sliding log for specific high-value operations. Don’t apply the same algorithm everywhere just because one approach is “best.”

The Hybrid Pattern

Most applications benefit from combining algorithms — sliding log where accuracy matters, cheaper algorithms everywhere else.

<?php
class RateLimiter
{
    public function __construct(
        private Redis $redis,
        private SlidingLogLimiter $preciseLimiter,
        private TokenBucketLimiter $generalLimiter,
    ) {}

    public function check(string $userId, string $endpoint): bool
    {
        // Critical endpoints use sliding log for exact accuracy
        if ($this->isCriticalEndpoint($endpoint)) {
            return $this->preciseLimiter->check(
                userId: $userId, 
                endpoint: $endpoint, 
                limit: 10, 
                windowSeconds: 60,
            );
        }

        // Everything else uses token bucket for efficiency
        return $this->generalLimiter->check(
            userId: $userId, 
            capacity: 100, 
            refillPerSecond: 10,
        );
    }

    private function isCriticalEndpoint(string $endpoint): bool
    {
        return in_array($endpoint, [
            'payment.authorize',
            'auth.login',
            'auth.password_reset',
            'account.delete',
        ], true);
    }
}

Why this works: the vast majority of API traffic goes through general endpoints where approximation is fine and memory efficiency matters. A small fraction goes through critical endpoints where exactness matters and memory cost is acceptable. Each algorithm plays to its strengths.

Symfony Rate Limiter Integration

Symfony’s symfony/rate-limiter component supports multiple algorithms out of the box, including sliding window. As of current versions, it doesn't ship with a native "sliding log" policy, but you can plug in a custom implementation.

Here’s a Symfony-integrated version:

<?php
namespace App\RateLimit;
use Symfony\Component\RateLimiter\LimiterInterface;
use Symfony\Component\RateLimiter\RateLimit;
use Symfony\Component\RateLimiter\Reservation;
class SlidingLogLimiter implements LimiterInterface
{
    public function __construct(
        private \Redis $redis,
        private string $id,
        private int $limit,
        private int $windowSeconds,
        private string $luaScript,
    ) {}

    public function reserve(int $tokens = 1, ?float $maxTime = null): Reservation
    {
        if ($tokens > $this->limit) {
            throw new \InvalidArgumentException('Cannot reserve more than the limit');
        }

        $now = microtime(true);
        $result = $this->redis->eval(
            $this->luaScript,
            ["ratelimit:{$this->id}", $now, $this->windowSeconds, $this->limit],
            1
        );

        $allowed = (bool) $result[0];
        $current = (int) $result[1];

        $rateLimit = new RateLimit(
            availableTokens: $this->limit - $current,
            retryAfter: new \DateTimeImmutable('@' . (int)($now + $this->windowSeconds)),
            accepted: $allowed,
            limit: $this->limit,
        );

        return new Reservation(0, $rateLimit);
    }

    public function consume(int $tokens = 1): RateLimit
    {
        return $this->reserve($tokens)->getRateLimit();
    }

    public function reset(): void
    {
        $this->redis->del("ratelimit:{$this->id}");
    }
}

What this does: implements Symfony’s LimiterInterface so the sliding log integrates with Symfony's rate limiter middleware, event listeners, and metrics. Applications using Symfony can drop in the sliding log policy where they need exact accuracy while keeping the standard policies elsewhere.

Note: this is a simplified illustrative version. Production integration should handle Symfony’s exact contract around Reservation, timing, and edge cases as documented in the Symfony rate limiter component. [Medium confidence — Symfony API surfaces evolve; verify against current documentation.]

Pitfalls to Avoid

Not making the check atomic. The naive PHP version has a race condition that allows concurrent requests to bypass the limit. Use the Lua script version in production.

Using integer timestamps. Second-level precision causes collision issues at high throughput. Use microtime(true) for sub-second precision, or add a random suffix to the member string.

Forgetting the TTL. Without EXPIRE, keys for inactive users accumulate indefinitely. The window+buffer expiration keeps memory bounded.

Storing all keys in one Redis instance for very large user bases. At scale, shard across multiple Redis instances by user ID hash. Each instance handles its own subset atomically.

Applying sliding log to every endpoint. The memory cost is real. Use sliding log where exact accuracy is needed; use cheaper algorithms elsewhere.

Not monitoring memory. Set up alerts on Redis memory usage. The sliding log’s memory grows with active users and request rates; unexpected traffic patterns can produce unexpected memory consumption.

Ignoring clock skew across servers. If multiple PHP servers write to the same Redis with clocks that disagree by seconds, timestamp ordering breaks. Use NTP-synchronized clocks or use Redis’s own time via TIME command inside the Lua script.

Confusing “requests” with “successful requests”. Should failed requests count against the rate limit? Depends on your policy. Sliding log records whatever you tell it to record; the design decision is separate.

Mini Q&A

How accurate is “sliding window counter” compared to sliding log?

Sliding window counter approximates the exact count using a weighted formula. For steady traffic, it’s very close (within 1–2%). For bursty traffic exactly at window boundaries, it can allow slightly more or fewer requests than the exact number. For most applications, this approximation is acceptable. For compliance or high-stakes scenarios, the difference matters.

Can I use sliding log without Redis?

You can implement the algorithm in any data store that supports sorted collections with efficient range operations. Memcached doesn’t have sorted sets, so it’s not directly suitable. PostgreSQL with a table and index works but is slower per operation. MySQL, MongoDB, and DynamoDB all have paths to implement it. Redis is the most common choice because sorted sets and Lua scripting are designed for exactly this pattern.

What if the Redis instance goes down?

Depends on your fail-open vs fail-closed policy. Fail-open (allow requests when Redis is down) prioritizes availability; you accept some rate limit bypasses during Redis outages. Fail-closed (deny requests when Redis is down) prioritizes rate limit enforcement; you accept downtime during Redis outages. For security-critical rate limits, fail-closed is usually correct. For UX-oriented rate limits, fail-open is usually correct.

How does sliding log handle distributed systems?

If all your servers write to the same Redis instance, distribution is natural — Redis serializes all operations. If you shard across multiple Redis instances, you need a consistent sharding strategy: usually hash the user ID to route consistently. Each shard operates independently. If a user’s requests reach different shards (shouldn’t happen with consistent hashing), the rate limit becomes per-shard rather than global.

What’s the actual worst-case memory for one user?

The theoretical maximum is requests_per_window × bytes_per_entry. With Redis 7 and typical entry sizes (~40 bytes for small sets, ~100 bytes for large), 1,000 requests/window per user ≈ 100 KB. For 100 users at that rate, 10 MB. For 10,000 users at that rate, 1 GB. Practical maximums depend on your specific traffic patterns.

Does the algorithm work for distributed rate limiting across regions?

Not directly. Cross-region latency to a single Redis instance is slow. Solutions include: (1) per-region rate limits accepting some cross-region approximation, (2) Redis with cross-region replication accepting eventual consistency, (3) more sophisticated distributed algorithms like distributed sliding window using multiple regional instances.

How do I choose the right window size?

Match the window to the business rule. “10 login attempts per hour” → 3600 second window. “100 API calls per minute” → 60 second window. For sliding log, larger windows mean more timestamps stored per user. If the window is too small (like 1 second), you may see excessive limit resets; too large means large memory per user.

Can I have multiple limits for the same user (per-second and per-minute)?

Yes, with separate keys. Store per-second limits in one key with a 1-second window; per-minute limits in another with a 60-second window. Check both. Real APIs often have layered limits — 10 per second, 100 per minute, 1000 per hour — each enforced independently.

Wrap-Up

The sliding log rate limiter is the most accurate rate limiting algorithm available, and the most expensive. The core insight is that these two facts are inseparable — the accuracy comes from storing individual timestamps, which is exactly why the memory cost scales with request volume rather than being constant per user.

For teams evaluating rate limiting options, the question isn’t “which algorithm is best?” but “what does this specific endpoint need?” Public APIs with millions of users generally benefit from token bucket or sliding window counter. Critical operations — authentication, payments, regulatory-tracked actions — often benefit from sliding log despite the cost. Most applications need both, applied to different endpoints.

For teams implementing sliding log, the practical work is: use Redis sorted sets, wrap the check-and-add in a Lua script for atomicity, monitor memory usage, set appropriate window sizes, and combine with other algorithms in a hybrid pattern. The code is straightforward once you understand the primitives; the real skill is choosing when to apply it.

The deeper takeaway: infrastructure algorithms have trade-offs, and the trade-offs matter. “Best” depends on constraints — accuracy requirements, memory budget, user scale, compliance context. Sliding log is a good example: obviously worse than token bucket on memory efficiency; obviously better on accuracy; the choice depends on which of those matters more for your specific use case. Building intuition for these trade-offs — knowing when to reach for the expensive-but-accurate option and when the cheap-but-approximate option is fine — is a real engineering skill.

Closing Loop

The fintech team eventually implements sliding log for the high-risk authorization endpoint. Compliance is satisfied — the audit shows exact 10-per-60-second enforcement, no boundary bursts, no approximations. Redis memory increases 40% but stays within budget. The team documents the choice: “Sliding log for regulated endpoints. Token bucket for general API. The extra memory pays for the accuracy where accuracy is required.”

Six months later, a new endpoint added for KYC verification uses the same pattern — sliding log because compliance requires exact accounting. The team’s rate limiting architecture handles it without additional design work. The hybrid pattern generalizes: new critical endpoints get sliding log, new general endpoints get token bucket, and the operational overhead stays constant.

A year later, the team scales significantly — user base grows 5×, request volume grows 8×. The token-bucket-limited endpoints handle the growth naturally. The sliding-log-limited endpoints require Redis capacity planning, but the memory growth is predictable (linear with users) and budgeted for. The choice made years ago — sliding log where it mattered, cheaper algorithms elsewhere — proves robust at scale. The engineering discipline of “match the algorithm to the requirement” continues to pay off as the system grows.

“People Also Ask”

1. What is a sliding log rate limiter? A rate limiting algorithm that stores the exact timestamp of every request in a sorted structure (typically a Redis sorted set). On each new request, it removes timestamps older than the window, counts what remains, and allows the request only if the count is under the limit. The result is exact enforcement — no boundary bursts, no approximation. The cost is memory: storage is O(N) per user where N is requests-per-window, compared to O(1) for counter-based algorithms.

2. How does sliding log compare to sliding window counter? Sliding window counter approximates the count using a weighted formula based on the previous and current windows: current + weight × previous. It uses two integers per user (~16 bytes) and gives ~99% accuracy for typical traffic. Sliding log stores every timestamp (~30-100 bytes each) and gives exact accuracy. For general rate limiting, sliding window counter is usually sufficient. For compliance, anti-abuse where exact enforcement matters, and audit-critical operations, sliding log is worth the extra cost.

3. What’s the memory cost of sliding log? Verified with Redis 7: about 30–40 bytes per entry for small sets (< 128 entries), about 100–120 bytes per entry for larger sets after Redis switches from listpack to skiplist representation. Practical numbers: 100 timestamps per user = ~3 KB; 1,000 timestamps = ~100 KB; 5,000 timestamps = ~600 KB. At 100k users × 100 requests/window, total memory is roughly 300–500 MB — about 150× more than token bucket for the same user count.

4. Why does the check-and-add need to be atomic? Without atomicity, concurrent requests from the same user create a race condition. Process A reads the count (say, 2 out of 3), decides it’s below limit, and starts to add its timestamp. Process B reads the count in the same window (also 2), decides it’s below limit, and adds its timestamp. Now there are 4 timestamps but the limit is 3 — both requests were allowed. Under high load with concurrent requests, this can significantly bypass the intended limit. A Lua script executes the entire check-and-add as one atomic operation, eliminating the race.

5. When should I use sliding log instead of token bucket? Use sliding log when you need exact accuracy: regulatory compliance requiring precise enforcement of “N requests per T seconds”, anti-abuse scenarios where boundary bursts matter, audit trails requiring exact request timings, premium API tiers with contractual accuracy expectations, small user counts where memory cost is negligible. Use token bucket for general API rate limiting, high-scale public APIs, burst-tolerant workloads, and scenarios where approximation is acceptable.

6. How do I implement sliding log without Redis? The algorithm requires efficient sorted collections with range-remove and count operations. PostgreSQL: a table with a timestamp column and index, using DELETE WHERE ts < now - window and SELECT COUNT(*). MongoDB: a collection with an indexed timestamp field. DynamoDB: a table with timestamp as sort key. The performance vs Redis is usually worse — SQL databases add per-operation overhead that Redis avoids. If you're already using Redis, it's the natural choice; if you're not, evaluate whether the extra dependency is worth it.

7. Can sliding log work in distributed systems? Yes, with care. If all servers write to the same Redis instance, the atomicity works globally. If you shard across multiple Redis instances, use consistent hashing on the user ID so a user’s requests always route to the same shard. Cross-region rate limiting is harder — Redis with cross-region replication accepts eventual consistency; alternatively, per-region limits accept some cross-region approximation. For very large systems, more sophisticated distributed algorithms exist but add complexity.

8. How does the sliding log handle clock skew? If servers writing to Redis have significantly different clocks, timestamp ordering breaks — a server with a slow clock might add a “past” timestamp that shifts the window incorrectly. Solutions: use NTP-synchronized clocks (standard on most cloud platforms); use Redis’s own time via the TIME command inside the Lua script (guarantees consistent time regardless of client clocks); use monotonic time sources if available. For most production setups with NTP, clock skew is measured in milliseconds and doesn't materially affect rate limiting accuracy.

Note: All Redis behaviors, memory measurements, and PHP code in this article were verified against Redis 7.4 and PHP 8.3.6. Memory usage for Redis sorted sets depends on the internal representation (listpack vs skiplist), which is controlled by configuration parameters zset-max-listpack-entries and zset-max-listpack-value — default thresholds may vary between Redis versions. The specific per-entry byte counts reflect these defaults; other configurations produce different numbers. Symfony rate limiter component integration reflects current documented API; specific version details should be verified against Symfony documentation. Rate limiting is a design domain with many valid approaches; the trade-offs discussed here are general guidance, not universal rules. For production systems, benchmarking your specific traffic patterns is more reliable than theoretical projections.