How I Achieved Sub-20ms Response Times in FastAPI Using Cython and In-Memory Caching

Combining Native Python Speedups with Smart API Design for Ultra-Low Latency

Achieving lightning-fast API responses — under 20 milliseconds — isn’t just about fancy infrastructure or switching languages. It’s about precision engineering. When I began optimizing my FastAPI application, I wasn’t chasing vanity metrics. I had a real problem: a system under stress, with growing user traffic and expectations of instant interactions. Shaving off even a few milliseconds wasn’t just performance art — it was survival.

In this article, I’ll walk you through how I brought my FastAPI-based backend down to sub-20ms response times by combining Cython, in-memory caching, and smart API design principles. If you’re scaling a Python-based service and want to go toe-to-toe with Go or Rust in performance-critical paths, this is for you.

The Challenge: Python’s Bottlenecks in High-Traffic APIs

Python is brilliant for developer productivity but notoriously lacks raw speed due to its interpreted nature and the Global Interpreter Lock (GIL). My FastAPI service, which powers a real-time data analytics dashboard, was struggling once concurrent users spiked beyond a few hundred.

Endpoints that were supposed to return within 50ms began taking 80ms… then 100ms… and eventually choked at peak hours. No matter how fast the database or optimized the queries were, Python’s overhead was beginning to show.

The user-facing experience degraded. And so did our dashboard engagement rates.

Diagnosing the Latency Stack

Before optimizing, I profiled everything:

• Cold vs. warm request timings
• Uvicorn + Gunicorn workers
• Middleware overhead
• Function-level bottlenecks (via cProfile and line_profiler)

It became clear that the latency wasn’t database-bound. Most of the delay was happening inside CPU-bound pure Python logic — data transformations, JSON response generation, and some math-heavy routines used in scoring algorithms.

So I decided to bring out the big guns: Cython and caching.

Strategy 1: Accelerating Core Logic with Cython

Cython compiles Python code to C, dramatically increasing the execution speed of CPU-bound functions.

I identified two functions that contributed disproportionately to latency:

1. A scoring algorithm that processed numerical arrays
2. A data aggregator that compressed JSON payloads for response

After annotating these with Cython’s static type declarations (cdef int, cdef float[:], etc.), I compiled them into .so shared object modules and imported them directly into my FastAPI app like native extensions.

Result? ~8× speedup in the scoring function and 4× in data aggregation.

This shaved 10–15ms from every response that used those routines.

Strategy 2: In-Memory Caching with cachetools

Caching is the oldest trick in the book — but it’s how you cache that makes the difference.

For endpoints with deterministic results (e.g., same input → same output for the next 10s), I used cachetools.LRUCache with time-based expiry:

from cachetools import TTLCache, cached

score_cache = TTLCache(maxsize=1000, ttl=10)

@cached(score_cache)
def compute_user_score(user_id: int, data: dict):
    ...

No Redis, no memcached. Just blazing-fast, in-process memory that avoided cross-network lookups.

This removed entire compute cycles for 60–70% of repeated requests within the TTL window, dropping response times by another 5–10ms.

Strategy 3: Lightweight JSON Responses

Another unexpected win came from optimizing response payloads. FastAPI’s auto-generated Pydantic responses are elegant, but for ultra-low latency use cases, even that becomes overhead.

I swapped out Pydantic models with custom orjson-based responses for time-sensitive endpoints:

from fastapi.responses import ORJSONResponse

@app.get("/score", response_class=ORJSONResponse)
def get_score(...):
    return ORJSONResponse(content={"score": 98.7})

orjson is a high-performance JSON library written in Rust, and it halved the time needed to serialize responses.

Bringing It All Together: The Final Result

After deploying these optimizations:

• Average response time dropped from ~60ms to 18ms
• 99th percentile latency dropped to ~25ms
• CPU usage decreased by 35% during peak hours

Most importantly, users noticed. Interaction latency was gone. Charts updated faster. Clicks felt instant.

A Quick Note on Deployment

These improvements were deployed with:

• Uvicorn with --workers 4 --loop uvloop
• Gunicorn for multi-process scaling
• Dockerized .so Cython modules, compiled at build time
• Prometheus + Grafana for tracking latency percentiles

⚠️ Note: If you’re using Cython, make sure your CI/CD pipeline is equipped to compile native extensions. Use python setup.py build_ext --inplace or pyproject.toml with build-backend.

Real-World Use Cases for This Optimization Stack

If you’re building:

• Real-time analytics dashboards
• High-frequency trading APIs
• Leaderboards with live scoring
• ML-inference endpoints (non-GPU)

…this Cython + cache-first + low-level FastAPI design pattern is a game changer.

Final Thoughts: Python Can Be Fast

Too often, Python is dismissed as “slow.” But with Cython, smart caching, and lean response design, I turned a sluggish API into a real-time beast — without rewriting in Rust or Go.

If you’re hitting latency walls with Python APIs, don’t jump ship just yet. Optimize the critical path, measure deeply, and you might find the gains hiding right under your stack.

Let’s Push Python Further 🚀

If this deep dive helped or sparked ideas, give it a 👏, leave a comment, or share it with a fellow backend engineer struggling with latency. I’d love to hear how you’re tackling performance in your Python stack!

You can also follow me for more hands-on performance engineering and backend architecture deep dives.