vLLM-Inspired LLM Serving Engine on Apple Silicon with MLX

Recreating core ideas behind vLLM on Apple Silicon using MLX while benchmarking the system against real vLLM serving.

This project explores the core systems ideas behind modern LLM serving engines such as vLLM through a lightweight MLX-based implementation built on Apple Silicon. The server implements continuous batching, paged KV-cache abstractions, prefix-aware KV reuse, and chunked prefilling to study how inference systems optimize throughput and memory efficiency under concurrent workloads. The implementation is benchmarked against both a CPU-only llama.cpp baseline and real vLLM deployments to analyze scaling behavior, latency tradeoffs, and hardware bottlenecks during concurrent inference serving.

What vLLM Actually Does

vLLM’s performance comes from three core ideas:

Continuous batching: Instead of waiting for an entire batch to finish before starting the next one, requests are admitted and retired individually during generation. New requests can join while others are still decoding, keeping the GPU continuously busy.

PagedAttention: KV cache is stored in fixed-size physical blocks, similar to virtual memory pages. Logical token positions map to physical blocks through a block table, reducing fragmentation and allowing memory to be allocated dynamically as sequences grow. In vLLM, the attention kernel reads directly from these block mappings without reconstructing a dense KV tensor.

Prefix caching: If multiple requests share a common prefix, such as a system prompt, vLLM reuses previously computed KV states instead of recomputing the entire prefix for every request. Only the unique suffix needs additional processing.

PagedAttention is implemented using fused CUDA kernels that directly operate on paged KV memory layouts. That is the component that cannot be reproduced efficiently on Apple Silicon without equivalent Metal kernels.

The Stack (which mimics vLLM)

Continuous Batching      → Requests join and leave the running batch 
                           mid-generation, no waiting for others to finish
KV Management            → KV memory stored in fixed-size blocks and 
                           allocated dynamically as sequences grow
Prefix Caching           → Shared prompt prefixes computed once and 
                           reused across matching requests
Dynamic KV Allocation    → Sequences receive KV blocks on demand, 
                           with eviction/preemption when memory is exhausted
Chunked Prefill          → Long prompts processed incrementally while 
                           preserving accumulated KV cache state
Streaming Inference      → Tokens sent to the client as they generate, 
                           (OpenAI-compatible)
KV Paging Experiment     → KV physically stored in pages, reassembled 
                           into a dense tensor before each attention call

How Each Component Is Implemented

Continuous batching: The scheduler was built around mlx_lm’s BatchGenerator, which processes decoding one step at a time across active requests. New requests can join the batch while existing ones are still generating, allowing the system to keep the GPU continuously occupied under concurrent workloads.

One interesting observation from benchmarking was that continuous batching is not a free optimization. At low concurrency, the scheduling overhead actually reduced throughput compared to simpler wave-based generation. The benefits only became visible once concurrency increased enough to keep decoding slots consistently occupied.

Paged KV management: A custom paged KV memory manager was implemented using fixed-size KV blocks, per-sequence block tables, and dynamic allocation during generation. This reproduced the scheduler-side behavior of PagedAttention, including fragmentation tracking and memory preemption.

An experimental gather-based KV path was later added to store KV tensors physically in paged layouts, effectively “paged allocation + gather, dense SDPA.” However, unlike real vLLM, attention still operated on reconstructed dense tensors rather than reading directly from paged memory through fused kernels. Building both versions made it clear that the real performance advantage of PagedAttention comes from the attention kernel itself, not just the block-table abstraction.

Prefix caching: Prefix caching was implemented using a radix-tree KV cache with block-aligned sharing. Requests with common prompt prefixes reused previously computed KV states instead of recomputing them from scratch.

One subtle detail was that caching only occurred at fixed token boundaries, matching how real paged KV systems treat blocks as the fundamental unit of sharing.

Chunked prefill: Chunked prefill was implemented by processing long prompts incrementally while carrying KV cache state across chunks. This allowed large prompts to be interleaved with active decoding workloads instead of monopolizing the scheduler during a single long prefill phase.

Full implementation details are available in the GitHub repository.

With the core scheduling and KV-management components implemented, the next step was understanding how closely the system behaved compared to baseline inference and production vLLM serving.

Benchmarking Against vLLM

The implementation was benchmarked against:

• a CPU-only llama.cpp setup,
• and production vLLM running Llama-3.2–1B-Instruct FP16 on a Colab T4 GPU.

All systems were tested using the same prompts and concurrency levels.

One important detail is that the latency measurements are not identical quantities. Local TTFT was measured internally from request arrival to first-token generation inside the scheduler, while the vLLM measurements include full end-to-end latency through an ngrok tunnel. Because of this, the absolute latency numbers should not be treated as a strict head-to-head comparison.

Throughput scaling across CPU inference, the MLX-based near-vLLM stack, and production vLLM under increasing concurrency.

The latency trends told the same story. vLLM’s P50 latency remained nearly flat under increasing concurrency, while the local implementation degraded significantly once memory bandwidth became saturated. The scheduler itself continued behaving correctly under load, but the underlying hardware and kernel efficiency eventually became the limiting factors.

P50 latency under increasing concurrency. The MLX-based near-vLLM stack begins degrading once memory bandwidth becomes saturated, while vLLM maintains more stable latency under load.

The three systems showed very different scaling behavior under load.

The CPU-only llama.cpp baseline improved throughput gradually with concurrency, but latency remained extremely high because requests were effectively competing for limited compute resources without modern serving optimizations.

The MLX-based near-vLLM stack scaled significantly better, peaking around concurrency 4 before degrading as memory bandwidth became saturated on the M1 Air.

vLLM on the T4 GPU continued scaling much more consistently at higher concurrency levels due to fused attention kernels, significantly higher memory bandwidth, and production-grade GPU scheduling.

Where the Hard Part Actually Is

The benchmark results also made it clear where the largest remaining gap with production vLLM still exists.

The fused PagedAttention kernel is the one major component that was not implemented in this project, and it is also the part that matters most for high-throughput inference at scale.

While the scheduler and KV paging logic can be reproduced at the Python level, real PagedAttention requires the attention kernel itself to read directly from paged KV memory without reconstructing dense tensors during every decode step. That requires custom GPU kernels rather than scheduler-side abstractions alone.

In this implementation, paged KV tensors were reconstructed into dense views using mx.take before attention execution. The structure was correct, but the additional re-materialization overhead remained. Real vLLM avoids this cost entirely through fused CUDA kernels, while newer Apple Silicon efforts such as vllm-metal implement equivalent behavior using Metal kernels.

Unsurprisingly, during testing the same model still produced different outputs across both servers for identical prompts. LLMs being LLMs, small differences in runtime execution and decoding behavior can still lead to divergent generations.

Unsurprisingly, during testing the same model still produced different outputs across both servers for identical prompts. LLMs being LLMs, small differences in runtime execution and decoding behavior can still lead to divergent generations.

While this project does not implement fused Metal attention kernels, the serving stack is fully usable for local MLX-based LLM inference experiments on Apple Silicon. The repository can be used as a lightweight playground for exploring continuous batching, KV-cache management, prefix caching, and scheduling behavior without requiring CUDA or custom GPU kernels.

Code, benchmark scripts, and experiment configurations are available in the GitHub repository.