Colibri Runs a 744B Model on 25 GB of RAM. The Real Breakthrough Is Weight Streaming

Colibri treats VRAM, RAM and NVMe as one inference hierarchy. That makes frontier-scale Mixture-of-Experts models accessible on smaller machines, but the latency and storage trade-offs matter more than the headline.

A 744-billion-parameter model reports itself ready on a machine with about 25 GB of RAM. (Project overview)

That sounds like a compression story. It is easy to imagine that Colibri somehow shrank the complete model until it fit inside a laptop but, it did not.

The model still occupies hundreds of gigabytes on storage. Colibri changes something more fundamental: which weights must be in fast memory at the same time.

If you hit the paywall, use the friend link here: **read the full post**.

That distinction is what makes the project technically interesting. It also explains why the result is both real and much narrower than claims such as “GPUs are no longer needed” or “an office laptop can now replace an inference server”.

The useful question is not whether Colibri kills GPU serving. It is:

Can we run a massive sparse model by moving its active weights through VRAM, RAM, and NVMe only when they are needed?

Colibri shows that we can. The cost is that storage bandwidth and latency become part of the inference engine.

What Colibri actually is

Colibri is an Apache-2.0 open-source inference runtime written primarily in C. Its reference model is GLM-5.2, a 744B-parameter Mixture-of-Experts model with roughly 40B parameters active per token.

The project can run without a GPU. It also supports CUDA, Metal, Vulkan, NUMA-aware placement and partial or full expert residency when more capable hardware is available.

The same front-end exposes:

• a terminal chat interface;
• an OpenAI-compatible API;
• an Anthropic Messages API translation layer;
• a browser dashboard with latency, cache, storage-tier and expert-routing metrics.

But these interfaces are not the core contribution. The important part is the memory model underneath them.

Most inference engines begin with a hard question:

Does the model fit in GPU memory, or at least in RAM?

Colibri changes the question:

Which weights are needed now, where should they live and how quickly can we move them?

Why Mixture-of-Experts makes this possible

A dense model uses nearly all of its parameters for every token. A Mixture-of-Experts model contains a large bank of specialist feed-forward networks, but a router selects only a small subset for each token.

GLM-5.2 has 744B total parameters, while about 40B are active for a token. Colibri separates that active work into two broad groups. (Official model card)

The dense portion includes attention, embeddings, shared experts and other components needed repeatedly. In the project’s int4 layout, this resident set is about 9.9 GB.

The routed experts are much larger as a collection, but only selected experts are needed at each layer. Colibri keeps about 19,456 routed experts on storage and stages them when the router selects them. (Architecture notes)

The int4 GLM-5.2 container is still about 372 GB. The model has not become small. Only its fast-memory working set has become manageable. (Quick start)

A useful mental model is:

VRAM  -> hottest experts and GPU-resident compute
RAM   -> dense weights, warm experts, and caches
NVMe  -> the cold expert bank

The placement changes performance, not the router’s decision or the model’s intended precision. That is an important design boundary. A low-memory machine should become slower, not silently run a different model.

The per-token path

When the model generates a token, each MoE layer follows roughly the same sequence:

1. The dense part computes the router scores.
2. The router selects the experts required for that token.
3. Colibri checks whether those experts are already in VRAM or RAM.
4. Missing experts are read from NVMe, with asynchronous loading and prefetch where possible.
5. The selected experts run, their outputs are combined and execution moves to the next layer.

This is why the project compares its design to a just-in-time compiler for weights.

A compiler JIT does not optimize every possible code path in advance. It observes which paths execute and spends resources on the hot ones. Colibri records routing activity maintains per-layer LRU caches, pins frequently used experts and can prefetch one layer ahead.

The analogy is useful, but it has a boundary. Program branches are often more predictable than expert routing. A cache can also overfit one prompt or workload. Colibri’s current documentation treats learned placement and lookahead as measurable policies, not universal wins.

That engineering honesty matters. Once model weights live across three storage tiers, every optimization can simply move the bottleneck somewhere else.

Storage becomes part of inference

On a GPU server, we normally discuss compute throughput, HBM capacity, KV-cache pressure, batching and kernel efficiency.

With disk-streamed experts, we also have to discuss:

• NVMe bandwidth and random-read behavior;
• page-cache pressure;
• direct I/O versus buffered I/O;
• asynchronous reads;
• expert reuse and cache hit rate;
• overlap between storage and compute;
• whether a second SSD adds independent bandwidth.

Colibri stores an expert’s matrices together so they can be read in one operation. It overlaps missing-expert reads with resident-expert computation, loads each unique expert only once for a batch of positions and supports weighted distribution across two byte-identical model copies on separate drives.

This creates an unusual inference architecture: the storage subsystem is no longer a startup concern. It sits inside the token-generation critical path.

That is the real trade.

Colibri reduces the fast-memory requirement by accepting weight movement during inference.

Possible does not mean practical

The benchmark numbers make the boundary clear.

25 GB Development Machine

• Decode speed: 0.05–0.1 tokens/sec
• Meaning: The model can run, but performance is dominated by cold disk streaming.

128 GB CPU-Only Desktop

• Decode speed: About 1.8 tokens/sec when warm
• Meaning: Usable for patient local experimentation.

RTX 5070 Ti System

• Decode speed: About 1.07 tokens/sec
• Meaning: Partial GPU residency helps, but the complete model still does not fit in GPU memory.

Six RTX 5090 GPUs

• Decode speed: 5.8–6.8 tokens/sec
• Meaning: Full expert residency removes disk access from the decode path.

At 0.1 tokens per second, 100 output tokens take about 16 minutes and 40 seconds. (Reported input speed)

At 1.8 tokens per second, the same 100 tokens take about 56 seconds. (Reported input speed)

Those are very different user experiences, even though both machines are running the same model container.

So the 25 GB result is best understood as a feasibility floor. It proves that total parameter count no longer has to equal fast-memory capacity. It does not prove that minimum-memory inference is comfortable, economical, or suitable for an interactive agent. (Project benchmark boundary)

The repository also reports that the same hierarchy can scale upward. If all experts become resident across multiple GPUs, disk disappears from decode and throughput rises.

Colibri is therefore not purely a CPU runtime. It is one engine that changes placement according to the machine.

Colibri and vLLM solve different problems

It is tempting to compare Colibri directly with vLLM because both run language models. The comparison becomes misleading unless we name the constraint each system is optimizing.

***vLLM* is designed for efficient serving when the model can be placed on GPU infrastructure. Its strengths include continuous batching, efficient KV-cache management, parallel serving and high throughput across concurrent requests.

Colibri is designed for models whose full weights cannot remain in fast memory. It prioritizes heterogeneous placement, expert streaming, cache learning and the ability to trade latency for lower hardware requirements.

A production API serving many users normally values predictable latency and throughput. vLLM is the more natural category of runtime there.

A researcher who wants to inspect expert routing, run a frontier-scale open-weight MoE locally or experiment with memory placement may value something different.

Colibri makes that class of work possible without first building a large GPU cluster.

Neither result makes the other obsolete.

The design now extends beyond one model

GLM-5.2 remains the reference implementation, but the repository now includes sibling engines for Inkling, OLMoE and Kimi K3. (Current model roster)

Kimi K3 makes the memory argument even more visible. Its checkpoint contains about 2.8 trillion parameters, with 104B active.

The routed experts are already quantization-aware trained in MXFP4 and account for most of an approximately 1.5 TB text-only container. (Kimi K3 engine notes)

Colibri streams those native expert bytes directly rather than expanding and requantizing them. The engine’s documentation reports that direct I/O and pipelined reads improved one full-model path from about 21 seconds per token to 9.4 seconds per token. (Kimi K3 storage measurement)

That is a meaningful systems improvement. It is still far from production-serving latency.

The lesson is not that every huge MoE can suddenly run well on a laptop. It is that the same placement problem appears across model families and a runtime can attack it systematically rather than treating insufficient VRAM as an immediate stop condition.

What a practical local setup requires

The runtime binary is small. The model is not.

For the GLM-5.2 reference path, we need roughly:

• a 372 GB int4 model container;
• enough free RAM for the resident dense set, runtime state, and operating system;
• fast NVMe storage;
• patience proportional to the number of experts that miss the warm tiers.

The official quick-start flow is straightforward:

git clone https://github.com/JustVugg/colibri
cd colibri/c
./setup.sh

COLI_MODEL=/nvme/glm52_i4 ./coli doctor
COLI_MODEL=/nvme/glm52_i4 ./coli plan
COLI_MODEL=/nvme/glm52_i4 ./coli chat

doctor checks readiness, while plan shows the intended VRAM, RAM and disk placement before a full run.

We have not executed the 372 GB model in this review environment, so these commands are source-verified rather than hands-on validated here.

Anyone evaluating the runtime should record the exact commit, model container, storage devices, cache state, prompt, context length and tokens per second.

A single unexplained speed number is not enough for this architecture. (Benchmark protocol)

Where Colibri is useful

Colibri is compelling when our goal is:

• local research on very large open-weight MoE models;
• understanding expert routing and reuse;
• privacy-sensitive, low-throughput inference;
• testing weight placement, storage scheduling or quantization choices;
• using available RAM and NVMe instead of renting a large GPU cluster for occasional experiments.

It is a poor fit when we need:

• low-latency interactive chat on minimum hardware;
• high-throughput multi-user serving;
• predictable agent-loop latency;
• a mature production SLA;
• simple deployment without hundreds of gigabytes or terabytes of model storage.

The project itself describes Colibri as both an inference engine and an open research platform. That is the right expectation. Its value today is as much in the architecture we can inspect and modify as in the tokens it generates.

The broader lesson

We often treat model size as a capacity question:

How much GPU memory do we need to hold the weights?

Sparse models turn it into a scheduling question:

Which weights are active, where are they placed, and can we move them without making latency unacceptable?

Colibri does not remove the hardware cost of frontier inference. It redistributes that cost across storage capacity, I/O bandwidth, RAM, cache behavior and time.

That is why “NVIDIA killer” misses the point.

The useful result is more precise:

A frontier-scale sparse model does not have to live entirely in GPU memory or system RAM. Its weights can be staged across a heterogeneous memory hierarchy, as long as we accept and manage the resulting latency.

That is not a replacement for GPU serving. It is a new and valuable systems design point between “the model fits” and “the model cannot run”.

References

• Colibri repository and architecture overview
• Colibri benchmark methodology and community measurements
• Colibri Kimi K3 engine notes
• GLM-5.2 model card and MIT licence