Running a 744 Billion Parameter AI Model on a Regular Laptop: Inside the Colibri Inference Engine

How a tiny, dependency free C program makes frontier scale Mixture of Experts models usable on hardware most developers already own

The common assumption in artificial intelligence circles is straightforward: bigger models require bigger hardware, usually meaning multiple high end GPUs with enormous amounts of video memory. A newly released open source engine called Colibri challenges that assumption directly. It runs GLM 5.2, a 744 billion parameter Mixture of Experts model, on a consumer machine equipped with roughly sixteen to twenty five gigabytes of RAM, no GPU required. The trick is not compression in the traditional sense. It is a fundamentally different way of deciding what actually needs to live in memory at any given moment.

This article explains how Colibri works, what makes Mixture of Experts architectures uniquely suited to this kind of streaming approach, how to install and run it, and what its early benchmark data reveals about the real world tradeoffs of trading GPU memory for disk bandwidth.

The Core Idea Behind Colibri

A 744 billion parameter model sounds impossible to run without serious hardware, until the architecture underneath it is properly understood. GLM 5.2 is built as a Mixture of Experts model, meaning it does not use every single parameter for every token it generates. Instead, it activates only around forty billion parameters per token, selecting a small subset of specialized expert subnetworks through a routing mechanism. Even more importantly, only around eleven gigabytes worth of those active parameters actually change from one token to the next, since the dense, always active portion of the model stays constant.

Colibri exploits this structure directly. The dense part of the model, which includes attention layers, shared experts, and embeddings, totals roughly seventeen billion parameters and stays permanently resident in RAM using four bit integer quantization, occupying about 9.9 gigabytes. The remaining 21,504 routed experts, spread across seventy five Mixture of Experts layers with 256 experts each plus a multi token prediction head, live on disk instead, occupying roughly 370 gigabytes in their quantized form. These experts are streamed on demand as needed, backed by a per layer least recently used cache, an optional pinned hot store for frequently used experts, and the operating system’s own page cache serving as a free secondary layer.

The entire engine is implemented as a single C file of roughly 2,400 lines, along with a handful of small header files. There is no dependency on BLAS libraries, no Python required at runtime, and no GPU required for the base configuration, although an optional CUDA tier exists for pinning frequently used experts in video memory.

What the Engine Actually Implements

Colibri is not a simplified approximation of GLM 5.2. Its forward pass has been validated as token exact against a reference implementation built with the transformers library, matching output identically across both teacher forced and greedy decoding tests on a small model sharing the same real architecture.

Several architectural details specific to GLM 5.2 are faithfully implemented. Multi head Latent Attention compresses the key value cache down to 576 floating point values per token instead of the 32,768 values a naive implementation would require, a reduction of roughly fifty seven times, which matters significantly given that GLM 5.2 uses sixty four attention heads without grouped query attention. The routing mechanism follows a DeepSeek V3 style sigmoid router with auxiliary loss free routing and a routed scaling factor, alongside a shared expert and dense processing for the first three layers.

One of the more advanced features is native speculative decoding using the model’s own multi token prediction head, located at layer seventy eight. This head drafts candidate tokens that the main model then verifies in a single batched forward pass. The precision of this head matters considerably. When quantized to four bit precision, draft acceptance collapses to somewhere between zero and four percent, effectively disabling speculation entirely. At eight bit precision, community measurements show acceptance rates between thirty nine and fifty nine percent, translating to between 2.2 and 2.8 tokens generated per forward pass. The speculation remains lossless even under sampling, thanks to rejection sampling. An honest caveat applies here: on a cold cache, each verified draft token can trigger additional expert loads, sometimes pushing expert loads per token from around 660 up to 1,100, meaning speculation can actually slow things down until the cache has warmed up. An adaptive guard and a manual override exist specifically to handle this tradeoff.

A related feature allows grammar constrained drafting for structured output tasks such as JSON generation or function calling. When a formal grammar is supplied, any position where the grammar admits exactly one legal byte, such as a brace, a quote character, or an enum value, can be treated as a pre accepted draft with essentially perfect acceptance, without relying on the draft head or a lookup table at all. Because these forced spans are still verified through the same batched forward pass as any other draft, an incorrect or outdated grammar cannot alter the actual output. The worst case is simply a rejected draft.

Sampling defaults have been deliberately tuned for the realities of four bit quantization, using a temperature of 0.7 and a nucleus value of 0.90, rather than the official defaults of 1.0 and 0.95, which tend to sample noise introduced by quantization in the probability distribution’s tail.

On the performance side, integer dot product kernels using AVX2 instructions deliver meaningful speedups, with eight bit integer matrix multiplications measured at roughly 119 GFLOP per second, between 1.4 and 2.5 times faster than floating point equivalents, while four bit operations show gains primarily in batched scenarios. A related trick called MLA weight absorption avoids reconstructing key and value vectors on every token during decoding, instead absorbing that computation into the query projection itself, with results validated as exactly matching an unabsorbed implementation.

Additional engineering details include asynchronous expert readahead, where the engine begins reading the next block of experts from disk while the current block is still being multiplied; a sparse attention mechanism, borrowed from GLM 5.2’s own lightning indexer, which selects only the top 2048 relevant keys per layer rather than attending to the full context; and persistent key value caching, which allows a conversation to be closed and reopened later with the full context intact and zero reprocessing required.

Getting Started With Colibri

Setting up Colibri begins with building the engine and running its self test suite.

cd c
./setup.sh

The full model setup, including downloading GLM 5.2 in its original FP8 format and converting it to the engine’s four bit container, can be handled with a single command. Because the conversion happens shard by shard, the full 756 gigabyte checkpoint never needs to exist on disk all at once.

./coli convert --model /nvme/glm52_i4

Once conversion is complete, starting an interactive chat session automatically detects the RAM budget, expert cache size, and multi token prediction settings.

COLI_MODEL=/nvme/glm52_i4 ./coli chat

For anyone who prefers to skip the conversion step entirely, a pre converted version of GLM 5.2 is available for download, along with a community provided variant that includes the eight bit multi token prediction head needed for effective speculative decoding.

Before committing to a full model load, two diagnostic commands are available. The coli plan command inspects only the safetensors headers to report the model's exact memory footprint across dense parameters, expert storage, and any configured GPU tier, without allocating any actual tensors. The coli doctor command performs a broader readiness check, validating the model directory, tokenizer, available RAM, and any requested GPU devices, returning a clear pass or fail result before any real loading begins.

COLI_MODEL=/nvme/glm52_i4 ./coli plan --gpu 0,1 --ram 128 --vram 48
COLI_MODEL=/nvme/glm52_i4 ./coli doctor --gpu 0 --ram 128

Cross Platform Support

Colibri runs on Linux, macOS, and natively on Windows 11 using the MinGW w64 toolchain, without requiring the Windows Subsystem for Linux. The Windows port introduces a compatibility layer that maps POSIX style file operations onto their equivalent Windows API calls, keeping the core engine source code completely unchanged across platforms. Building on Windows follows a familiar pattern.

make glm.exe
SNAP=D:\glm52_i4 ./glm.exe 64 4 16
python coli chat --model D:\glm52_i4

An OpenAI Compatible API Server

Beyond the interactive terminal chat, Colibri can also run as a persistent server exposing a text only, OpenAI compatible HTTP API, using nothing beyond the Python standard library for the web gateway while inference itself still runs through the same dependency free C engine.

COLI_MODEL=/nvme/glm52_i4 COLI_API_KEY=local-secret ./coli serve --host 127.0.0.1 --port 8000
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Authorization: Bearer local-secret" \
  -H "Content-Type: application/json" \
  -d '{"model": "glm-5.2-colibri", "messages": [{"role": "user", "content": "Hello"}], "stream": true}'

Because a 744 billion parameter model cannot practically be loaded into memory more than once, the server keeps a single persistent model process and handles concurrent requests through a bounded first in first out queue rather than pretending to run genuinely parallel sequences. Requests that arrive when the queue is full receive a standard rate limit style error before any streaming begins. For applications that genuinely need multiple independent conversations, the server supports isolated key value cache slots, each maintaining its own token history and persistence file, up to sixteen slots in total.

The Learning Cache and Live Adaptation

One of the more distinctive aspects of Colibri is that it improves with continued use. The engine records which experts actual usage routes to in a local usage file, and at startup it automatically pins the most frequently used experts into whatever spare RAM is available. Over time, and particularly with an optional live adaptation mode, the engine can even swap out currently pinned experts for ones that have recently become hotter, guided by a decaying session level heat map with safeguards in place to prevent excessive back and forth swapping.

Understanding the Performance Tradeoffs

Because Colibri deliberately keeps the bulk of the model on disk, performance is fundamentally shaped by disk speed and available RAM rather than raw compute alone. On the original development machine, a modest setup with twelve cores, twenty five gigabytes of RAM, and networked NVMe storage capped at roughly one gigabyte per second, cold decoding runs at only around 0.05 to 0.1 tokens per second, since each token can require reading somewhere around eleven gigabytes of expert weights from disk in the worst case.

Community reported benchmarks illustrate how much this changes across different hardware profiles. Machines with faster local NVMe storage and larger RAM budgets for pinning hot experts consistently show meaningfully higher throughput, and in several cases doubling or nearly tripling disk bandwidth on the same machine produced a corresponding jump in tokens generated per second, while shifting the primary bottleneck away from disk reads and toward raw matrix multiplication throughput. On one high memory Apple laptop with a fast internal SSD, warmed caching pushed throughput close to two tokens per second, a meaningful result for a model of this scale running without a dedicated GPU at all.

An important structural finding from community testing is that available RAM, not disk speed alone, often becomes the binding constraint on smaller memory machines. With only twenty four gigabytes of RAM, the expert cache is automatically capped low enough that decoding stays effectively cold even on disks several times faster than the original development environment, underscoring that both factors need to scale together for real gains.

Accuracy and What Remains Unmeasured

While functional correctness has been carefully validated against a reference implementation, the accuracy cost of aggressive four bit quantization on real world benchmark tasks remains an open question that the project explicitly calls out as needing community help. Because scoring standard benchmarks like MMLU, HellaSwag, and ARC requires a full forward pass for every answer option, a complete benchmark run can take the better part of a day on slower storage. The tooling to run these evaluations is already built and ready to use.

./coli bench
./coli bench mmlu arc_challenge --ram 100

Published full precision scores for GLM 5.2 on these tasks sit roughly in the eighty five to ninety five percent range. If the four bit quantized version lands within a few points of that range, it would validate the quantization approach used throughout the engine. If not, it would point toward the need for more sophisticated mixed precision or grouped scale quantization strategies going forward.

Why This Project Matters

Colibri represents a meaningful data point in an ongoing conversation about accessibility in artificial intelligence. The prevailing narrative has long assumed that meaningfully large language models are simply out of reach without institutional scale hardware budgets. By treating a large Mixture of Experts model as something closer to a database with a small, always resident working set and a much larger set of rarely touched records, Colibri demonstrates that the underlying architecture of modern frontier models leaves real room for creative engineering around memory and storage constraints, even if the resulting performance is, by the project’s own admission, far from fast in its baseline configuration.

Conclusion

Colibri shows that running a genuinely large scale Mixture of Experts model does not strictly require an expensive cluster of GPUs. By keeping only the consistently active dense portion of a model resident in memory and streaming the much larger set of routed experts from local disk storage, the engine makes a 744 billion parameter model usable on hardware many developers already own. Features such as validated token exact correctness, native speculative decoding, grammar constrained drafting, persistent conversation memory, and a self improving expert cache round out a project that treats efficient local inference as a serious engineering problem rather than a compromise.

The performance numbers so far are modest, especially on constrained hardware, but the community benchmark data already shows a clear and encouraging trend: faster local storage and more available RAM translate directly into real gains, suggesting plenty of headroom remains as more people test it on stronger machines.

The repository is available at: https://github.com/JustVugg/colibri