- Introduction — a 744B model on a 25GB-RAM laptop?
- The core trick — MoE sparsity plus disk streaming
- The machinery that makes it bearable
- The honest part — so how fast is it?
- Closing — between "it works" and "it is usable"
- References
Introduction — a 744B model on a 25GB-RAM laptop?
colibrì is an inference engine built to run GLM-5.2 on an ordinary consumer PC. As colibrì describes it, GLM-5.2 is a Mixture-of-Experts model with 744B total parameters. A model this size normally demands several GPUs and hundreds of gigabytes of VRAM. colibrì runs it on a 12-core CPU with 25GB of RAM — from roughly 1,300 lines of pure C, with zero runtime dependencies: no BLAS, no Python.
It is not free, of course. The project's stated goal is not speed at all. Paraphrased: make GLM-5.2 answer correctly on a 12-core laptop with 25GB of RAM, "even if speed is measured in minutes per paragraph." This post looks honestly at what colibrì actually does, the tricks that make it possible, and how usable the result really is.
The core trick — MoE sparsity plus disk streaming
Two facts interlock.
First, GLM-5.2 is MoE. Generating one token activates roughly 40B of the 744B, not all of it. The parameters are split into "experts," and per token a router picks only the few that fit the input. In colibrì's build there are 21,504 routed experts (75 MoE layers × 256), each about 19MB at int4.
Second, that means you do not need all of it in RAM. colibrì splits the model in two.
- Dense part — attention, shared experts, embeddings, about 17B. Compressed to int4, only 9.9GB stays resident in RAM.
- Routed experts — about 370GB (int4 container). They live on an NVMe SSD, and only the ones a token needs get read in.
So: a giant model, a tiny engine. The per-token bottleneck is not compute but disk reads — cold, one token costs roughly 11GB read from SSD (75 layers × 8 experts). That disk bandwidth sets the speed ceiling.
The machinery that makes it bearable
Disk streaming is inherently slow. The interesting part of colibrì is how it chips away at that slowness.
- Learning cache — it records which experts your real usage routes to, into
.coli_usage, updated every turn, and at the next startup pins the "hottest" ones into spare RAM. In other words it gets faster the more you use it;--repineven re-tiers live mid-session. - MTP speculative decoding — GLM-5.2's multi-token-prediction head drafts several tokens ahead and verifies them in one pass. With an int8 head, colibrì measures 2.2–2.8 tokens per forward at 39–59% acceptance. Drop the head to int4 and acceptance collapses to 0–4%, so only the head is kept at int8.
- Compressed KV cache — 576 floats per token versus the original 32,768, 57× smaller. Conversations reopen warm even after the engine restarts.
- Async readahead and batch union — the next block of experts is read while the current one is being multiplied, and within a batch each unique expert is read only once.
Quantization underpins all of it: an int4 container with per-row scales, AVX2 kernels doing dequant-on-use. The fight I covered in the quantization roundup — spend fewer bits to buy memory and bandwidth — carries straight over here as "read fewer bytes off the disk."
The practical floor to run it, and the four entry points:
# hardware floor (from the README)
OS: Linux or WSL2 CPU: AVX2, 12+ cores
RAM: >=16 GB (cache auto-capped, ~20 GB peak)
Disk: ~370 GB local NVMe (ext4) — never a network mount
# run modes
./coli chat # interactive CLI
./coli serve # OpenAI-compatible API (text-only)
./coli bench # MMLU / HellaSwag / ARC
./coli plan # header-only dry run, zero allocation
The honest part — so how fast is it?
Slow. Cold, about 0.05–0.1 tok/s — effectively one word every 10–20 seconds. Load alone takes about 30s. Community-reported measurements vary by hardware: M5 Max (128GB) 1.06 tok/s, Framework 13 (128GB + learned pin) 0.37 tok/s, a 9950X with PCIe5 NVMe 0.28 tok/s. Even the best case is barely over a token per second.
This is not a conversational tool. A paragraph takes minutes. So when do you reach for it?
- Where the cloud is impossible or forbidden — offline, air-gapped, regulated environments.
- Where privacy is non-negotiable — the data must not leave the machine.
- When you just want to experiment without buying hardware — to see a 744B model's output on an old desktop with your own eyes.
If you need production throughput the answer is clear: allocate proper GPUs. colibrì is not trying to replace that — you can open an OpenAI-compatible API with ./coli serve, but you cannot serve real traffic at that speed.
Closing — between "it works" and "it is usable"
What is genuinely interesting about colibrì is not the speed but the mere fact that it runs. A scale long treated as "impossible without a GPU" nonetheless produces answers on a 25GB-RAM machine, via MoE sparsity plus disk streaming plus clever caching. "It works" and "it is usable" are different things — but "it works" has to be proven first before "it is usable" can follow.
Democratizing local inference has usually arrived in that order — first "this is insane but it runs," then, over years, "this is actually practical." llama.cpp began exactly that way. Where colibrì sits today, the numbers above say honestly enough. Still, it is not hard to imagine the day minutes-per-paragraph becomes seconds-per-sentence.
References
현재 단락 (1/34)
colibrì is an inference engine built to run GLM-5.2 on an ordinary consumer PC. As colibrì describes...