Split View: 26B 모델을 2GB 램으로 돌리는 원리 — 상주 메모리와 워킹셋은 같은 숫자가 아니다

How a 26B Model Runs in 2GB of RAM — Resident Memory and Working Set Are Not the Same Number

• Intro — what number is "2GB" counting
• Where does 14.3GB come from — start with the storage arithmetic
• The two kinds of weights MoE creates
• Why they dropped mmap and went to pread
• 285MB per token — bandwidth sets the ceiling
• Resident memory and working set — where 2GB is true and where it misleads
• The quality cost of 4-bit, and where this approach fits
• Closing — check what a number counts first and most of the magic explains itself

Intro — what number is "2GB" counting

On July 29, 2026, a project called TurboFieldfare went up on Show HN. It is an inference runtime written in Swift and Metal, and it comes with the claim that it runs Gemma 4 26B-A4B on M-series Macs in about 2GB of RAM. The figures the repository README lists are 5.1 to 6.3 tokens per second on an M2 MacBook Air with 8GB, and 31 to 35 on an M5 Pro with 24GB.

The sentence "26B in 2GB" makes you reflexively suspicious. And yet this project's numbers mesh well with one another. Work them out and most of them are correct. The interesting point is not that the claim is wrong; it is what number the 2GB is counting. It is the process resident set, and that is a different value from the memory the system actually uses. And this difference explains a good part of the sixfold performance gap between the M2 Air and the M5 Pro.

Rather than take the README and HN thread figures at face value, this post derives them and checks that they line up. Let us separate the items where the arithmetic works from the items where it does not and something else is needed to explain them.

Where does 14.3GB come from — start with the storage arithmetic

Disk first. The README lists the install size of the text-only model at about 14.3GB, and the quantization scheme as "MLX affine 4-bit, group 64, router at 8-bit." By the Gemma 4 model card, 26B-A4B has 25.2B total parameters, 3.8B active, and a 256K context.

It is 4-bit, so why is it not 12.6GB, half of 25.2B? Because group-wise affine quantization does not store only the weights.

Real storage cost of group-64 affine 4-bit

  64 weights   x 4 bits        = 32 bytes
  1 scale      x fp16          =  2 bytes
  1 zero point x fp16          =  2 bytes
  ------------------------------------------
  total                        = 36 bytes / 64 weights

  bits per weight = 36 x 8 / 64 = 4.5 bpw

  whole model = 25.2e9 x 4.5 / 8 = 14.18e9 bytes = about 14.2 GB

That comes within 1% of the reported 14.3GB. The leftover 0.1GB is adequately explained by the 8-bit router, differences in how embeddings are handled, and metadata.

One lesson already comes out here — the phrase "4-bit quantization" does not make the file size half the parameter count. The smaller the group, the better the quality and the larger the overhead. At group 32, bpw would be 5.0 and the file would be 15.8GB. When you are planning a local deployment, that 0.5 bit is a 1.5GB difference. How to compute weights and KV cache from formulas is laid out in How Much VRAM to Run an LLM Locally.

The two kinds of weights MoE creates

Now let us look at what the 1.35GB that stays resident in RAM actually is. The README calls it the "shared core." In an MoE structure, parameters split into two classes.

• What every token uses — embeddings, attention, shared experts, the router. This has to stay resident.
• What each token selects — the routed experts. Roughly 8 out of 128 are activated per layer.

From the two numbers 3.8B active and 25.2B total, this split can be worked out backwards.

S = always-used parameters, R = routed expert parameters
active ratio f = active experts / total experts

  S + R       = 25.2B
  S + f x R   =  3.8B      (assuming f = 8/128 = 0.0625)

  0.9375 R = 21.4B  ->  R = 22.8B,  S = 2.4B

  storing S at 4.5 bpw
  2.4e9 x 4.5 / 8 = 1.35e9 bytes = 1.35 GB

That matches exactly the 1.35GB shared core the README states. It means the structural assumption of 8 active experts out of 128 was right, and at the same time it shows the core of this design. Keep only 5% of the whole resident, and the remaining 95% only has to be fetched a piece at a time, per token.

The point to stress here is that this approach works because Gemma 4 is an MoE. Had it been a dense model of the same size, every token would have to read all 25.2B, and no SSD could carry that. Streaming inference is not a quantization technique but a door the architecture opened.

Why they dropped mmap and went to pread

The first method that comes to mind for lazily loading weights is mmap. Map the file into the address space, access it, and the kernel fetches the pages on its own. The code is simple, and most local inference engines, llama.cpp included, use this approach.

But according to what the author stated in the HN thread, the mmap implementation stalled at 0.5 tokens per second, and switching to parallel pread calls brought it up to about 4 tokens. That is eight times. The reason is that the two approaches have different concurrency models.

If a page accessed through mmap is not in RAM yet, a page fault occurs and that thread stops until the fault is resolved. The kernel's readahead assumes sequential access, but the access pattern that expert routing produces is random reads scattered across the whole file. The prediction is wrong, so you end up waiting one at a time, every time. pread, by contrast, is an explicit request, so you can throw the whole list of needed offsets at it at once and fill the SSD's queue depth. NVMe is not a device that services one request quickly but one that services dozens concurrently to fill its bandwidth, so this difference shows up directly as eight times.

The actual implementation stacks two more things on top. It puts a 16-slot LFU cache on each layer to hold on to frequently used experts, and in the prefill stage it batches up to 128 tokens at a time so that an expert fetched once handles multiple tokens. And while the CPU reads in the next expert, Metal computes the shared expert branch — the classic technique of overlapping I/O with computation.

285MB per token — bandwidth sets the ceiling

Now the performance ceiling can be computed.

all routed experts   = 22.8e9 x 4.5 / 8 = 12.8 GB
active ratio / token = 8 / 128 = 6.25%

  bytes to read per token with no cache at all
  12.8 GB x 0.0625 = 800 MB

  measurement the author reported on HN: 250~320 MB per token

  back-derived cache hit rate = 1 - (250~320) / 800 = 60% ~ 69%

The value the author reported separately was 59 to 69% on the M2, about 67% with the 16-slot cache. That agrees with the 60 to 69% I derived. In other words, the numbers scattered across the README and HN come out of one consistent model. Take the effective transfer per token as about 285MB and the per-device ceiling works out like this.

The M2 Air side computes cleanly. At 83 milliseconds of disk time per token the ceiling is 12 tokens per second, and adding the remaining compute time makes the measured 5 to 6 tokens natural.

The problem is the M5 Pro row. The ceiling computed from the quoted SSD bandwidth is about 22 tokens, but the measurement is 31 to 35 tokens. Reading 285MB in 12 milliseconds per token would require 23.75GB per second, and that is a speed no consumer NVMe can deliver. It is not that the arithmetic is broken; it means a substantial part of that read is not coming from the SSD.

Resident memory and working set — where 2GB is true and where it misleads

The answer is macOS's unified buffer cache. File pages read with pread are held in RAM by the kernel, and reading the same offset again does not touch the SSD. On a 24GB machine, a large part of a 14.3GB model file can stay in cache. After warm-up, a "disk read" is really closer to a RAM copy.

But these pages are not counted in the process's resident memory. File-backed pages belong to the kernel's page cache, and the kernel reclaims them when memory pressure arrives. So in Activity Monitor the process still shows 2GB. The claim is not false — it is just counting something different.

To sum up.

• The true part: the weights this runtime holds in its own heap amount to the 1.35GB shared core plus roughly a 4K KV cache, and the process RSS is around 2GB. The bare fact that a 26B-class model runs at all on a machine with only 8GB of RAM is an achievement of this design.
• The misleading part: the memory the whole system uses is not 2GB. To get performance, the kernel page cache has to be holding a large chunk of the model file, and that memory competes with other apps. Open twenty browser tabs, push the cache out, and throughput falls to the SSD bandwidth ceiling. The gap between the M2 Air's 5 to 6 tokens and the M5 Pro's 31 to 35 is not only a difference in chip performance but also a difference in how much spare RAM is available to use as cache.
• So, the part to doubt: the objection raised on HN that the gain comes from OS caching rather than from the technique is half right. But the eightfold jump from mmap to pread is not explained by caching, so the technique's contribution is real too. Both factors are present.

And there is one more item where the arithmetic does not close. A 16-slot cache per layer is 12.5% of the 128 total, so if it were fully resident across all layers it would come to about 1.6GB, 12.5% of 12.8GB. Add the 1.35GB shared core and it goes past 2GB. Either the actual implementation counts slots differently, or a large part of that buffer is file-backed and therefore does not show up in RSS — but I could not read the source to confirm which. Either way the conclusion points the same direction — 2GB is a number from the process's accounting ledger.

The quality cost of 4-bit, and where this approach fits

Finally, the part that is not free. The widely quoted summary on the quality loss of 4-bit quantization is that perplexity gets 1 to 3% worse relative to FP16, and that the loss is smaller the larger the total parameter count. There is also a claim that because MoE has a large total parameter count it withstands 4-bit relatively well even with a small active count. But these are generalities repeated in aggregator posts, not measurements of this particular build.

To put it precisely. The public benchmarks for Gemma 4 26B-A4B are those of the unquantized model. Numbers the model card carries, such as MMLU Pro 82.6, GPQA Diamond 82.3, and LiveCodeBench v6 77.1, are not the scores of this build squeezed down to 4.5 bpw. And nobody has measured this build's scores yet. The README only attaches the sentence that the model may repeat itself or give wrong answers, so check important results. Quoting the original model card's numbers when discussing the quality of a quantized local model is exactly the kind of thing this post argues we should not do.

So where does this approach fit?

• Where it fits: offline work on a RAM-starved machine, on tasks that are not latency-sensitive. Document summarization, local code explanation, environments with no network. Five tokens per second is frustrating for conversation but plenty for a background batch. If the data must not leave the machine, there are not many alternatives.
• Where it does not fit: machines with enough RAM. At 24GB or more, simply loading the whole 14.3GB model is simpler and faster. The reason this approach exists is a RAM shortage, not performance. And an always-on server is even further from a fit — the runtime explicitly states that only a single process per instance owns the model and that concurrent execution is not supported.
• What has not been confirmed: there is no quantitative data on SSD lifetime. Since it is read-heavy, write wear does not look like it will be large, but the repository also states that it has not quantified this. macOS 26 and Metal 4 are also requirements, and using it on macOS 15 needs a workaround that gives up the 2.4x prefill acceleration. There is also no evidence yet that the same results hold across every generation from M1 to M5.

Closing — check what a number counts first and most of the magic explains itself

"26B in 2GB" was not an exaggeration. The 4.5 bpw that 4-bit group 64 produces, the 1.35GB shared core derived backwards from 25.2B total and 3.8B active, the 800MB per token that is 6.25% of the 12.8GB of routed experts, and the effective transfer that caching brings down to 285MB — these four numbers connect in a single line and agree with the reported values. It is good engineering.

At the same time, 2GB is the process resident set, not the system working set. The fact that the measured M5 Pro throughput exceeds the SSD bandwidth ceiling is the evidence for that, and what fills the gap is the kernel page cache. So when you put this project on an 8GB Mac, the number to expect is not 31 to 35 tokens but 5 to 6.

The question to ask when reading performance claims about local inference is always the same. What is that number counting, and where is what it did not count?