Chaos and Order

💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction — Don't Look for a Table, Memorize the Formulas
What Fills Your VRAM — Three Buckets
Computing the Weights — "4-bit" Isn't 4 Bits
- Verifying the Formula
Why Q4_K_M Is 4.89, Not 4.5
The KV Cache — The Part That Grows With Context
So What Actually Fits on Your Card
How Much Quality Does Quantization Cost — Measured Values Only
- The Warnings the README Attaches, Verbatim
- The Same Quant Hurts Different Models Differently
If a tok/s Number Comes Without Hardware, Throw It Out
Do Fewer Bits Mean More Speed — Only Up to a Point
- But the Formula Breaks at the Bottom
When the Math Changes — MoE and Hybrid Attention
So, What Are You Giving Up
- Decision Rules
- When Not to Go Local
Closing
References

Introduction — Don't Look for a Table, Memorize the Formulas

"How many GB of VRAM do I need to run an 8B model?" The internet's standard answer to this is a table copied from somewhere. That table usually has no source, doesn't state the context length, is vague about which quantization, and above all doesn't hold for your case. You pick the model, the context, and the quantization — the table is just a snapshot of one combination somebody chose once.

So let me give you the answer first. VRAM comes in two big chunks, and each has a formula.

(1) weights = parameter_count × bpw / 8      [bytes]
       bpw = bits per weight (the "real" bit count of a quant level)

(2) KV cache = 2 × num_layers × num_KV_heads × head_dim × bytes/elem × num_tokens
       2 = K and V, two copies
       bytes/elem = 2 for f16, 1 for q8_0

(1) is a constant. It's fixed the moment you pick a model and a quantization. (2) grows in direct proportion to context. And the reason most people hit a wall with local LLMs is (2), not (1). To state the conclusion up front — quantize Llama-3.1-8B to Q4_K_M and the weights are 4.58 GiB, but if you use this model's full 128K context with an f16 KV cache, the cache alone is exactly 16 GiB. That's 3.5× the weights.

This post sets up these two formulas and validates them directly against llama.cpp's source code and its official published tables. Then it covers the honest part — how much quality you lose when you drop bits (measured values only), and whether dropping further actually makes things faster (it doesn't).

Related: the constraints of running inside the browser are covered separately in the llama.cpp WebGPU backend piece (there the bottleneck is not VRAM but tab memory), and the lay of the land for quantization techniques themselves — what GPTQ, AWQ, FP8, and MXFP4 each are — is in The Complete Guide to LLM Quantization. This post focuses on one thing only: the memory arithmetic.

What Fills Your VRAM — Three Buckets

First, the terrain. Three things go into GPU memory.

Weights. The model file itself. Fixed, independent of context.
KV cache. The stored attention K and V for every token already processed. Directly proportional to token count.
The rest. Temporary compute buffers, the CUDA context, and — if you've hung your desktop off the same card — the display output too.

You can't write #3 as a formula. It depends on the backend, the batch size, and whether flash attention is on. I won't make up a number here — instead, llama.cpp prints it directly when it loads a model. Read the KV self size and compute buffer size lines in the log. This post focuses on computing (1) and (2), the lower bound, exactly. You need the lower bound before you can judge how much #3 piled on top.

Computing the Weights — "4-bit" Isn't 4 Bits

Formula (1) looks easy, but the trap hides in bpw. The bpw of "4-bit quantization" is not 4.

The reason is in how quantization works. To turn a weight into a 4-bit integer, you also have to store a scale that maps those integers back to real values. That scale is not 4 bits. So the real bit count is always larger than the nominal one.

You don't have to guess how much larger. It's written right there in ggml's block structs. Take the simplest, Q8_0, in ggml/src/ggml-common.h:

#define QK8_0 32
typedef struct {
    ggml_half d;       // delta
    int8_t  qs[QK8_0]; // quants
} block_q8_0;

Here's how to read it. 32 weights (QK8_0) make one block, and each block holds 32 int8s (32 bytes) plus one ggml_half scale (2 bytes). So:

Q8_0 = (32 × 8 + 16) bits / 32 weights
     = 272 / 32
     = 8.5 bits per weight

Nominally 8 bits, actually 8.5. One fp16 scale per 32 weights adds exactly 0.5 bits.

Now look at a k-quant. Q4_K groups 256 weights (QK_K) into one super-block:

#define QK_K 256
#define K_SCALE_SIZE 12
typedef struct {
    ggml_half d;                  // super-block scale                  2 bytes
    ggml_half dmin;               // super-block scale for the min      2 bytes
    uint8_t scales[K_SCALE_SIZE]; // scales & mins, quantized to 6 bits 12 bytes
    uint8_t qs[QK_K/2];           // 4-bit quant values                 128 bytes
} block_q4_K;

That sums to 144 bytes for 256 weights, so 144 × 8 / 256 = 4.5 bpw. And this isn't my math — the source comment says it outright: right above block_q4_K sits // Effectively 4.5 bits per weight. There are five of them in the same file:

"Effective" bpw stated directly in the ggml-common.h comments
  Q2_K   2.625      Q3_K   3.4375     Q4_K   4.5
  Q5_K   5.5        Q6_K   6.5625
  (Q8_0 has no comment, but the struct gives 8.5 outright)

Verifying the Formula

Up to here it's derivation. We should check it. llama.cpp's official docs (tools/quantize/README.md) publish per-quant bpw and file size for Llama-3.1-8B. Invert the formula to back out the parameter count — parameter count = size × 8 / bpw — and every row should give the same value:

quant      published bpw   published size (GiB)   back-computed params
F16          16.0005          14.96                  8.031 B
Q8_0          8.5008           7.95                  8.033 B
Q6_K          6.5633           6.14                  8.036 B
Q5_K_M        5.7036           5.33                  8.027 B
Q4_K_M        4.8944           4.58                  8.038 B
Q3_K_M        3.9960           3.74                  8.040 B
Q2_K          3.1593           2.95                  8.021 B
IQ1_S         2.0042           1.87                  8.015 B

They all cluster at 8.02–8.04B. And Llama-3.1-8B's actual parameter count is embedded as an integer in the Hugging Face safetensors metadata — 8,030,261,248. The formula holds. Within rounding error, every row hits the real value.

There's an even better check. Put the value derived from the block struct next to the value llama.cpp actually measured and published:

                 from struct    published measured    diff
Q8_0               8.5000           8.5008          +0.0008
Q6_K               6.5625           6.5633          +0.0008
Q5_K_S             5.5000           5.5704          +0.0704
Q5_K_M             5.5000           5.7036          +0.2036
Q4_K_S             4.5000           4.6672          +0.1672
Q4_K_M             4.5000           4.8944          +0.3944
Q3_K_S             3.4375           3.6429          +0.2054
Q3_K_M             3.4375           3.9960          +0.5585
Q2_K               2.6250           3.1593          +0.5343

Q8_0 and Q6_K match to three decimals (8.5000 vs 8.5008). Pure block arithmetic predicted the published figure. But the rest overshoot by as much as 0.56 bits. That overshoot is the subject of the next section.

Why Q4_K_M Is 4.89, Not 4.5

_S (small) and _M (medium) are not quantization types but mixing recipes. Rather than stamping every tensor as Q4_K, they leave some tensors at higher precision.

The recipe is spelled out in src/llama-quant.cpp. For Q4_K_M, rules like these apply:

The attention attn_v tensor — promoted to Q6_K on some layers (those where use_more_bits is true)
The FFN ffn_down tensor — likewise promoted to Q6_K on some layers
The attention QKV tensors — promoted to Q5_K

So Q4_K_M is "mostly 4.5 bits, but a few quality-sensitive tensors at 6.5 bits." That's why the average rises from 4.5 to 4.8944. The source comment explains the design philosophy well — to paraphrase the comment on a 70B model's attn_v: because 8 heads share the attn_v weights, that tensor is 8× smaller than attn_q, so you can raise quantization accuracy quite a bit while barely growing the model.

One practically important conclusion here. This does not mean _M is always better than _S. _M buys quality by spending more bits, and whether that trade is worth it you can only tell from the quality numbers later. And why Q4_K_M has calcified into the local camp's default also comes later.

The KV Cache — The Part That Grows With Context

Now the real problem.

Rewrite formula (2):

KV cache = 2 × L × n_kv_heads × head_dim × bytes × n_tokens

You can confirm this formula against the llama.cpp source too. src/llama-kv-cache.cpp builds the K and V tensors per layer like this:

const uint32_t n_embd_k_gqa = hparams.n_embd_k_gqa(il);
ggml_tensor * k = ggml_new_tensor_3d(ctx, type_k, n_embd_k_gqa, kv_size, n_stream);
ggml_tensor * v = ggml_new_tensor_3d(ctx, type_v, n_embd_v_gqa, kv_size, n_stream);

And the definition of n_embd_k_gqa is in src/llama-hparams.cpp:

uint32_t llama_hparams::n_embd_k_gqa(uint32_t il) const {
    const uint32_t n_head_kv = this->n_head_kv(il);
    return n_embd_head_k(il) * n_head_kv;
}

So a vector of size head_dim × n_kv_heads is allocated per token (kv_size), in two copies for K and V, per layer. Exactly formula (2).

The key here is n_kv_heads. The KV head count, not the attention head count. Almost all modern models use GQA (Grouped-Query Attention) to shrink KV heads dramatically. Llama-3.1-8B has 32 attention heads but only 8 KV heads. Put 32 in here and your answer is off by 4×. This is exactly where the internet tables so often go wrong.

Working It Out for Llama-3.1-8B

The config values are in config.json — 32 layers, 32 attention heads, 8 KV heads, hidden 4096, max context 131,072. There's no separate head_dim, so it's hidden / n_heads = 4096 / 32 = 128. (Note: Meta's official repo requires access approval, so anonymous lookups are blocked. The values above and the parameter count later were read from a mirror repo, with the source noted in the references.)

per token = 2 × 32 layers × 8 KV heads × 128 × 2 bytes (f16)
          = 131,072 bytes
          = 128 KiB / token

full 128K context (131,072 tokens):
  131,072 bytes × 131,072 tokens = 17,179,869,184 bytes = exactly 16 GiB

128 KiB per token. Easy to remember. And fill all of 128K and it's 16 GiB, right on the nose.

Now set that beside the earlier number:

Llama-3.1-8B, Q4_K_M
  weights        4.58 GiB   (fixed)
  KV @  4K        0.5 GiB
  KV @ 32K        4.0 GiB   ← already matches the weights here
  KV @ 128K      16.0 GiB   ← 3.5× the weights

What stops you is not the model but the context. Succeeding in shrinking an 8B model to 4 bits does not, in any way, mean you can use a 128K context.

Other Models Differ — Which Is Why You Need the Formula

Even the same "8B" gives different values. Qwen3-8B's config.json has 36 layers, 8 KV heads, head_dim 128, max context 40,960.

Qwen3-8B: 2 × 36 × 8 × 128 × 2 = 147,456 bytes = 144 KiB / token
          use the full 40,960-token context → 5.625 GiB

Per token it eats 12.5% more than Llama (because it has 36 layers), but with a max context of 40,960 the total is far smaller. The "8B" label tells you nothing about the KV cache. Layer count, KV head count, head_dim, and the context you'll actually open — those four decide it.

You Can Quantize the KV Cache Too

llama.cpp gives you the option to store the KV cache itself at lower precision. Straight from tools/server/README.md:

-ctk, --cache-type-k TYPE — allowed values: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 (default f16)
-ctv, --cache-type-v TYPE — same allowed values, same default

Switch to q8_0 and bytes-per-element drops from 2 to 1, exactly halving the KV cache. Llama-3.1-8B's 128K cache goes from 16 GiB down to 8 GiB. One thing to note in the allowed-values list — k-quants (like q4_K) are not on it. The KV cache can only use the legacy quantizations.

There is a quality cost here, but that cost is not as well-measured and published as weight quantization is. The latest research in this direction is covered separately in the INT4 KV-cache quantization piece. Here we just nail down the arithmetic — the memory is exactly halved — and move on.

So What Actually Fits on Your Card

Now assemble it. If you load Llama-3.1-8B as Q4_K_M (4.58 GiB) and spend all the remaining space on KV, how much context can you open?

Caution — the below is a theoretical upper bound, minus the compute buffer and display output. In practice it shrinks further from here. That portion, as said, I won't invent — you have to read it from your own log.

VRAM     minus weights    f16 KV context (upper bound)    with q8_0 KV
16 GiB      11.4 GiB           ~ 93,000 tokens              ~187,000
24 GiB      19.4 GiB           ~159,000 tokens              ~318,000
32 GiB      27.4 GiB           ~224,000 tokens              ~449,000

How to read it matters.

16GB card: the 8B model fits comfortably (4.58 GiB). But a 128K context, in f16, has an upper bound of 93,000 tokens, so you can't open it. Subtract the compute buffer and you can open even less. Turn on -ctk q8_0 -ctv q8_0 and only then does 128K come within range.
24GB card (RTX 3090/4090): in f16, 128K does come under the bound (159,000), but not with much headroom.
32GB card (RTX 5090): f16 128K fits comfortably.

One-line summary — "does the model fit on the card" is the wrong question. The right question is "does the model plus the context I'll actually use fit?"

How Much Quality Does Quantization Cost — Measured Values Only

From here I'll stop asserting and only cite.

llama.cpp publishes measured per-quant quality loss in tools/perplexity/README.md. This table is good because it states its conditions:

Source: llama.cpp tools/perplexity/README.md — "LLaMA 3 8b Scoreboard"
  Revision : f364eb6f
  Backend  : CUDA
  CPU      : AMD Epyc 7742
  GPU      : 1x NVIDIA RTX 4090
  Dataset  : Wikitext-2 test set

First, one big warning. Query that revision f364eb6f through the GitHub API and the commit date is April 30, 2024. So although this table sits in the current master's README, the data itself came from a build over two years old. llama.cpp's quantization implementation kept changing in the meantime. So use these numbers to read the relative ranking between quant levels, and don't carry the absolute values straight over to today's build. That said, this is the only public, condition-stated data the project measured and published itself.

Two metrics to watch. PPL (perplexity) — lower is better. KLD (KL divergence) — how much the probability distribution diverged from fp16, where 0 is identical. The README itself sorts the table trusting KLD more.

quant      size (GiB)    PPL        ΔPPL       KLD        mean Δp
f16          14.97    6.233160        —      0.000551     0.001 %
q8_0          7.96    6.234284   0.002650   0.001355    -0.019 %
q6_K          6.14    6.253382   0.021748   0.005452    -0.007 %
q5_K_M        5.33    6.288607   0.056974   0.010762    -0.114 %
q4_K_M        4.58    6.407115   0.175482   0.031273    -0.596 %
q3_K_M        3.74    6.888498   0.656864   0.101913    -1.990 %
q2_K          2.96    9.751568   3.519934   0.445132    -9.123 %
iq1_S         1.88   58.097760  51.866126   2.211278   -32.471 %
   (no imatrix, except iq1_S which uses the "WT 1m" imatrix)

How to read this table.

q8_0 is essentially free. PPL goes from 6.233 to 6.234. Half the size, and KLD is 0.0014. There's almost no reason to insist on fp16.
q6_K and q5_K_M are very cheap too. Mean Δp (the change in probability of the correct token) is -0.007% and -0.114% respectively.
q4_K_M is the knee. Size is about 31% of fp16, and mean Δp is -0.596%. This is why Q4_K_M became the local camp's default.
q3_K_M starts to visibly hurt. PPL jumps from 6.41 to 6.89.
q2_K becomes a different model. PPL 9.75, mean Δp -9.1%.
iq1_S collapses. PPL 58.1. The original was 6.23, so that's 9.3×.

The Warnings the README Attaches, Verbatim

Before you use these numbers, there are caveats the doc attaches itself. Rendered faithfully to the original's meaning:

Perplexity cannot be compared directly across models, especially when tokenizers differ.
Fine-tuning usually raises perplexity even when human-rated output quality goes up.
llama.cpp's numbers depend heavily on implementation details, so they can't be compared directly against another project's numbers.

The third matters especially. You must not line up this table's PPL against the PPL in a GPTQ or AWQ paper. They are self-measurements of different implementations, not a common benchmark.

And the second warning has a large implication — perplexity is only a proxy for quality, not quality itself. The sentence "q4_K_M's mean Δp is -0.596%" does not mean "your code-generation task gets 0.596% worse." Nobody measured that for you. You have to measure it with your own task.

The Same Quant Hurts Different Models Differently

This is the least-known, and most important, fact in this table. The same doc's "LLaMA 2 vs. LLaMA 3 Quantization comparison" table (same revision, same 4090) shows:

Mean PPL ratio (1.0 = no loss)
                    L2 7b        L3 8b
  q8_0            1.000689     1.000425
  q6_K            1.002406     1.003490
  q4_K_M          1.014242     1.028160      ← 1.4% vs 2.8%, double
  q2_K            1.107955     1.564849      ← 10.8% vs 56.5%, five times

Mean KLD
  q2_K            0.108903     0.445132      ← 4.1×

The same q2_K raises PPL by 10.8% on Llama-2 7B but by 56.5% on Llama-3 8B. q4_K_M too, at 1.4% vs 2.8%, exactly double.

The implication is large. The sentence "Q4_K_M's quality loss is negligible" is not true without qualifying the model. The more heavily a model was trained, on more tokens, the less slack in its weights, and so the more it loses at the same bit budget. The models of 2026 are trained even more than Llama-3. So the figures in the table above are likely wrong on the optimistic side for today's newest models. There's no way around re-measuring on your own model.

For reference, the README leaves an honest observation about imatrix (importance matrix) too — using more Wikitext tokens to build the imatrix did not produce a consistent improvement. The imatrix itself helps (q4_K_M's KLD goes from 0.0313 to 0.0282), but "more tokens = better imatrix" is not supported by this data.

If a tok/s Number Comes Without Hardware, Throw It Out

Now speed. And what I most want to say here is about methodology.

For a single tok/s to mean anything, at least four things must come with it — the exact hardware (GPU model name), the model, the quantization, and the context length (or batch length). Miss even one of the four and the number tells another person nothing.

Let me show you this isn't nagging. llama.cpp's official README (tools/quantize/README.md) carries this table:

Measure                       Q4_K_M          Q8_0            F16
bits/weight                   4.8944          8.5008        16.0005
size (GiB)                      4.58            7.95          14.96
prompt processing t/s @ 512   821.81 ±21.44   865.09 ±8.30   923.49 ±0.53
text generation t/s @ 128      71.93 ±1.52     50.93 ±0.08    29.17 ±0.04

The model is stated (Llama-3.1-8B). The quantization is stated. The context length is stated (@ 512, @ 128). Error bars are even attached. And yet the hardware is nowhere. Comb the whole doc and there's no GPU, no CPU, no backend, no build revision. Not even in the body of the PR that last edited this table (#24133, merged June 5, 2026).

So the sentence "Q4_K_M is 71.93 tok/s" is useless on its own. On a 3090? A 5090? An M4 Max? A 3090 and a 5090 differ by 1.9× in memory bandwidth. Given that the same project writing the same kind of doc stated CPU, GPU, backend, and revision in the perplexity README, this is closer to a documentation mistake.

But — and this is the fun part — there is still something you can read off this table. You can't read the absolute values, but since every row came from the same (unknown) hardware, the ratios between rows are valid. That's the next section.

Do Fewer Bits Mean More Speed — Only Up to a Point

First, why quantization speeds things up at all. When generating tokens at batch 1, the model reads the entire weight set from memory once per token. There's little compute but a lot to read. So decoding is memory-bandwidth-bound. From this comes a ceiling formula.

tok/s ceiling = memory bandwidth / model size      (dense model, batch 1)

Let's compute it with the bandwidth specs from NVIDIA's RTX Blackwell architecture whitepaper. Straight from the whitepaper body — the RTX 5090 carries 28 Gbps GDDR7 and delivers 1.792 TB/sec of peak memory bandwidth. The same whitepaper's comparison table lists the 3090 at 936 GB/sec and the 4090 at 1008 GB/sec.

Theoretical ceiling for Llama-3.1-8B Q4_K_M (4.58 GiB)
  RTX 3090 (936 GB/s)  →  190 tok/s
  RTX 4090 (1008 GB/s) →  205 tok/s
  RTX 5090 (1792 GB/s) →  364 tok/s

This is the ceiling. Real measurements are always below it. But this formula tells you, for your card, "you can't be faster than this," and lets you judge which benchmark numbers physically make sense. If some post claims it ran 8B Q4_K_M at 300 tok/s on a 4090, either it wasn't batch-1 decoding or it measured something else.

But the Formula Breaks at the Bottom

Now use the "ratios are valid" point from before. If decoding were purely bandwidth-bound, size × tok/s should give a constant value (= effective bandwidth) across all quantizations. Let me multiply it out directly with the numbers from llama.cpp's official table. (The multiplication below is my own computation from their published figures.)

quant       size (GiB)   gen tok/s    size × tok/s
F16           14.96        29.17          436
Q8_0           7.95        50.93          405
Q6_K           6.14        58.67          360
Q5_K_M         5.33        67.23          358
Q4_K_M         4.58        71.93          329
Q3_K_M         3.74        71.68          268
Q2_K           2.95        79.85          236
IQ2_XXS        2.23        79.86          178
IQ1_S          1.87        79.73          149

Not constant. It keeps falling, from 436 to 149. It means the lower you drop the bits, the less of the bandwidth you're using — there's less data to read, but the dequantize work to unpack it grows, so the bottleneck moves from bandwidth to compute.

Seen as ratios, the result is painfully clear.

F16    → Q4_K_M :  3.27× smaller, 2.47× faster    (trade holds)
Q4_K_M → Q2_K   :  1.55× smaller, 1.11× faster    (bad trade)
Q4_K_M → IQ1_S  :  2.45× smaller, 1.11× faster    (worst trade)

Drop from Q4_K_M to IQ1_S and the model gets 2.45× smaller but only 11% faster. Meanwhile quality — from the table in the previous section — goes from PPL 6.41 to 58.1 (9.1×) and KLD 0.031 to 2.21 (71×). You get a model 9× worse in exchange for 11% speed.

An honest caveat. This speed table (Llama-3.1-8B, hardware unknown) and the earlier quality table (Llama-3 8B, RTX 4090, April 2024 revision) are different runs. The model differs subtly and the build differs. So reading across the two tables isn't a rigorous comparison but a look at direction. Even so, the internal ranking within each table is solid, and those rankings point the same way.

So the only legitimate reason to go to very low bits is not speed but "nothing else fits." Choosing IQ1_S for speed doesn't add up.

When the Math Changes — MoE and Hybrid Attention

The two formulas have assumptions. Recent models break them.

MoE (Mixture of Experts). Quoting OpenAI's gpt-oss-20b model card directly, this model is "21B parameters with 3.6B active parameters." Here the two formulas split.

Memory needs the whole thing. Since you don't know in advance which expert gets picked, you have to load all 21B.
Bandwidth reads only the active part. Only 3.6B worth is read per token.

So in the earlier ceiling formula bandwidth / model size, the denominator changes from total size to active size. This is why MoE is "heavy but fast." You pay 21B for memory and get 3.6B for speed. The same model card states that gpt-oss-20b quantizes its MoE weights with MXFP4 and runs within 16GB of memory.

Hybrid attention. Look at gpt-oss-20b's config.json and layer_types alternates sliding_attention and full_attention, with sliding_window at 128. A sliding-window layer only needs to hold the KV of the most recent 128 tokens, so that layer's KV does not grow with context. Formula (2)'s n_tokens becomes per-layer.

The point is that this isn't an exception. The llama.cpp source already reflects it — the reason the n_embd_k_gqa(il) we saw earlier takes a layer index il is precisely that the KV head count can differ per layer. No table can replace opening the model's config.json.

So, What Are You Giving Up

To sum up. What you give up with a local LLM is three things.

First, quality. In measured terms — q8_0 is essentially free (KLD 0.0014), through q5_K_M it's still cheap (KLD 0.011), q4_K_M is the knee (KLD 0.031), q3_K_M starts to hurt (KLD 0.102), and below q2_K it's a different model (KLD 0.445+). But this ranking is based on an April 2024 build of Llama-3 8B, and the more heavily trained a model, the more it loses at the same bits.

Second, context. This is the most underrated item. Loading an 8B model onto a 16GB card is easy. Giving that model a 128K context in f16 is impossible. Most people get stuck at "the model loaded, so why can't I feed it a long document?"

Third, predictability of speed. Someone else's tok/s is not your tok/s. Throw out numbers with no hardware stated, and even for stated ones, recompute the ceiling with your card's bandwidth.

Decision Rules

I won't end with "it depends." Do this.

Start quantization at Q4_K_M. It's the quality/size knee, and dropping lower adds almost no speed. If your card has headroom, moving up to Q5_K_M or Q6_K is always better than dropping below Q4 and leaving the freed space unused.
After fixing the weights, treat the remaining VRAM as the KV budget and back out the context. Don't do it the other way. If the context is fixed first, that decides the quantization.
If the KV doesn't fit, turn on -ctk q8_0 -ctv q8_0 before shaving the weights further. The cache halves exactly. This usually beats dropping the weights from Q4 to Q3.
Open config.json every time you switch models. Layer count, KV head count, head_dim. Thirty seconds.
Measure once with your card, your model, your context. And once you've measured, state the hardware and share it.

When Not to Go Local

Honestly — local often isn't the answer.

When quality is the top priority. A 4-bit 8B is not a frontier model. That gap can't be closed with quantization.
When long context is the essence. The KV arithmetic beats you. If you always use 128K, a single local card is the wrong tool.
When there are concurrent users. The formulas above are all batch-1. Take multiple requests at once and the KV cache multiplies by the number of users.
When you use it occasionally. The price of a card equals a whole lot of API tokens.

Conversely, the cases where local wins are clear — when the data must not leave the device, when it has to be offline, when the call volume is large and predictable so the fixed cost amortizes, and when latency matters more than a network round trip. And as the hands-on log of running small models on an RTX 5090 shows, the tasks a small model handles well are more numerous than you'd think.

Closing

The right answer to the local-LLM VRAM question is not a number but two formulas.

weights  = parameter_count × bpw / 8
KV cache = 2 × layers × KV_heads × head_dim × bytes × num_tokens

The first is fixed, the second grows. And as this post confirmed, the second easily overtakes the first — Llama-3.1-8B Q4_K_M's weights are 4.58 GiB, but the f16 KV for a 128K context is 16 GiB.

The reason you can trust these formulas is not that I say so. It's that the 8.5 bpw for Q8_0 derived from the ggml block struct matches llama.cpp's published 8.5008, and the parameter count of 8.02–8.04B reverse-computed from that formula matches the real value of 8,030,261,248. When a formula reproduces the vendor's own published figures, that formula works for your case too. A table does not.

One last piece of methodology. The problem I held onto longest in this post was that the speed table in llama.cpp's official README has no hardware. That table stated the model, the quantization, the context length, even the error bars — but no GPU name. So "71.93 tok/s" becomes an uncitable number. When you share a benchmark — GPU, model, quantization, context length. Write all four. A number without the four is one nobody can use.