The State of LLM Quantization — From GPTQ and AWQ to FP8, MXFP4, and KV-Cache Quantization

Introduction — Why Shrink Bits, and Why Does It Hold Up?
Notation First — What Does W4A16 Mean?
The INT Classics — a History of Outlier Diplomacy
The Local Camp — GGUF k-quants, and QLoRA's NF4
2026's Center of Gravity — FP8 and Microscaling FP4
The Next Bottleneck — KV-Cache Quantization
I Measured It — fp16 vs int8 vs nf4 on an RTX 5090
Selection Guide — Hardware × Goal Matrix
Closing — A War History Against Outliers
References

Introduction — Why Shrink Bits, and Why Does It Hold Up?

Load a 70B model in fp16 and the weights alone take 140 GB — two H100s. Quantize the same model to 4 bits and it's about 35 GB — it fits on one card with room to spare. Quantization cuts the bits used to represent parameters, slashing memory, bandwidth, and cost — the number-one weapon in the "fight that outlives training" from the serving section of the model development lifecycle.

The surprise is how well quality holds. Two reasons. First, neural-network weights cluster around zero, so few bits can represent them densely. Second, the real problem is a small number of outliers — unusually large values — and every modern technique is a different answer to "how shall we treat the outliers?" Hold that one lens and the methods below read as a single lineage.

Two terms first: converting a finished model as-is is PTQ (Post-Training Quantization); retraining the model to tolerate low precision is QAT (Quantization-Aware Training). Most of this article is PTQ — the practical mainstream.

Notation First — What Does W4A16 Mean?

Quantization discussions constantly use W{bits}A{bits} notation.

W4A16   = Weights 4-bit, Activations 16-bit
          → stored/loaded at 4 bits, dequantized to compute at 16 bits
W8A8    = weights AND activations 8-bit → arithmetic itself is 8-bit (needs HW support)
W4A4KV4 = weights, activations, and the KV cache all at 4 bits

Weight-only schemes (W4A16) save memory and bandwidth (and speed up small-batch inference, since decode is memory-bound); quantizing activations too (W8A8/W4A4) also raises compute throughput. But activations have far nastier outliers — that's the problem SmoothQuant solves below.

The INT Classics — a History of Outlier Diplomacy

LLM.int8() (2022) — a mixed decomposition: "compute the outlier channels separately in fp16, everything else in INT8." It first showed lossless 8-bit inference for large models and formalized the outlier problem.

GPTQ (2022) — the PTQ that opened the 4-bit era. It quantizes weights column by column, and compensates each column's error into the not-yet-quantized remaining weights (second-order, Hessian-based), suppressing error accumulation. With only a small calibration set, it made W4 practical.

AWQ (2023) — flipped the perspective. Not all weights are equal: the ~1% of weights multiplied by large activations dominate quality. AWQ protects those salient channels via scaling and boldly drops the rest to 4 bits. No retraining, fast, and kernel-friendly — the de facto INT4 standard in vLLM-style GPU serving.

SmoothQuant (2022) — the bridge to W8A8. It mathematically migrates the activation outlier scale into the weights (smoothing activations, roughening weights slightly) so both can be computed in 8 bits.

The Local Camp — GGUF k-quants, and QLoRA's NF4

GGUF (llama.cpp) is the common format of the local-inference ecosystem (Ollama, LM Studio). Its signature is k-quant mixed precision: different bit-widths per layer/tensor by importance, inside one model. The filename is the spec:

Q4_K_M  = 4-bit k-quant, Medium mix — the classic quality/size balance
Q5_K_M  = 5-bit — take this if you have headroom
Q8_0    = 8-bit — near-lossless, half the size savings
Q2_K    = 2-bit — extreme compression, big quality trade
(newer importance-matrix i-quants like IQ4_XS also exist)

Designed for CPU+GPU mixed inference — the workhorse of "run a 70B on a gaming PC" scenarios.

NF4 (QLoRA, 2023) — quantization on the fine-tuning side. It freezes the base model in NormalFloat4 — an information-theoretically optimal 4-bit grid for normally distributed weights — and trains only LoRA adapters on top. The technique that enabled "fine-tune a 65B on one 48 GB GPU"; remember it as a set with LoRA from the model development post.

2026's Center of Gravity — FP8 and Microscaling FP4

The biggest recent shift is the move from integers (INT) to low-precision floating point (FP). Thanks to the exponent, floating point represents large and small values simultaneously — more forgiving of outliers than integers.

FP8 — 8-bit floating point, hardware-accelerated since Hopper (H100), in range-flavored E5M2 and precision-flavored E4M3. Half the memory of fp16 with nearly no quality loss, it has become the de facto default in datacenter serving stacks (vLLM, TensorRT-LLM). "On Hopper/Blackwell, start with FP8" is today's common sense.

Microscaling FP4 — MXFP4 and NVFP4 — the key that made 4-bit floating point practical is block-level scaling: each block of ~32 values carries its own shared scale factor, shifting FP4's narrow range to the optimal position per block. The OCP-standard MXFP4 has reached public model distribution (OpenAI's gpt-oss shipping in MXFP4 was the symbolic moment), while NVIDIA's NVFP4 targets Blackwell-generation hardware acceleration with W4A4 — and even W4A4KV4, quantizing the KV cache too. "FP4 is the FP8 of the Blackwell era" is the emerging consensus.

The Next Bottleneck — KV-Cache Quantization

Once weights are shrunk, the remaining big memory consumer is the KV cache — as seen in the LLM caching post, at long context it can outgrow the weights. Hence serving engines now offer storing the KV cache itself in FP8 (or even INT4/FP4). The payoff is double: more context and more concurrent users per GPU, and since decode is memory-bound, token generation itself gets faster. For long-context services, it's the mandatory next checkbox after weights.

I Measured It — fp16 vs int8 vs nf4 on an RTX 5090

That was all theory. So I checked whether the "4-bit = a quarter of the memory" arithmetic above actually holds. On a single RTX 5090 (32GB) I loaded an 8B model (DeepSeek-R1-Qwen3-8B) at three precisions and measured post-load VRAM and decode speed (128 tokens, greedy), using bitsandbytes 0.49.2 — each precision in a separate process for a clean VRAM reading.

precision   VRAM       decode speed   note
─────────  ─────────  ─────────────  ─────────────────────────────
fp16        16.41 GB   54.8 tok/s     baseline
int8         9.60 GB   21.1 tok/s     LLM.int8() — slowest of the three (!)
nf4          7.57 GB   52.0 tok/s     4-bit, speed on par with fp16

Three things came out different from the arithmetic. First, 4-bit cut memory not to a quarter but to about half (16.4 → 7.6GB, 2.2×): only the weights become 4-bit, while activations, the KV cache, and non-quantized layers stay — "4-bit = a quarter" is a weights-only story, not total VRAM. Second, the biggest surprise: int8 was the slowest of the three (21 tok/s, under half of fp16). That is the opposite of the "fewer bits = faster" intuition — LLM.int8() uses a mixed-precision decomposition that handles outliers separately in fp16, and that overhead eats decode speed at small batch sizes. Third, nf4 is the sweet spot — the least memory while keeping speed on par with fp16 (52 tok/s). All three precisions produced coherent reasoning output.

The lesson matches this article's conclusion: "fewer bits is better" must always be measured on your own hardware. In particular, remember that bitsandbytes int8 "saves memory but can be slower." The selection guide below reads more sharply on top of these measurements.

Selection Guide — Hardware × Goal Matrix

Situation                              Recommended starting point
─────────────────────────────────    ─────────────────────────────
Datacenter with H100/Blackwell         FP8 (the serving engines' default path)
Blackwell + maximum throughput         consider NVFP4/MXFP4 (W4A4 family)
VRAM-constrained GPU (A100/4090)       AWQ INT4 (W4A16); GPTQ as alternative
Local PC / CPU-mixed / Ollama          GGUF Q4_K_M first, Q5_K_M with headroom
Cheap fine-tuning                      QLoRA (NF4 + LoRA)
Long context is the bottleneck         add KV-cache quantization (start FP8)
Quality first, memory to spare         stop at W8A8 (FP8/INT8)

Whatever you pick, the last step is the same — validate on your own task's eval set. Perplexity is only a hint; quantization damage is uneven across tasks (math, code, and long reasoning crumble first). The eval-first principle from the model development post applies verbatim. Calibration data (the small sample GPTQ/AWQ use) should also resemble your real usage distribution.

Closing — A War History Against Outliers

In hindsight, the history of quantization compresses to one sentence: "how shall we treat the outliers?" LLM.int8() gave them a separate room; GPTQ compensated their error onto neighbors; AWQ bodyguarded the salient channels; SmoothQuant migrated roughness from the hard side to the smooth side; FP8 absorbed them with an exponent; MXFP4/NVFP4 charged head-on with per-block scales. The next battlefields are the KV cache and 4-bit activations. Bits will keep shrinking — and with a validation habit, those savings are very nearly free.