Chaos and Order

💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction — Why Shrink Bits, and Why Does It Hold Up?
Notation First — What Does W4A16 Mean?
The INT Classics — a History of Outlier Diplomacy
The Local Camp — GGUF k-quants, and QLoRA's NF4
2026's Center of Gravity — FP8 and Microscaling FP4
The Next Bottleneck — KV-Cache Quantization
Selection Guide — Hardware × Goal Matrix
Closing — A War History Against Outliers
References

Introduction — Why Shrink Bits, and Why Does It Hold Up?

Load a 70B model in fp16 and the weights alone take 140 GB — two H100s. Quantize the same model to 4 bits and it's about 35 GB — it fits on one card with room to spare. Quantization cuts the bits used to represent parameters, slashing memory, bandwidth, and cost — the number-one weapon in the "fight that outlives training" from the serving section of the model development lifecycle.

The surprise is how well quality holds. Two reasons. First, neural-network weights cluster around zero, so few bits can represent them densely. Second, the real problem is a small number of outliers — unusually large values — and every modern technique is a different answer to "how shall we treat the outliers?" Hold that one lens and the methods below read as a single lineage.

Two terms first: converting a finished model as-is is PTQ (Post-Training Quantization); retraining the model to tolerate low precision is QAT (Quantization-Aware Training). Most of this article is PTQ — the practical mainstream.

Notation First — What Does W4A16 Mean?

Quantization discussions constantly use W{bits}A{bits} notation.

W4A16   = Weights 4-bit, Activations 16-bit
          → stored/loaded at 4 bits, dequantized to compute at 16 bits
W8A8    = weights AND activations 8-bit → arithmetic itself is 8-bit (needs HW support)
W4A4KV4 = weights, activations, and the KV cache all at 4 bits

Weight-only schemes (W4A16) save memory and bandwidth (and speed up small-batch inference, since decode is memory-bound); quantizing activations too (W8A8/W4A4) also raises compute throughput. But activations have far nastier outliers — that's the problem SmoothQuant solves below.

The INT Classics — a History of Outlier Diplomacy

LLM.int8() (2022) — a mixed decomposition: "compute the outlier channels separately in fp16, everything else in INT8." It first showed lossless 8-bit inference for large models and formalized the outlier problem.

GPTQ (2022) — the PTQ that opened the 4-bit era. It quantizes weights column by column, and compensates each column's error into the not-yet-quantized remaining weights (second-order, Hessian-based), suppressing error accumulation. With only a small calibration set, it made W4 practical.

AWQ (2023) — flipped the perspective. Not all weights are equal: the ~1% of weights multiplied by large activations dominate quality. AWQ protects those salient channels via scaling and boldly drops the rest to 4 bits. No retraining, fast, and kernel-friendly — the de facto INT4 standard in vLLM-style GPU serving.

SmoothQuant (2022) — the bridge to W8A8. It mathematically migrates the activation outlier scale into the weights (smoothing activations, roughening weights slightly) so both can be computed in 8 bits.

The Local Camp — GGUF k-quants, and QLoRA's NF4

GGUF (llama.cpp) is the common format of the local-inference ecosystem (Ollama, LM Studio). Its signature is k-quant mixed precision: different bit-widths per layer/tensor by importance, inside one model. The filename is the spec:

Q4_K_M  = 4-bit k-quant, Medium mix — the classic quality/size balance
Q5_K_M  = 5-bit — take this if you have headroom
Q8_0    = 8-bit — near-lossless, half the size savings
Q2_K    = 2-bit — extreme compression, big quality trade
(newer importance-matrix i-quants like IQ4_XS also exist)

Designed for CPU+GPU mixed inference — the workhorse of "run a 70B on a gaming PC" scenarios.

NF4 (QLoRA, 2023) — quantization on the fine-tuning side. It freezes the base model in NormalFloat4 — an information-theoretically optimal 4-bit grid for normally distributed weights — and trains only LoRA adapters on top. The technique that enabled "fine-tune a 65B on one 48 GB GPU"; remember it as a set with LoRA from the model development post.

2026's Center of Gravity — FP8 and Microscaling FP4

The biggest recent shift is the move from integers (INT) to low-precision floating point (FP). Thanks to the exponent, floating point represents large and small values simultaneously — more forgiving of outliers than integers.

FP8 — 8-bit floating point, hardware-accelerated since Hopper (H100), in range-flavored E5M2 and precision-flavored E4M3. Half the memory of fp16 with nearly no quality loss, it has become the de facto default in datacenter serving stacks (vLLM, TensorRT-LLM). "On Hopper/Blackwell, start with FP8" is today's common sense.

Microscaling FP4 — MXFP4 and NVFP4 — the key that made 4-bit floating point practical is block-level scaling: each block of ~32 values carries its own shared scale factor, shifting FP4's narrow range to the optimal position per block. The OCP-standard MXFP4 has reached public model distribution (OpenAI's gpt-oss shipping in MXFP4 was the symbolic moment), while NVIDIA's NVFP4 targets Blackwell-generation hardware acceleration with W4A4 — and even W4A4KV4, quantizing the KV cache too. "FP4 is the FP8 of the Blackwell era" is the emerging consensus.

The Next Bottleneck — KV-Cache Quantization

Once weights are shrunk, the remaining big memory consumer is the KV cache — as seen in the LLM caching post, at long context it can outgrow the weights. Hence serving engines now offer storing the KV cache itself in FP8 (or even INT4/FP4). The payoff is double: more context and more concurrent users per GPU, and since decode is memory-bound, token generation itself gets faster. For long-context services, it's the mandatory next checkbox after weights.

Selection Guide — Hardware × Goal Matrix

Situation                              Recommended starting point
─────────────────────────────────    ─────────────────────────────
Datacenter with H100/Blackwell         FP8 (the serving engines' default path)
Blackwell + maximum throughput         consider NVFP4/MXFP4 (W4A4 family)
VRAM-constrained GPU (A100/4090)       AWQ INT4 (W4A16); GPTQ as alternative
Local PC / CPU-mixed / Ollama          GGUF Q4_K_M first, Q5_K_M with headroom
Cheap fine-tuning                      QLoRA (NF4 + LoRA)
Long context is the bottleneck         add KV-cache quantization (start FP8)
Quality first, memory to spare         stop at W8A8 (FP8/INT8)

Whatever you pick, the last step is the same — validate on your own task's eval set. Perplexity is only a hint; quantization damage is uneven across tasks (math, code, and long reasoning crumble first). The eval-first principle from the model development post applies verbatim. Calibration data (the small sample GPTQ/AWQ use) should also resemble your real usage distribution.

Closing — A War History Against Outliers

In hindsight, the history of quantization compresses to one sentence: "how shall we treat the outliers?" LLM.int8() gave them a separate room; GPTQ compensated their error onto neighbors; AWQ bodyguarded the salient channels; SmoothQuant migrated roughness from the hard side to the smooth side; FP8 absorbed them with an exponent; MXFP4/NVFP4 charged head-on with per-block scales. The next battlefields are the KV cache and 4-bit activations. Bits will keep shrinking — and with a validation habit, those savings are very nearly free.