Skip to content
Published on

Anatomy of config.json — Reading a Model From One Settings File

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

Introduction

Download any open-source model and you will find a config.json sitting next to the weights. Those twenty-odd lines contain nearly the entire skeleton of the model. How many layers, how many attention heads, and why this particular model has a small KV cache are all decided here.

This series is not about what a model scores. It is about how a model is built and why those choices were made. The goal of this first post is simple: when you finish it, you should be able to open a config.json you have never seen before and read its structure.

All figures were verified directly against papers, official reports, and config.json files on 2026-08-12. Models get updated, so check the originals again.

Start With a Real File

Here is the settings file for Qwen3-8B. Every value below is taken verbatim from the public repository.

{
  "hidden_size": 4096,
  "num_hidden_layers": 36,
  "num_attention_heads": 32,
  "num_key_value_heads": 8,
  "head_dim": 128,
  "intermediate_size": 12288,
  "rope_theta": 1000000,
  "vocab_size": 151936,
  "tie_word_embeddings": false,
  "max_position_embeddings": 40960,
  "rms_norm_eps": 1e-06,
  "hidden_act": "silu"
}

The source is https://huggingface.co/Qwen/Qwen3-8B/raw/main/config.json.

Width and Depth

hidden_size is the width of the model. It is the length of the vector a token carries as it moves between layers, and it forms one side of nearly every weight matrix. num_hidden_layers is the depth. Together these two values account for most of the parameter count.

Increasing width makes the matrix multiplications larger, which fills the GPU well, but parameters grow with the square of the width. Increasing depth grows parameters only linearly, yet every layer must be traversed in sequence, so latency rises and the number of pipeline-parallel stages rises with it. Whether to build a given size out of width or depth is a decision made together with training stability and the intended serving shape.

Two Kinds of Attention Head

This is where newcomers stumble most often. num_attention_heads is the number of query heads and num_key_value_heads is the number of key/value heads. When the two are equal you have MHA, when key/value heads number one you have MQA, and anything in between is GQA.

Qwen3-8B has 32 and 8, so four query heads share one key/value head. That ratio translates directly into KV cache size. The number of KV cache elements one token occupies is computed like this.

KV elements/token = 2 x num_hidden_layers x num_key_value_heads x head_dim
                  = 2 x 36 x 8 x 128
                  = 73,728

Had the same model been built as MHA (32 key/value heads)
                  = 2 x 36 x 32 x 128 = 294,912   ->  4x

At fp16 each element is 2 bytes, so caching 32,768 tokens costs 4.50 GiB. As MHA it would be 18.00 GiB. The saving is not free. Cutting key/value heads reduces representational capacity, and the GQA paper frames the technique as an interpolation between MQA and MHA, reporting that converting an existing checkpoint to GQA takes additional training equal to 5 percent of the original pre-training compute (Ainslie et al., arXiv:2305.13245).

head_dim is the dimension of a single head. Older models defined head dimension implicitly as hidden_size divided by head count, but these days it is written into the config explicitly. For Qwen3-8B, 32 times 128 equals 4096 and matches the width, though it does not have to.

FFN Size and Activation

intermediate_size is the inner dimension of the FFN. Qwen3-8B expands from 4096 to 12288 and comes back down. A hidden_act of silu signals a SwiGLU-family block, and that structure uses three matrices: gate, up, and down. So FFN parameters must be counted as three times, not two. Miss this and your parameter arithmetic will never close.

Vocabulary Size and Tied Embeddings

vocab_size is the number of rows in the embedding matrix. One caution here. The Qwen3 config says 151936, but the Qwen3 technical report gives the tokenizer vocabulary as 151,669 (arXiv:2505.09388). Downloading the tokenizer file and checking it directly also yields 151,669. The gap is padding that rounds the embedding matrix up to a hardware-friendly size. In other words, vocab_size is the embedding matrix size, not the tokenizer vocabulary count.

tie_word_embeddings decides whether the input embedding and the output layer share weights. Table 1 of the Qwen3 report shows 0.6B, 1.7B, and 4B tying them while 8B, 14B, and 32B do not. The smaller the model, the larger the share of the total that embeddings occupy, so tying pays off more. For Qwen3-8B a single embedding matrix is 151936 times 4096, roughly 620 million parameters.

Counting Parameters by Hand

Now let us count the full parameter total using only the values above.

Embedding    : 151,936 x 4,096                     =   622,329,856
Output layer : untied, so the same size            =   622,329,856

One layer:
  q_proj     : 4,096 x (32 x 128)                  =    16,777,216
  k_proj     : 4,096 x ( 8 x 128)                  =     4,194,304
  v_proj     : 4,096 x ( 8 x 128)                  =     4,194,304
  o_proj     : (32 x 128) x 4,096                  =    16,777,216
  q_norm, k_norm : 128 x 2                         =           256
  FFN (gate/up/down) : 3 x 4,096 x 12,288          =   150,994,944
  2 x RMSNorm : 4,096 x 2                          =         8,192
  subtotal                                         =   192,946,432

36 layers    : 192,946,432 x 36                    = 6,946,071,552
final norm   : 4,096                               =         4,096

total        = 6,946,071,552 + 622,329,856 x 2 + 4,096
             = 8,190,735,360

The parameter count Hugging Face reports for Qwen3-8B through safetensors metadata is 8,190,735,360. Not a single parameter off. Counting Mixtral-8x7B the same way gives 46,702,792,704, which also matches the published value exactly. If this arithmetic closes, you have read the config correctly.

rope_theta and Context Length

rope_theta is the base frequency of rotary position embeddings. A larger value makes the positional signal rotate more slowly, which favors long context. Llama 3 uses 500,000 (arXiv:2407.21783, Table 3). Qwen3 and Mixtral use 1,000,000.

max_position_embeddings is the trained positional range, and there is a trap here. The Qwen3-8B config says 40960 while Table 1 of the technical report lists context length as 128K. Reading the report, long-context pre-training was done at 32,768, and a fourfold extension is obtained at inference time with YaRN and DCA. So 128K is not a trained length but the figure once extension techniques are switched on. When a config number and a marketing number disagree, this is usually why.

Closing

config.json is a summary of the model. Width and depth set the parameters, the ratio of query heads to key/value heads sets the KV cache, tie_word_embeddings sets the weight class of small models, and rope_theta together with max_position_embeddings tells you what the context length really is. Get into the habit of counting parameters by hand and reconciling them against the published total, and you will be able to read the structure of any model you meet.

References

Try It Yourself

Series

Discuss on Twitter • View on GitHub