Skip to content
←

필사 모드: Reading a Technical Report Critically — What Gets Written Down and What Goes Missing

English
0%
정확도 0%
💡 왼쪽 원문을 읽으면서 오른쪽에 따라 써보세요. Tab 키로 힌트를 받을 수 있습니다.

Introduction

This is the last post in the series. Along the way we pulled dozens of numbers out of config.json files and technical reports and computed with them. This post sets out the judgment criteria used in that process: which numbers to trust, which to hold in reserve, and which were never used at all.

All figures were verified directly against papers, official reports, and config.json files on 2026-08-12. Models get updated, so check the originals again.

What Can Be Verified and What Cannot

The contents of a technical report fall into three layers by how hard they are to verify.

The first layer is immediately verifiable: structural values such as layer count, head count, vocabulary size, and expert count. Cross-check them against a public config.json and you have an answer in seconds. In this series, comparing the report tables of Qwen3, DeepSeek-V3, and Kimi K2 against their configs produced agreement in every case.

The second layer is verifiable by computation, and parameter count is the prime example. In the first post we computed the parameters of Qwen3-8B as 8,190,735,360 from the config alone, matching the value the repository reports exactly. The same held for Mixtral-8x7B and Qwen3-30B-A3B. That all three matched without error means the structure described in the reports agrees with the actual weights.

The third layer cannot be verified: training token counts, data mixture ratios, GPU hours, and benchmark scores. For these there is no way to confirm anything beyond trusting the report.

The Limits of Self-Reported Benchmarks

Within that third layer, the most frequently cited and most caution-worthy item is the benchmark score. There is a reason this series barely cited any.

Numbers a development team publishes from evaluating its own model are not an independent evaluation. This is not an accusation of dishonesty but a structural limitation. Evaluation prompts, the number of examples used, decoding settings, and grading methods differ by team, and the comparison models are generally either run by that team or quoted from other reports.

The only performance figures this series cited were from the data-rephrasing experiment in the Kimi K2 report: 23.76, 27.39, and 28.94 for the original at 10 epochs, one rephrasing at 10 epochs, and ten rephrasings at 1 epoch. They were cited because they are a controlled comparison by the same team under the same conditions, not a ranking between models. Even then it was stated that these are the authors' own experiments and not an independent evaluation.

The reading habit is simple. When you see a score, first find who measured it and under what settings. Without that information the score cannot be used for comparison.

When the Config and the Report Disagree

Two genuine discrepancies came up in this series.

First, the vocabulary size of Qwen3. The config vocab_size is 151936 while the report says 151,669. Opening the tokenizer file directly gave 151,669. The conclusion is that both are correct. They were simply measuring different things: the config states the embedding matrix size, the report states the tokenizer vocabulary count.

Second, the context length of Qwen3-8B. The config max_position_embeddings is 40960 while Table 1 of the report says 128K. Reading the body of the report, long-context training was done at 32,768, with a fourfold extension obtained at inference through YaRN and DCA. The conclusion is the same: they were talking about different things.

The lesson from both cases is one. When you find a discrepancy, before asking which side is wrong, check whether the two values are measuring the same object. Usually the definitions differ.

What Could Not Be Confirmed

The heart of honest writing is recording what you could not confirm as unconfirmed. This series had such items.

The config.json files for Llama 3.1 and Gemma 2 were access-restricted and could not be read. For Llama 3.1 the structural values are all present in Table 3 of the technical report, so that became the source, and the text states that the figures came from Table 3 rather than a config. For Gemma 2 the config could not be read either, and since the values this series needed were never obtained, the model was left out entirely.

That was the rule this series held to. Values that could not be confirmed are left blank, or the model is not covered. Filling in a plausible number makes the writing smoother but makes it impossible for a reader to base a judgment on the text.

What Reports Commonly Omit

Reading several reports side by side reveals consistently empty cells.

The largest gap is data mixture ratios. Reports that disclose exactly what percentage came from which source are rare. Often only domain names are listed.

The second is a record of failures. Only the final configuration gets written down, and what was tried and abandoned is usually missing. As an exception, the Kimi K2 report stating that doubling attention heads lowered validation loss by only 0.5 to 1.2 percent and therefore did not justify the inference cost is a rare case. Descriptions like this let a reader examine the basis for a decision.

The third is the detail of evaluation conditions. Scores are printed prominently while prompts and decoding settings sit in an appendix or nowhere at all.

Signals of a Good Report

Conversely, some kinds of writing raise credibility.

One is noting that a value changed during training. The DeepSeek-V3 report stating that the bias update speed was 0.001 until 14.3T tokens and then switched to 0.0 is an example. Details like this are hard to fabricate and genuinely help reproduction.

Stating limits about one's own work works the same way. The Kimi K2 report noting that using synthetic data for continued scaling remains an active area of investigation, with key challenges in generalizing across diverse source domains without compromising factual accuracy and in minimizing hallucinations, is such a passage.

Attaching conditions to a figure is another good signal. When Kimi K2 gives the increase in inference compute from raising head count, it states the conditions alongside: a sequence length of 128k with the total expert count fixed at 384. A multiplier without conditions cannot be interpreted.

A Procedure for Checking Things Yourself

Finally, here is the procedure actually used throughout this series.

First, fetch the config.json directly. Accessing the raw file address in a model repository lets you see the structure without downloading weights.

Second, count parameters by hand and reconcile them against the published value. If they match, you have read the config correctly and the report's structural description is verified along with it.

Third, find the corresponding table in the report and compare it against the config. If they disagree, suspect a difference in definitions.

Fourth, if tokenization matters to you, download the tokenizer file and measure with real text. Every per-language token count in this series was obtained that way.

Fifth, write down unconfirmed values as unconfirmed.

Closing

Across eight posts we went from the fields of config.json through MoE routing, attention variants, positional encoding, normalization, tokenizers, and training recipes. One theme ran through all of it: every design choice has a cost, and that cost usually shows up as memory, latency, or training stability.

Models keep changing. The numbers confirmed today will be different next year. But the procedure — open the config, count the parameters, compare against the report's tables, and leave unconfirmed values blank — carries over unchanged. That is what this series set out to leave behind.

References

Try It Yourself

Series

현재 단락 (1/43)

This is the last post in the series. Along the way we pulled dozens of numbers out of config.json fi...

작성 글자: 0자원문 글자: 7,401자작성 단락: 0/43