Skip to content
Published on

The Training Recipe — From Pre-training to Post-training, and What Reports Write Down

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

Introduction

So far we have dealt with structure. But the same structure produces very different results depending on how it was trained. The training recipe is the longest section of a technical report, the hardest to reproduce, and for that reason the most interesting.

This post compares how the reports of four models describe their training process. The goal is not to recommend a particular recipe but to build the skill of reading this section.

All figures were verified directly against papers, official reports, and config.json files on 2026-08-12. Models get updated, so check the originals again.

Token Budgets

Scale first. Here are the pre-training token counts the reports state.

model             pre-training tokens    source
Llama 3 405B            15.6T            arXiv:2407.21783
Qwen3                     36T            arXiv:2505.09388
DeepSeek-V3             14.8T            arXiv:2412.19437
Kimi K2                 15.5T            arXiv:2507.20534

The Qwen3 report states that this data covers 119 languages and dialects, and that it collected twice as many pre-training tokens as Qwen2.5 covering three times more languages.

Why Split Into Stages

All four reports describe pre-training not as a single process but as several stages. The reason is cost. Long-context training is far more expensive than short-context training, so training mostly short and extending only at the end is the sensible arrangement.

The three Qwen3 stages run as follows. The first trains on over 30 trillion tokens at a sequence length of 4,096 to build general knowledge. The second strengthens reasoning with knowledge-intensive data in areas such as STEM and coding. The third uses long-context data to raise context length from 4,096 to 32,768.

Llama 3 likewise states that pre-training of the 405B model consists of initial pre-training, long-context pre-training, and annealing. The initial stage starts with a batch of 4M tokens at sequence length 4,096, then doubles to a batch of 8M at sequence length 8,192 after 252M tokens of training.

How Context Extension Gets Written Down

Two reports handle the same goal differently.

Llama 3 grew it by training. The report states that context length was increased gradually in six stages from the original 8K to a final 128K, and that this long-context pre-training stage used approximately 800B training tokens. It also describes continuing to train at each step until the model had successfully adapted to the increased length.

DeepSeek-V3 grew it with an extension technique. The report sets the pre-training sequence length at 4K and states that after pre-training, YaRN was applied through two phases of 1000 steps each, expanding from 4K to 32K and then to 128K.

The difference between the two approaches is a difference in cost. 800B tokens and 2000 steps are entirely different scales. Checking the extension method in a report lets you gauge how deeply that context length was actually validated.

Annealing and the Finish

The handling of the final stretch of training is explicit in these reports too.

Llama 3 states that during pre-training on the final 40M tokens the learning rate was linearly annealed to 0 while maintaining a context length of 128K, and that during this annealing phase the data mix was adjusted to upsample data sources of very high quality. The same report also describes measuring the value of a dataset by annealing the learning rate of a half-trained 8B model linearly to 0 on 40B tokens.

DeepSeek-V3 writes its learning-rate schedule in units of tokens. It rises linearly from 0 to 2.2e-4 over the first 2K steps, holds until 10T tokens, then decays to 2.2e-5 over 4.3T tokens on a cosine curve. Over the final 500B tokens, the first 333B run at 2.2e-5 and the remaining 167B at 7.3e-6.

Descriptions like these are not mere record-keeping. Knowing what changed at which point is what makes results interpretable.

Hyperparameters That Change During Training

The DeepSeek-V3 report contains several values that shift mid-training. The load-balancing bias update speed is 0.001 for the first 14.3T tokens and switches to 0.0 for the remaining 500B. The multi-token prediction loss weight is 0.3 for the first 10T tokens and drops to 0.1 for the remaining 4.8T. Batch size grows from 3072 to 15360 over the first 469B tokens and then holds.

A pattern emerges. Early in training, balance and auxiliary objectives are applied firmly, and toward the end that pressure is released so the model concentrates on the real objective. Schedules like these never appear in config.json. They are information you can only get by reading the report.

Rewriting Data Instead of Adding More

The Kimi K2 report starts from the premise that as high-quality human data grows scarce, token efficiency is emerging as a critical coefficient, and it presents a data-rephrasing approach.

The report includes an experiment comparing three training strategies: repeating the original dataset for 10 epochs, rephrasing the data once and repeating for 10 epochs, and rephrasing the data 10 times with a single training pass. SimpleQA accuracy is reported as 23.76, 27.39, and 28.94 respectively.

These figures are results the authors produced themselves and are not an independent evaluation. That said, the experimental design is stated, so what is being compared is clear. This distinction matters when reading reports.

Why Post-training Gets Written Up Briefly

Compared with pre-training, post-training descriptions are generally brief. The Llama 3 report does state a concrete problem it hit in long-context post-training: applying the existing supervised fine-tuning recipe with only short-context data resulted in significant regressions in the long-context capability gained during pre-training.

The remedy is stated as well. Because getting humans to annotate lengthy contexts is largely impractical, the team relied predominantly on synthetic data, and reports that mixing 0.1 percent of synthetically generated long-context data with the original short-context data optimized performance across both short-context and long-context benchmarks.

That 0.1 percent figure is striking. It illustrates how sensitive data mixture ratios can be.

What to Check in a Report

When reading a training recipe section, these are worth confirming.

First, the pre-training token count together with the sequence length used. You need both to know the real scale of training.

Second, whether context extension came from training or from an extension technique. This is where the substance behind an advertised context length is decided.

Third, hyperparameters that changed during training. A report that writes these down is one with higher reproducibility.

Fourth, the data mixture. Most reports do not disclose exact proportions. This is the largest gap in currently published reports.

Closing

Training recipes are harder to reproduce than architecture, so the level of disclosure varies from report to report. Some write down token counts, stage composition, learning-rate schedules, and extension methods, while others present only results. What is missing matters as much as what is present. The next post sets out how to read for that.

References

Try It Yourself

Series

Discuss on Twitter • View on GitHub