필사 모드: Lessons from Published Training Runs — What Was Tried, and What Failed

• Introduction — What to Read From a Report Is Not the Scoreboard
• The Cases Chosen and the Verification Standard
• Llama 3 405B — Designing at a Scale Where Failure Is a Constant
• DeepSeek-V3 and Kimi K2 — Two Layers for Killing a Spike
• The Olmo Family — Stability Fixes Left as a Trace in the Architecture
• OPT-175B and BLOOM — The Archetype of a Culture That Logs Failure
• SmolLM3 and Marin — Published Experiment Notes at Smaller Scale
• Reading Across — What Repeats, and What Never Shows Up
• Closing — Knowing What You Can't Reproduce Is the Skill of Reading

Introduction — What to Read From a Report Is Not the Scoreboard

If you close a technical report after just looking at the benchmark table, you get almost nothing out of it. The scores don't transfer to your situation, but what broke and how it got fixed transfers regardless of scale.

This post pulls out only that part from public reports and logbooks: loss spikes and the response to them, the relationship between hardware failure rates and checkpoint intervals, the effect of changing the data mix midway through, and the reasoning behind scaling decisions. At the end of each case is a separate note on what the report doesn't say. That side matters more for judging reproducibility.

The Cases Chosen and the Verification Standard

All links were checked on August 2, 2026. Quoted figures are taken from the original documents or reporting that directly quotes the originals; anything unverified is noted as such in the text.

Llama 3 405B — Designing at a Scale Where Failure Is a Constant

The most citation-worthy thing in this report is its interruption statistics. Over 54 days running on 16,384 H100 80GB GPUs, there were 419 unexpected interruptions, and the report classifies 78 percent of them as confirmed or suspected hardware issues. The detailed breakdown, according to datacenter-industry reporting that quotes the report's table, is as follows.

419 incidents over 54 days is about once every 3.1 hours on average. At this scale, failure is not an exception, it's a constant. So the design changes accordingly. Operations where a human intervenes to restart no longer hold up; failure detection, node replacement, and resumption all have to run automatically.

The practical calculation that follows directly is the checkpoint interval. With a mean time between failures of 3.1 hours and a save time of 5 minutes, the optimal interval comes out around 43 minutes — and even then, more than 20 percent of total time still disappears into saving and recomputation. The full derivation is worked out in the LLM Ops post.

For parallelism they used a 4D combination — tensor, context, pipeline, and data — and reported model FLOPs utilization in BF16 landing in the 38-43 percent range. Worth keeping in mind: even a well-tuned large-scale training run doesn't use even half of theoretical peak. Landing in the 30s on your own cluster isn't necessarily proof something is wrong.

What the report doesn't say: the exact composition ratios of the training data and the filtering classifier's thresholds, the failed preliminary experiments that got discarded, and the implementation of the internal operations software responsible for failure detection and automated recovery. The statistics were published; the tooling that produced them wasn't.

DeepSeek-V3 and Kimi K2 — Two Layers for Killing a Spike

Read the two reports side by side and two different strategies for handling loss spikes come into view.

DeepSeek-V3 trained a MoE with 37B of its 671B parameters active per token, over 14.8T tokens, and reports 2.788M H800 GPU hours for the full run — including 119K hours for context extension and 5K hours for post-training. And the report contains this sentence: over the entire training run, they never experienced an unrecoverable loss spike and never performed a rollback.

The report doesn't directly assert where this stability came from. But what's described alongside it is a hint. They introduced FP8 mixed-precision training while keeping precision-sensitive operations at higher precision, used DualPipe to overlap computation and communication and cut pipeline bubbles, and handled MoE load balancing without an auxiliary loss. That last item matters especially: balancing expert usage with an auxiliary loss creates instability because that loss term fights the main objective, and they removed the term entirely.

The follow-up retrospective paper, Insights into DeepSeek-V3, covers the hardware side. It gives a figure of the per-token KV cache being cut to 70.272KB with multi-head latent attention, versus 516.096KB for Llama-3.1 405B in the same table. It also states they used a multi-plane fat-tree topology for the cluster network. This is a case where a model-architecture decision is directly an infrastructure decision.

Kimi K2 solves the same problem at a different layer. Training a MoE with 32B of 1.04T total parameters active, over 15.5T tokens, they report observing no loss spikes — and the mechanism is the optimizer. They combined QK-Clip with the Muon optimizer to form MuonClip, rescaling attention's query and key weights right after each update to suppress logit blowup. The learning-rate schedule is WSD-family: a 500-step warmup, then 10T tokens at a constant rate, with the remaining 5.5T decayed on a cosine schedule. The pretraining context length is 4,096.

Summarizing the difference between the two approaches:

What the reports don't say: both reports describe only the successful final configuration. Which layer broke first during the FP8 rollout, how the clipping threshold for MuonClip was found, how many failed runs preceded it — none of that appears. And neither discloses the data composition.

The Olmo Family — Stability Fixes Left as a Trace in the Architecture

Allen AI's Olmo 3 released 7B and 32B models with the entire development pipeline made public — checkpoints, datasets, dependencies, all of it. This full openness matters not for the benchmark scores but because it's the only form in which a third party can directly audit contamination and data provenance.

Technically, what's worth noting is that the stability fixes ended up as architecture, not hyperparameters. Olmo 3 keeps the configuration that OLMo 2 confirmed as a stability improvement: RMSNorm placed post-normalization, an extra normalization applied to queries and keys before the attention computation, and no weight decay on embeddings.

All three are decisions you cannot change once pretraining has started. So the lesson from this case is practically cold: the best time to respond to instability is before the first spike happens. There is no option to change the normalization placement once you've already run 3T tokens.

What the report doesn't say: the data and checkpoints are public, but most organizations don't have the compute to rerun this scale. "Fully open" grants auditability, not reproducibility. Worth keeping that distinction clear.

OPT-175B and BLOOM — The Archetype of a Culture That Logs Failure

If today's technical reports are polished final drafts, these two 2022 cases are the raw record.

OPT-175B's chronicles directory holds the full logbook PDF alongside update notes at 10 percent, 27 percent, 56 percent progress, and the final point. The repository describes it as "the complete logbook we used to train OPT-175B, and the notes documenting the difficulties encountered along the way." The logbook has repeated restarts, loss divergences, mid-training hyperparameter changes, and hardware swaps, each dated.

What's worth learning here isn't a specific technique, it's the attitude that the record itself is a deliverable. If a decision made mid-training and its reasoning aren't recorded at the time, no one can reconstruct it two months later.

BLOOM trained for about 3.5 months on France's Jean Zay supercomputer, on 48 nodes with 8 A100 80GB per node, 384 total, consuming 1,082,990 compute hours. What stands out in the resourcing plan is that they set aside 4 spare nodes separately, precisely because of the possibility of hardware failure. Inter-node links were 4 x 100Gbps Omni-Path per node, and since it was a shared supercomputer, they shared the filesystem with other users.

Reserving roughly 8 percent as spare nodes at a scale of 384 is a rule you can carry over directly into your own practice. Skip the spare-node line in your resource request, and the moment one node dies, the entire training run goes to the back of the queue.

What the reports don't say: both sit on top of the software stack of their era (older Megatron-DeepSpeed, older PyTorch), so the code can't just be run as-is today. And the judgment calls in the logbook are bound to hardware and framework constraints of that time — take the thought process from it, not the conclusions.

SmolLM3 and Marin — Published Experiment Notes at Smaller Scale

The cases above are at a scale most organizations can't mimic. So here are two more practical ones.

SmolLM3 published how it decided its data mixture while training a 3B model on 11.2T tokens. It describes a three-stage pretraining schedule that shifts the proportions of web, code, and math data at each stage, with those proportions decided by small-scale experiments running the 3B model at 50B-to-100B token scale. Stage 1's composition is spelled out down to specific numbers: 85 percent web (12 percent of that multilingual), 12 percent code, and so on.

This methodology is the crux. An 11.2T-token decision was made from a 100B-token experiment. The procedure — set a few candidate mixture ratios, run them at small scale, compare, then apply to the full run — works the same at 3B as at 30B. The accompanying Smol Training Playbook covers things that never make it into a paper, like the process of debugging a loss spike on a 384-GPU job. 384 GPUs is a scale many organizations can actually get their hands on.

Marin is a public research lab started out of Stanford where every research attempt is first registered as a GitHub issue, which then serves as a kind of pre-registration. While training an 8B model past 12T tokens, they kept refining the data mix, and documented the late-stage cooldown phase — lowering the learning rate while shifting the data toward higher quality — in a retrospective document.

The practice of pre-registration is especially worth taking away. Write the hypothesis after seeing the results, and almost every result looks like a success. Simply writing down what you expect before you run it cuts this bias substantially.

What the report doesn't say: how well SmolLM3's small-scale proxy experiments actually predicted the full-scale outcome was never quantitatively verified. Small-scale experiments genuinely do fail to transfer to large scale in some cases, so this methodology is better received as "a better procedure than an unfounded guess," not as "a perfect predictor."

Reading Across — What Repeats, and What Never Shows Up

Overlaying all seven cases, five recurring patterns show up.

1. There are four layers for handling a loss spike, and the lower you go the cheaper it is, the higher you go the later it is. Cost of intervention rises in this order: objective-function design, architecture, optimizer, operations. And yet most teams only respond at the most expensive layer — operations.
2. Failure rate scales with GPU count, and it determines checkpoint interval and how much automation you need. At 384 GPUs, spare nodes are enough; at 16,384, automatic detection and replacement are mandatory. A tenfold increase in scale requires a different mode of operations.
3. Data mixture is not fixed from start to finish. Multiple cases describe staged mixtures and a late-cooldown shift to higher-quality data. The model of "pick a dataset and start training" doesn't match reality.
4. Scaling decisions come from small-scale proxy experiments. The bigger the decision, the smaller the experiment that grounds it. This is the point where experiment-design skill matters more than GPU count.
5. Model-architecture decisions are infrastructure decisions. An attention design that shrinks the KV cache changes serving cost; an MoE choice changes network topology requirements.

On the other side, here's a list of what never shows up in almost any report. Knowing this list is what keeps you from over-trusting a report.

• The number and cost of failed runs. The final run's GPU-hours get published, but the resources burned getting there almost never are. Using the published GPU-hours as your budget baseline will leave you badly short.
• The exact data composition and filter thresholds. Ratios show up, but the original sources, deduplication parameters, and quality-classifier cutoffs are usually left out.
• The hyperparameter search process. Only the final values show up, not how they were found.
• Operations staffing and tooling. How many people, on how many shifts, and which internal tools detected failures — all missing. This is, in practice, the biggest barrier to entry.
• Details of eval-set contamination checks. There's often a sentence saying it was checked, but rarely the method, the threshold, or the size of overlap found.

Closing — Knowing What You Can't Reproduce Is the Skill of Reading

A public report is not a blueprint for success, it's a record of decisions one team made under a specific set of constraints. The automated-recovery design that worked at 16,384 GPUs is overkill on an 8-GPU node; the manual response that worked at 384 doesn't hold up at 16,384.

So the way to read them settles into this: the numbers are bound to that team's scale, don't take them; take the reasoning behind the decision. Why was the checkpoint interval set at this point, why was instability blocked at this layer, why did this experiment lead to that decision. And always carry the list of what's missing from the report alongside it. Only someone who knows what's absent can plan on the basis of that report.

The fundamentals of parallelism are covered in The Four Kinds of Multi-GPU Parallelism, and the tools for executing these decisions are covered in the LLM training stack map and the practical Slurm guide.