How These Models Were Actually Built — Dissecting the 2026 Open-Weight Pipeline

Introduction — Cards Write Architecture in Paragraphs and Data in One Line

Lay several open-weight model cards published in the summer of 2026 side by side and read through them, and one pattern shows up. Architecture gets explained in tables, in paragraphs, in diagrams. They write down layer count, expert count, attention type, even the activation function. But training data usually gets one paragraph, sometimes just one sentence.

This isn't an oversight — it's a structural choice. Architecture is sitting right there in config.json regardless, so it can't be hidden. Data composition, on the other hand, is the one thing that can be hidden, and it's the spot where competitive advantage and legal exposure both hang in the balance at once. So when reading a model's production pipeline, you have to separate which stages are verifiable from which stages are a matter of trust.

This post uses the trending models laid out in the previous post as examples, and walks through the pipeline in order from data collection to quantized deployment. There's one rule: content stated in a card or paper gets a citation, and an effect claimed only by the maker gets marked as a self-claim. Anything I couldn't verify, I write down as unverified. Every citation was checked directly against the original source on August 2, 2026.

Data — Almost No One Has Actually Disclosed It

Let's establish the facts first. In writing this post, I downloaded roughly 40 top-trending model cards and checked the datasets field in every frontmatter. Two of them named their training dataset explicitly.

What A.X K2 named is nvidia/Nemotron-CC-v2.1, nvidia/Nemotron-Pretraining-Code-v2, and HuggingFaceFW/fineweb-2. Even this doesn't disclose the full mixing ratio — it's at the level of pointing to sources. Still, it says far more than the rest. tmax-9b's disclosure is at the level that was possible because it's a small research model. Ai2 uploaded the rollouts and log-probabilities generated during training in their entirety, and from a reproducibility-research standpoint, this is far more useful.

What about the rest — how is theirs written? The training section of the Inkling card is representative. The gist is that it used a broad range of material including text, image, audio, and video, sourced from public material, material acquired from third parties, and synthetic or augmented material. It states that the cleaning process includes deduplication and a low-quality filter. Not a single word of it is wrong, but this paragraph tells you nothing about which data went in at what percentage. Most frontier-class releases sit at this level.

Token budgets, at least, tend to come as numbers. Here's what's been confirmed.

The circumstantial evidence that synthetic data's share has grown is much clearer on the trending-datasets side. Looking at the top twenty trending datasets on the same day, several of them are collections of frontier models' output traces. Names like Glint-Research/Fable-5-traces, Crownelius/Complete-FABLE.5-traces-2M, greghavens/kimi-k3-coding-and-debugging-traces, and ianncity/GLM-5.2-Conversation fit this pattern. The accounts are individuals or small research organizations, and the dataset name itself states which model's output it is.

Two things need to be separated here. One is whether training on this kind of trace data is technically valid; the other is whether it's permitted under the original model provider's terms of use. On the first question, the circumstantial evidence is clear that several models are actually being built this way. On the second, I haven't checked each provider's terms, so I won't render a judgment. If your organization plans to use datasets like this, that check needs to happen separately.

Meanwhile, public corpora for pretraining are still holding their ground. HuggingFaceFW/fineweb was released in April 2024 and is still near the top of trending datasets, and HuggingFaceCode/stack-v3-train is a code corpus uploaded in July 2026. The trend of disclosing data hasn't disappeared — it's the makers of frontier-class models who have stepped out of it.

Architecture — Sparsity Has Climbed Past 20x

MoE is now close to a default, not a choice. What's interesting isn't whether it's adopted — it's the magnitude of the sparsity. Here's what you get dividing total parameters by active parameters per token.

Read it vertically, and there's a tendency for sparsity to rise along with scale. The cards don't directly explain why this happens, but structurally it makes sense. Raise sparsity, and you can pack more parameters into the same inference FLOPs, at the cost of having to keep every parameter resident in memory, so weight memory scales up regardless. In other words, sparsity is the dial that separates compute cost from memory cost. In a large serving environment with plenty of memory, it's advantageous to crank this dial all the way; on a single card, it means nothing at all.

A configuration with shared experts is now nearly universal. All six models in the table above that disclosed their expert configuration have one or two shared experts that are always active, separate from the routed experts.

Only Kimi K3 tells a slightly different story. Its card states it raised sparsity with a framework called Stable LatentMoE, and that overall scaling efficiency improved about 2.5x versus Kimi K2. This figure is the maker's own self-measurement, and the card doesn't specify a definition of "scaling efficiency," so there's no way to verify it externally. The numbers in the architecture table (1 dense layer out of 93, attention hidden dimension 7168, vocabulary of 160K, context of 1,048,576) belong to the category verifiable via config.json; scaling efficiency doesn't.

Attention and Context — Four Paths to Cutting Down Global Attention

Now that "1M context" has started appearing in the first line of the card, attention design has all converged on one problem. Apply softmax attention over the full sequence at every layer, and the KV cache and compute cost become unmanageable. There are broadly four approaches in use right now.

First, mix linear attention with softmax attention. Solar Open 2 arranges its 48 layers at a 1-softmax-to-3-linear ratio. The card states two effects: since only 12 of the 48 layers keep a KV cache, long-context memory is about a quarter of an equally-sized all-softmax model; and because linear attention layers embed order inside their recurrent state, positional encoding was removed entirely (NoPE), eliminating RoPE's extrapolation limit. The latter claim is the maker's own explanation, and I haven't confirmed an external reproduction. The Qwen3.6 family made a similar choice — the Qwen3.6-27B card's layer arrangement is three Gated DeltaNet layers to one Gated Attention layer, repeated 16 times.

Second, mix sliding-window attention with global attention. Laguna S 2.1 has only 12 of its 48 layers as global, with the remaining 36 as sliding-window with a window of 512. The same pattern shows up in rerankers too — jina-reranker-v3.5 sets 28 layers at a 3-sliding-window-to-2-global ratio, with a window of 1024. This model ranks the entire list in a single forward pass, and the card states that the last layer has to be fixed as global for that reason.

Third, use a sparse indexer. This selects only the top few keys out of the full set and computes attention over those. GLM-5.2 states it reuses the same indexer across every four sparse-attention layers with a technique called IndexShare, cutting per-token FLOPs by 2.9x at 1M context. A.X K2 uses an SGA structure that layers a lightweight indexer selecting the top few tokens plus a per-head gate on top of MLA. Both figures are self-reported by their respective labs.

Fourth, compress attention itself. The DeepSeek-V4 card states it combines compressed sparse attention with high-compression attention, and that at a 1M-context setting, per-token inference FLOPs come to 27 percent and the KV cache to 10 percent, versus V3.2. This too is a self-measurement. Kimi K3 states it mixes KDA and Gated MLA at a 69-to-24 ratio, together with a residual structure called AttnRes.

The principles behind the attention variants themselves were covered in the rundown of GQA, MQA, and FlashAttention, so I won't repeat them here.

There's one practical trap worth flagging about context length. The maximum stated on a card is usually split into two stages. Qwen3.6-27B states 262,144 as native, extensible up to 1,010,000. A.X K2 states it trained natively up to 128K, then extended to 256K with YaRN scaling. In other words, even the same "260,000 tokens" is a mix of what was secured through training and what was stretched at inference time.

And the DeepSeek-V4 card recommends setting the context window to at least 384K to use max thinking mode. That means a substantial share of this model's 1M context gets used by the model's own thinking tokens, not user input. Leave this out when budgeting context and the math ends up badly off. The differences across extension methods are laid out in the context window extension guide.

Pretraining — Tilting Away From Training From Scratch

As the cost of training a frontier-class model from a blank slate has grown, using a prior-generation model as the initialization has visibly increased. There are two confirmed cases.

Solar Open 2 was initialized from the 102B Solar Open 1 via selective weight transfer. Per the card, only 2.3 percent of the total weights survived the architecture change intact, with the rest randomly initialized, and that 2.3 percent is described as pulling forward early convergence at the 250B scale. Disclosing the figure of 2.3 percent actually builds trust — it means most of the weights were trained fresh.

K-EXAONE 2.0's card states it grew the prior generation by over 3x through upcycling that expanded both depth and width, then chained on continued pretraining, difficulty-focused mid-training, and post-training. It also disclosed one stabilization technique found along the way: inserting clamping after the two SwiGLU branches mitigates activation blowup in deep layers.

Training stability turns out to be a surprisingly frequent topic in these cards. Three approaches stand out.

The A.X K2 card attaches one more purpose to outlier suppression: keeping outliers in the activation distribution pinned down favors low-precision serving like FP8 or NVFP4. In other words, a training-stabilization technique connects directly to the choice of deployment precision. These are two things that would have been treated as separate topics a few years ago.

Since precision has come up, here's a rundown of pretraining precision too. A.X K2 states it did native FP8 training with MXFP8 E4M3 for both forward and backward passes, keeping only the master weights and gradients in FP32. Kimi K3's architecture table states it secured MXFP4 weights and MXFP8 activations through quantization-aware training. General principles on scaling laws and token-budget decisions are covered separately in the pretraining scaling laws rundown.

Post-Training — the Stage After SFT Has a New Name

A few years ago, this section's title would have been "RLHF After SFT." What's actually written in cards today is more fragmented than that.

DeepSeek-V4's post-training is structurally the most unusual. The order stated by the card has two stages. First, grow a domain-specific expert model independently for each domain, using SFT and GRPO-based reinforcement learning. Then unify those capabilities into a single model through on-policy distillation. In other words, it builds several experts and then merges them back into one body. This is easy to confuse with the architecture's MoE since it's a different layer of the conversation — the "expert" here isn't a routed FFN, it's a separately trained checkpoint.

There's also a case that discloses in numbers what reinforcement learning actually fixed. The KAT-Coder-V2.5-Dev card states that reinforcement learning cut the abnormal-tool-label rate from 9.34 percent to 0.28 percent, and consecutive repetition within a single turn from 0.34 percent to 0 percent. A card that states failure-mode incidence rather than benchmark scores is rare. When choosing a model to use as an agent, numbers like this are far more useful than an average score.

The Nanbeige4.2-3B card writes up a small model's post-training in relatively detailed fashion. At the SFT stage, it increases training-environment diversity through real-environment integration and large-scale environment synthesis, filtering at both the trajectory and turn levels while using both test-case-based verification and rubric-based evaluation together. At the reinforcement-learning stage, it states that combining outcome reward with process reward raised the small model's training stability.

There's also one that disclosed its entire reinforcement-learning procedure. Ai2's tmax family states it ran 200 steps of DPPO on top of Qwen3.5-9B, scoring about 27 percent on Terminal Bench 2.0, roughly 6 points above the base model, and uploaded the training-time rollouts and log-probabilities to the repository alongside it. This is the only externally verifiable post-training case among the models covered in this post.

The user-facing features that post-training produces are also worth summarizing.

Kimi K3's caveat is a trap that's actually hit often in practice. Send back only the assistant message's body in multi-turn or tool-calling contexts, leaving out the reasoning content, and the model ends up not knowing what it was just thinking. The card explicitly states to feed the API response back in its entirety.

Distillation and Miniaturization — Same Name, Different Methods

The word distillation refers to at least three different things in cards right now.

First, the classic sense: training a student on a teacher model's output distribution. Microsoft's harrier-oss-v1 family explicitly states this method. The 270M and 0.6B variants state they additionally received knowledge distillation from a larger embedding model, on top of contrastive learning. There's no such mention for the 27B variant.

Second, shortening a diffusion model's sampling steps. Z-Image-Turbo is a distilled version of Z-Image, and its card states it produces a result in 8 function evaluations. What's being distilled here isn't knowledge — it's the sampling trajectory. Same word, but a different technique from distillation in a language model.

Third, using another model's output as SFT data. This is what the trace datasets seen in the data section earlier fall under, and it's the most common form in the community. Strictly speaking, it's output imitation, not distillation, but the word distillation gets attached to dataset names often anyway. There's no logit-level signal, so the teacher's uncertainty information isn't transmitted.

There's another form of capability transfer worth calling pipeline distillation. Fara1.5-27B is a supervised fine-tune of Qwen3.5-27B, but what made its training data was a multi-agent pipeline called FaraGen1.5. The order stated on the card is: synthesize a web task, execute a trajectory to solve it, verify the result, then feed it into training. Not a single teacher model, but a system with a verification loop attached, plays the teacher's role.

A comparison of the distillation techniques themselves was covered in Knowledge Distillation and Model Compression. There's one thing to stress here. When a card says distillation, check which kind first. Depending on which of the three it is, reproducibility, licensing risk, and the quality you can expect are all different.

Quantized Deployment — Now the Maker Ships It Directly

This is the part of the pipeline that has changed the most at the last stage. Just a few years ago, low-bit checkpoints were made by the community after the fact. Now the original maker ships them alongside.

The significance of this shift isn't small. First, it shakes the premise that "the original BF16 is the baseline and quantization is loss." For Kimi K3 and A.X K2, low precision isn't a post-hoc loss — it's a byproduct of training. A high-precision original to serve as a reference doesn't even exist in the first place.

Second, configurations that carry different precisions per parameter type, like DeepSeek-V4, are increasing. As the card states, only the MoE experts are FP4, and the rest is FP8. A post-hoc quantization tool would have a hard time reproducing this configuration as-is.

Third, community conversions still have a reason to exist, even so. What the maker ships targets data-center GPUs — FP8 and NVFP4 — while GGUF for running on a laptop or MLX for Apple silicon remains the community's job. Extreme compression like Bonsai-27B's 1.125 bits per weight, seen in the previous post, doesn't come from the maker's side.

How a precision choice actually reflects on performance was covered in the quantization format comparison and the inference VRAM math.

Closing — What's Verifiable and What You Have to Take on Trust

Once you've swept through this pipeline, the boundary comes into focus. Everything that goes into config.json is verifiable — layer count, expert count, KV head count, vocabulary size, max position embeddings. So are the file list and the license text. Training data composition, token budget, scaling-efficiency improvements, and benchmark scores, on the other hand, are all statements from the maker.

So there are two things you can do in practice: verify what's verifiable yourself, and treat what's a statement as a statement, reproducing it in your own work. Look at the safetensors header instead of the parameter count, measure quality on an actual long input instead of the stated context length, run your own eval set instead of the card's benchmark table.

The next post lays out an order for finishing that verification work in five minutes — where to start reading a model card, what to skip, and where things quietly break.