필사 모드: What Transfers Through Distillation and What Does Not — Distilling DeepSeek into GPT-OSS Did Not Bring the Censorship Along

• Intro — what that 147-point Show HN actually showed
• Transcribing the experimental design exactly
• The capability transferred — but which capability
• The censorship did not transfer — and why that is not surprising
• Censorship lives in three places
• Evidence in the opposite direction — subliminal learning and shared initialization
• What this one experiment supports, and the practical rules
• Closing — distillation does not copy the teacher, it copies what the teacher said

Intro — what that 147-point Show HN actually showed

On July 30, 2026, an experiment posted to Show HN received 147 points. Here is what a team called CTGT published as a research note. They took the Chinese model DeepSeek V4 Flash as the teacher and distilled financial reasoning capability into the American open-weight model GPT-OSS; the capability went up, and the political censorship the teacher displayed did not appear in the student.

The reason this conclusion drew attention is that it touches a policy argument. The worry that political bias embedded in Chinese open-weight models will ride into derivative models as those models proliferate has been raised steadily, and this experiment says that, in at least one setting, it did not.

But the purpose of this post is not that policy argument. What is interesting here is a question with a much longer shelf life — what exactly does distillation copy? What knowledge distillation copies is not weights but the outputs the teacher produced on particular inputs. If that is so, three things decide whether something transfers: what was observed, where it was observed, and who it is copied into. This experiment is a good case for explaining that frame, and at the same time a good case for showing how far a single informal experiment can carry.

Transcribing the experimental design exactly

The best-made part of the design is the matched pairs. Questions with the same structure where only one side is China-related — the Great Leap Forward and the Holodomor, for instance — are paired up and the score difference is read off. Doing it this way separates "this model is generally cautious about political questions" from "this model avoids only China-related questions." It is a distinction that could not come out of absolute scores alone. Calibrating the LLM judges against 96 human-scored items is a step commonly skipped in this kind of evaluation, and here it was performed; at a correlation of 0.948, the judges' scores are good enough to use as they are.

The teacher-side numbers are worth noting too. A 45.45-point gap that is consistently positive on 87% of the pairs is not sporadic avoidance but systematic behavior. In other words, this experiment starts by ruling out the trivial conclusion that the teacher had no censorship to begin with.

The capability transferred — but which capability

Start with the side that succeeded. In financial reasoning, the self-distilled 120B is reported at 83.61% on FinanceReasoning. That number itself is not the subject of this post, but why capability transfers so well is.

The reason is written right into the objective function of distillation. Distillation directly optimizes the student's output distribution to match the teacher's output distribution. If the training data is filled with CAPM, DCF, and option-pricing problems, what gets optimized is exactly the distribution on those problems. It is not that capability happens to carry over well; it is explicitly written into the loss function that it should carry over.

The methodology reflects this too. They find the point where an error occurs, insert a hint there, and apply reverse KL only over the 100 tokens that follow. That is, the student does not absorb the teacher's whole style, only a local correction at the point where the student gets it wrong. There is also a single experimental variable, the authorship of the hint. This design is efficient for capability transfer, but at the same time it makes the channel through which the teacher's other traits could cross over very narrow. We come back to this in the next section.

The censorship did not transfer — and why that is not surprising

The student's censorship gap was +0.43 at base and +3.94 after distillation. Statistically that is baseline. The result is clear; whether the result is surprising is a separate question.

The distillation data contained 0 items of China-sensitive content. The authors themselves note this as a limitation, writing that what the experiment measured is the absence of censorship, not a failure of active suppression. The HN thread pointed the same way. A comment near the top said there is no way fewer than 200 examples would change a model's stance on the Holodomor, and a summary also came up that the finance domain is not entangled enough with censorship behavior for it to bleed through.

Generalized, it comes to this. The only behavior that transfers through distillation is behavior the teacher actually exhibited inside the support of the distillation data. Even if the teacher is a model that gives evasive answers to Tiananmen questions, if not one such question appears in the training data, that evasion was never presented as a target. What is not presented as a target is not captured by the loss, and what is not captured by the loss does not move the weights.

Flip it around and a far more practical conclusion falls out. Safety alignment does not transfer either, for the same reason. If you distill coding ability from a well-aligned teacher, coding ability is all you get. There is no basis for assuming the student will be as safe as the teacher. If the distillation data contains no safety-related prompts, the student simply keeps the safety characteristics of its own base model. The fact that the student scores in this experiment were essentially the same as the base is exactly what demonstrates this proposition — distillation did not pull the student toward the teacher, it pulled only within a particular region.

Censorship lives in three places

The reason the question "does censorship transfer" so often spins in place is that censorship does not live in one place. It has to be split into at least three layers.

• Outside the weights (the service layer). Input and output filters in the app and the web interface, system prompts, server-side blocking rules. This is not in the weights, so forget distillation — it does not follow even if you download the entire weight file. Most cases where the answer differs between serving an open-weight model yourself and asking through the official app come from here.
• Inside the weights, installed as behavior. A tendency shaped during post-training to produce evasive responses on certain topics. One of the HN comments put it well: most censorship is not the removal of knowledge but the installation of behavior that keeps knowledge from surfacing. There is in fact a line of research showing that refusal behavior is represented as a linear direction in the residual stream, and results have been reported that manipulating that direction turns refusal on and off. If the knowledge had disappeared, it could not be revived by touching a single direction.
• Never in the pretraining data to begin with. This is not behavior but a genuine absence. It is not recovered by alignment manipulation either.

Once you make this distinction it becomes clear what this experiment touched. The student kept its own base model's state at the second layer, and the teacher's second layer was never presented as a target because the distillation data never stepped into that region. The first layer only affects API responses in the first place, so it is out of scope. The third layer is not something this experiment can address.

Evidence in the opposite direction — subliminal learning and shared initialization

Read only this far and "what is not in the distillation data does not come across" looks like a clean rule. But there is a research result pointing in exactly the opposite direction, and the rule is only complete once the two are set side by side.

The subliminal learning paper published in July 2025 reported that a teacher's traits transfer to the student even through data that is semantically entirely unrelated. Training on outputs that look to have nothing to do with the trait, such as number sequences or code, and even filtering out direct mentions of the trait, still produced transfer. There was one decisive condition — it appeared only when the teacher and the student started from the same base model, and did not appear across different bases.

Follow-up work in 2026 moved toward quantifying this. Results measuring how much safety degradation transfers through distillation have been reported, and the transfer pattern differed greatly by base model family. Some families showed a threshold shape where transfer jumped sharply above a certain strength; others transferred continuously and at a higher rate.

Put the two strands together and the rule sharpens into this.

• Behavior the teacher exhibited inside the support of the distillation data transfers. Because it is explicitly optimized.
• Behavior outside the support mostly does not transfer. That is the case this experiment observed.
• However, if the teacher and the student share an initialization, traits outside the support can leak through as well.

The third item was not verified in this experiment. The authors explicitly state that they did not test the scenario of distilling a Chinese teacher into a Chinese-family base. And yet a substantial share of the people who build derivative models in practice use exactly that configuration — an open-weight model from one family as the base, a higher-end model from the same family as the teacher. The condition this experiment covered least is the most common condition.

What this one experiment supports, and the practical rules

To summarize, the proposition this experiment supports is this one sentence. With finance-domain data containing no China-sensitive content at all, applied to an open-weight model from a different family, under localized reverse-KL distillation, the teacher's political censorship behavior did not appear in the student.

Let me be equally clear about the propositions it does not support. It does not support the general claim that distilling a Chinese model does not bring censorship along. It did not test what happens when politics- or history-related prompts are mixed into the distillation data, and there is no result for a same-family base. There was also a point in the thread that the evaluation set leaves out Hong Kong and the invasion of Ukraine, and 152 pairs is a size fit for an exploratory experiment, not a size that nails a conclusion down. Above all, this is a single informal experiment that has not been peer-reviewed and has not been replicated. The authors themselves write that they make no claim at all about where in representation space this behavior lives.

Even so, a few practical rules come straight out of this frame.

• Do not assume you inherited the teacher's safety properties. If you distilled only capability from a well-aligned model, the student's safety characteristics are still the base's. They have to be evaluated separately.
• Design evaluations as matched pairs. Absolute scores cannot distinguish a model's general caution from avoidance of a specific topic. The cost of building structurally matched pairs was the most valuable part of this experiment.
• Check whether the base and the teacher come from the same family. That is exactly the risk condition the subliminal learning research describes. If they are the same family, traits absent from the distillation data also have to go on the test list.
• Document the support of your distillation data. What can and cannot transfer is mostly predictable from that document alone. It is also the first artifact to look at when investigating anomalous behavior after the fact.
• Measure the difference between serving an open-weight model yourself and calling the official API. That difference is the size of the service-layer filter. Without this value you cannot tell whether the sentence "this model is censored" refers to the weights or to the service.

Closing — distillation does not copy the teacher, it copies what the teacher said

If you understand distillation as "the technique for packing a big model into a small one," this result looks unexpected. But what distillation copies is not a model, it is the output distribution over particular inputs, and seen that way the result is close to a necessity of the design. Only finance problems were asked, so only finance answers were copied.

So the next time you get the question "if I distill, does X come along," you can ask three things back before answering. Are there inputs in the distillation data on which the teacher exhibits X? How large a share of the data distribution are those inputs? And did the student start from the same base as the teacher? When the three answers come together, the direction is usually predictable without an experiment. Knowing the direction is not the same as knowing the magnitude, though, so what comes after that is still measurement.