Dark Knowledge and the Enclosure Problem

 A compression trick from 2015 became the subject of a joint NSA, CISA and FBI advisory. The path between those two facts is worth walking slowly.

A thing or two about distillation!

Copyright: Sanjay Basu

---

The mythical digit

Start with the experiment that named the thing.

In 2015, Hinton, Vinyals and Dean trained a large net with two hidden layers of 1,200 rectified linear units on MNIST, heavily regularized with dropout and weight constraints, and got 67 test errors. A smaller net with 800 units per layer and no regularization got 146 errors. When that same small net was trained to match the soft target distribution of the large net at a temperature of 20, it got 74 errors. Half the error rate, same architecture, same parameter count, different supervision signal.

Then they did the part I keep coming back to. They removed every example of the digit 3 from the transfer set, so the student had never seen a 3 in training. It still made only 206 test errors, 133 of them on the 1,010 threes in the test set, because the learned bias for that class was too low. Raise the bias for class 3 by 3.5 and the model makes 109 errors, 14 of them on threes. It classifies 98.6 percent of the test threes correctly having never been shown one.

The knowledge of what a 3 looks like arrived through the teacher’s opinions about everything that was not a 3. That leftover probability mass, the 0.008 on 7 and the 0.0004 on 8 when the teacher is 99 percent sure it is a 2, is what Hinton started calling dark knowledge. It encodes the similarity structure the teacher discovered during training. A hard label says 2 and nothing more. A soft target says 2, and also says that this particular 2 sits closer to the 7 manifold than most, which is a statement about the geometry of the learned representation rather than about the image.

Geoff Hinton famously said in his BayLearn keynote abstract that

this technique works because most of the knowledge in the learned ensemble is in the relative probabilities of extremely improbable wrong answers. For example, the ensemble may give a BMW a probability of one in a billion of being a garbage truck but this is still far greater (in the log domain) than its probability of being a carrot. This dark knowledge, which is practically invisible in the class probabilities, defines a similarity metric over the classes that makes it much easier to learn a good classifier.

Temperature, and why the gradient needs a correction

The mechanism is one line. A softmax converts logits to probabilities as q_i = exp(z_i/T) / Sum_j exp(z_j/T), where T is normally 1. Raise T and the distribution softens. Distillation trains the student on teacher outputs generated at high temperature, uses the same high temperature in the student’s softmax during training, then drops back to T equal to 1 at inference.

Two details matter in practice and get skipped in most explanations.

First, the gradient of the soft-target term scales as 1/T squared. If you mix the soft loss with an ordinary cross-entropy loss on hard labels, which you almost always want to do when labels exist, the soft term quietly vanishes as you raise the temperature. Multiply the soft loss by T squared and the two objectives stay comparable no matter what temperature you pick. Anyone who has trained a distillation run that mysteriously ignored the teacher above T equal to 10 has met this.

Second, in the high temperature limit with zero-mean logits, distillation reduces to matching logits under a squared error. Temperature is a dial between two regimes. Low temperature, the student chases the teacher’s argmax and learns almost nothing the labels did not already say. High temperature, the student fits the whole logit vector, including the parts the teacher is least confident about, which is also where the teacher’s noise lives. The paper found that with 300 or more units per layer, everything above T equal to 8 behaved about the same, but at 30 units per layer, temperatures between 2.5 and 4 worked significantly better. Small students cannot absorb the full structure, so you have to decide which parts of it to hand them.

The MNIST result is cute. The production result is the one that made people pay attention. On a commercial speech model of roughly 85 million parameters trained on 2,000 hours of audio, an ensemble of ten models raised frame accuracy from 58.9 percent to 61.1 percent, and a single model distilled from that ensemble reached 60.8 percent, capturing more than 80 percent of the ensemble’s gain at one tenth of the serving cost.

Rich Caruana and colleagues had shown model compression worked in 2006. What 2015 added was an account of why. A trained network is a compressed record of a search through parameter space. The training labels do not contain that search. The output distribution does, partially, which is why a student trained on teacher distributions generalizes better than a student trained on ground truth, even when ground truth is available and correct.

What breaks when the teacher is a language model

Everything above assumes a fixed label set and a single decision per input. A language model has a vocabulary of 100,000 or more tokens and emits a sequence, where the quality of the whole depends on choices made thousands of positions earlier. Per-token forward KL over the full vocabulary is the naive lift, and it works, but it is not where the field ended up. Three families are worth separating because they have different access requirements, and access requirements are the whole story in part three.

Synthetic data, or sequence-level distillation. Generate text with the teacher, train the student on it with ordinary supervised fine-tuning, discard the distributions entirely. Kim and Rush formalized this for translation in 2016. Alpaca made it famous in 2023 by generating 52,000 instruction-following demonstrations from an OpenAI model for a few hundred dollars of API spend. Microsoft’s Orca work extended it from answers to explanation traces, on the theory that the reasoning is the transferable asset. The Phi models pushed the curation angle.

The scale version is DeepSeek’s. R1 distilled its own long chain-of-thought reasoning into six smaller dense models built on Qwen2.5 and Llama backbones, using 800,000 prompt and response pairs generated by R1 itself, roughly 600,000 reasoning traces plus 200,000 general examples. No logits. No architectural access. Plain supervised fine-tuning on generated text. This is the technique at the center of the entire industry dispute, and it is the crudest one in the toolkit.

Logit-based. The student matches the teacher’s next-token distribution, either in full or over the top-k. This is the direct descendant of the 2015 method and it is much more sample-efficient than text alone, because every position carries a dense vector of signal instead of one sampled token. It requires the teacher’s distribution, which means either weight access or an API that returns log probabilities. Cross-family use runs into tokenizer mismatch, since a teacher and student with different vocabularies do not agree on what position they are at, and the alignment machinery is fiddly enough that most teams doing serious logit distillation are distilling within a family they own.

Feature and representation based. Match internal states rather than outputs. DistilBERT kept about 97 percent of BERT’s language understanding performance at 40 percent fewer parameters and 60 percent faster inference, using a triple loss over outputs, embeddings and cosine similarity of hidden states. TinyBERT added attention matrices and hidden-layer projections. MiniLM matched self-attention relations in the last layer rather than the layers themselves, which made the student’s depth and width choices independent of the teacher’s. These methods give the strongest signal per training example and are effectively unavailable to anyone outside the organization that owns the weights.

On-policy distillation, and where the signal actually lives now

Off-policy distillation has a structural flaw that shows up only in deployment. The student trains on the teacher’s trajectories, so it only ever sees states a competent model would reach. At inference it makes its own mistakes, lands in states the teacher never demonstrated, and has no idea what to do there. Errors compound down the sequence. Anyone who has watched a distilled reasoning model produce four flawless steps and then spiral has seen exposure bias in the wild.

The current answer is to invert who generates. On-policy distillation samples trajectories from the student and uses the teacher to grade each token of each trajectory, scoring every step of the student’s own solution rather than showing it a correct one. The standard objective minimizes per-token reverse KL between student and teacher on the student-generated prefixes, which is mode-seeking rather than mode-covering. The student learns to do one thing the teacher would endorse, not to spread mass across everything the teacher considers plausible.

The efficiency claims are large enough to be worth checking yourself. Thinking Machines reported reaching teacher-level performance roughly 7 to 10 times faster than reinforcement learning, and a follow-up writeup reports 70 percent on AIME’24 with a Qwen3–8B student at about a tenth of the compute of RL. The mechanism is dense supervision. RL with verifiable rewards gives you one scalar at the end of a long rollout. On-policy distillation gives you a per-token signal on the same rollout.

The 2026 literature on this is a construction site rather than a settled result. There are papers reweighting by student surprise, papers scheduling a mixture of forward and reverse KL gated on coverage, papers routing correct and incorrect rollouts to different losses, and papers arguing reverse KL supervision does not guarantee a reward-improving update at all. I would not treat any particular objective as the answer yet.

Here is the part that connects to the politics. Reverse KL on sampled trajectories requires only the teacher’s log probabilities on the tokens the student actually produced, not the full-vocabulary distribution. That is a very small ask of an API. It is also a much stronger signal than generated text. And notice what the teacher is being asked for now. Not the answer. An opinion about the student’s attempt.

The dark knowledge migrated. In 2015 it lived in the ratios among the wrong classes. In a reasoning model it lives in the trajectory, in where the model backtracks, how many tokens it spends before committing, which subproblem it attacks first. That is why the extraction campaigns target reasoning traces specifically, and why hiding raw chains of thought stopped being a safety-only decision and became a competitive one.

The part that ended up in a cybersecurity advisory

The timeline in 2026 moved fast.

On February 23, Anthropic published a report alleging that DeepSeek, Moonshot and MiniMax ran industrial-scale campaigns against Claude, generating more than 16 million exchanges through approximately 24,000 fraudulent accounts in violation of its terms of service and regional access restrictions. By Anthropic’s own breakdown, MiniMax accounted for roughly 13 million exchanges focused on agentic coding and tool orchestration, Moonshot about 3.4 million on agentic reasoning and computer use, and DeepSeek the smallest at around 150,000, weighted toward reasoning and chain-of-thought training data. Anthropic said it attributed the Moonshot campaign through request metadata matching the public profiles of senior staff, and that a later phase of that campaign attempted to extract and reconstruct Claude’s reasoning traces.

In a June 10 letter to Senate Banking Committee leadership, Anthropic alleged that operators affiliated with Alibaba and its Qwen lab generated more than 28.8 million exchanges through almost 25,000 fraudulent accounts between April 22 and June 5, exceeding the combined volume of the February cases, and asked Congress for better threat-information sharing, tighter chip access rules and penalties. On July 22, White House science adviser Michael Kratsios said publicly that the government had information Moonshot had distilled Anthropic’s Fable model to develop K3, without publishing the technical evidence, and Moonshot executives disputed the characterization in Chinese media, pointing to their own architectural work.

Then on September 8, the NSA, CISA and FBI released joint advisory AA26–251A, alleging systematic extraction by China-based AI companies through campaigns that form “the core, not merely a supplement, of their AI development strategy.” It names DeepSeek, Moonshot AI, Alibaba, MiniMax, StepFun and Z.AI, and says the firms pulled billions of tokens across millions of queries from Claude, GPT, Gemini and Grok since late 2024, likely with the knowledge of the Chinese government.

Five structural tensions are doing the work underneath all of this.

The cost asymmetry is the whole economic argument. Anthropic’s framing is that competitors acquired capabilities “in a fraction of the time, and at a fraction of the cost” that independent development would have required. That is accurate and it is also true of distillation generally. The teacher pays for the search. The student pays for the copy. A frontier pretraining run plus post-training plus safety work is a multi-billion dollar capital program. Twenty-nine million API exchanges is an operating expense, and a modest one.

The legal object is not copyright, which is why there is no lawsuit. Model outputs sit on unsettled copyright ground, so the claims that have teeth run through contract, trade secret and unauthorized access theories, plus export and sanctions policy. That is why the venue has been letters to senators and a joint advisory rather than a complaint. As of late July, no publicly identified court, regulator, arbitral tribunal or criminal proceeding had ruled that any of this was illegal. The Alibaba federal suit filed in June was about its Pentagon 1260H designation, a separate matter that gets conflated with the distillation claims in a lot of coverage.

Detection and attribution are weaker than the headlines suggest. Watermarking survives some distillation and supports a post-hoc ownership claim, but by the time you can read the watermark the student is already trained. Query-pattern detection catches coordination, shared payment instruments and implausible usage curves, which is evidence of fraud rather than evidence of distillation. Behavioral similarity between models proves very little when both were trained on overlapping web data. The attributions we have rest substantially on operational sloppiness by the other side.

The defenses degrade the product. This is the part I would put in front of any product team shipping an inference API. A less-noticed section of the advisory recommends targeted response changes for high-confidence malicious requests, including differential privacy or “using less sophisticated ‘downgraded’ models to respond to distillation requests,” suggesting reduced reasoning depth or stylistic inconsistency to evade detection, and advising firms not to inform suspected users that a switch has occurred.

Read that as a product requirement rather than as a security control. It says a commercial API may silently serve a worse model to accounts a classifier has flagged, and must not disclose this. Every enterprise buyer now has a new question for procurement, and the workloads most likely to trip such a classifier are high-volume synthetic data generation pipelines, which look exactly like distillation because they are distillation, authorized and paid for. The line between a customer and an adversary here is a terms-of-service clause and a classifier threshold, not a technical distinction.

The symmetry argument is real, and it is not dispositive. Chinese officials have called the crackdown a pretext for industrial monopoly and pointed out that American firms have themselves built on Chinese open-weight models, while arguing that distillation is a neutral and widely used practice. The first half of that is a fair observation about the industry’s history with scraped training data. The second half is where it stops working. The defensible distinction is not effort or originality, it is consent and contract. You can hold both that frontier labs built teachers on data they did not license, and that using 25,000 fraudulent accounts to evade regional access restrictions is a different kind of act.

There is also a safety claim running alongside the commercial one, which is that capabilities extracted from a guarded model arrive at the student without the guards, since the refusal behavior is the hardest part to transfer and the least useful to copy. Anthropic reported observing tasks in which Claude was used to generate censorship-safe alternatives to politically sensitive queries, apparently to train steering behavior away from censored topics. That detail cuts in an uncomfortable direction. The same extraction pipeline that strips one lab’s safety training can install another state’s.

What I cannot verify

The advisory did not publish its underlying technical evidence. Anthropic’s per-campaign numbers come from its own logs and its own attribution work, and no third party has audited them. The Kratsios statement about Fable and K3 was an official allegation without released technical support, and Moonshot denies it. One close reading of the 3,585-word advisory counts zero uses of the words theft, stolen or illegal, with the operative allegation being violation of terms of use. I have not verified that count myself, and it is the kind of claim worth checking before quoting.

Nothing here has been adjudicated. Treat every attribution in this piece as an allegation by an interested party, because that is what all of them currently are.

The question underneath

The uncomfortable thing about distillation is that it is not an attack on a flaw. It is a consequence of the product working. Carlini and colleagues showed in 2024 that the embedding projection matrix of OpenAI’s ada and babbage models could be extracted for under 20 dollars through ordinary API access, confirming hidden dimensions of 1024 and 2048, with the full matrix for GPT-3.5-turbo estimated at under 2,000 dollars in queries. Both providers hardened their APIs afterward. Log probabilities and logit bias were useful features. They were also the leak.

That pattern repeats at every level. Full distributions leak the representation. Top-k log probabilities leak enough for on-policy grading. Reasoning traces leak the search. Plain answers leak the capability slowly and reliably. The only response that closes the channel completely is not answering, and a model that does not answer is not a product. Every mitigation on the table is a decision about how much of the teacher’s usefulness to withhold from everyone in order to withhold some of it from six named companies.

So the questions I would rather see argued than the ones currently being argued. If the durable defense is making outputs less useful to copy, what does a frontier API look like in three years, and who bears that cost? If capability at this scale diffuses through any channel wide enough to be commercially viable, is the moat a real asset or an accounting convention? And if a student model that never saw a 3 can classify 98.6 percent of them correctly, what exactly did the teacher own?

---

Sanjay Basu | A Technocrat’s Discernment | sanjaysays.com