Smarter training trick boosts small AI reasoning without losing facts
Original: Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall
Source: arxiv.org ↗
Who: Jacqueline He, Howard Yen, and Shuyue Stella Li, researchers at the University of Washington and Meta, plus collaborators from multiple institutions, posted this paper on arXiv. They study how to train smaller AI models by learning from larger ones.
What's new: The team found that a common training shortcut behaves differently depending on when you use it. This shortcut, called , helps with reasoning skills but can actually slow down how fast a model learns to remember facts — but only in one specific phase of training.
How it works: AI models learn in stages, and this paper focuses on the "mid-training" phase, where the model reads a large collection of curated text. Normally, you train a model to guess the next word, an approach known as . With distillation, the student model instead tries to copy what the teacher model would predict. The researchers discovered teachers are very confident on procedural or step-by-step questions but less confident on fact-heavy ones, so copying them boosts reasoning but neglects facts. To fix this, they built a simple switch: distill only when the teacher feels sure, and otherwise fall back to ordinary next-word guessing. They call this "Switch Distillation."
The numbers: Switch Distillation beat existing methods across different teacher sizes. Compared with standard training, it improved reasoning performance by a factor of 1.61 to 1.71 and knowledge and commonsense skills by 1.13 to 1.19, while keeping almost all factual recall at 96.7 to 96.8 percent. Even after the final refined training stage, the reasoning gains stayed strong at 1.25 to 1.32 times better.
Why it matters: This shows that one-size-fits-all training tricks can backfire in unexpected phases. By designing a method that respects the teacher's own confidence, the team gets most of the reasoning benefit without sacrificing factual knowledge. This could make smaller models cheaper and better in practice.
Caveats: The study relies on controlled experiments, so real-world scale may differ. The authors do not fully explain why teachers feel more confident on reasoning tasks in the first place, leaving that question open.