Weak teachers can train large language models effectively

Original: Strong Teacher Not Needed? On Distillation in LLM Pretraining

Source: arxiv.org ↗