Get Started
Home
Topics
Search
Library
Research questionHow can we distill capable small language models with fewer training tokens without losing teacher behavior?Distillation can lose information when teacher weights are compressed, representations are poorly aligned, or informative feed-forward activations are ignored. The resulting small model must retain the teacher’s behavior while requiring substantially less training compute and data.
AI
LLM Pretraining & Post-training
Machine Learning
Natural Language Processing
Research Paper
Small / On-device Models
Latest papersRecent research connected to this question, newest first.A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank CloneThe source concerns pre-training small language models from open-source teachers, including Llama and Qwen instruction-tuned models. Its reported evidence comes from experiments claiming comparable or stronger performance with 20B training tokens; broader behavior preservation beyond those evaluations is not established.research paper · Sep 1, 2026
Related questions
How can contrastive distillation efficiently transfer representations to smaller students without memory banks or fixed temperatures?How can model distillation block hidden teacher-trait transfer through clean data without degrading the target task?Can post-training ternarization make language models smaller without unacceptable capability loss or slower inference?How can one deployable LLM learn from multiple specialized teachers when the most reliable teacher varies by sample?