Get Started
Home
Topics
Search
Library
Research questionHow can weight-inheritance distillation train the full deployed student matrix rather than a restricted slice?A student may deploy a full-width MLP matrix while optimization can vary only a teacher-induced slice of that matrix. The unreachable parameters consume inference capacity but cannot improve the student, potentially limiting distillation efficiency.
AI
LLM Pretraining & Post-training
Machine Learning
Natural Language Processing
Research Paper
Latest papersRecent research connected to this question, newest first.Train What You Deploy: Closing the MLP Reachability Gap in Low-Rank Clone DistillationThe problem concerns Low-Rank Clone distillation for compressed language models, evaluated on students derived from Llama3.2-3B, Llama3.1-8B, and Qwen2.5-3B teachers. The reported evidence uses single-seed runs on the LRC backbone, roughly 10B distillation tokens plus short SFT, and includes a residual MMLU deficit.research paper · Sep 2, 2026
Related questions
How can contrastive distillation efficiently transfer representations to smaller students without memory banks or fixed temperatures?How can one deployable LLM learn from multiple specialized teachers when the most reliable teacher varies by sample?How can we distill capable small language models with fewer training tokens without losing teacher behavior?How can model distillation block hidden teacher-trait transfer through clean data without degrading the target task?