Get Started
Home
Topics
Search
Library
Research questionHow can autoregressive logit distillation preserve local next-token preferences when teacher and student rankings disagree?Matching every vocabulary logit can underrepresent distinctions among likely alternatives. Selecting candidates from only the teacher or student can also omit useful tokens or trust unreliable early student rankings.
AI
LLM Pretraining & Post-training
Machine Learning
Natural Language Processing
Latest papersRecent research connected to this question, newest first.ALRA: Adaptive Local Relational Alignment for Logit-Based Pre-training Distillation of Autoregressive Language ModelsThe source studies logit-based distillation with randomly initialized 200M- and 500M-parameter students on The Pile, using teacher and student next-token distributions. Its evidence compares the proposed adaptive local alignment and relational objectives with competing distillation and no-distillation baselines across nine zero-shot benchmarks.research paper · Sep 3, 2026
Related questions
How can on-policy distillation remain stable when policy updates change future training states?How can on-policy distillation prevent early student errors from corrupting rollouts in autoregressive vision-language models?How can model distillation block hidden teacher-trait transfer through clean data without degrading the target task?How should on-policy distillation select examples for data-efficient reasoning post-training?