Get Started
Research questionHow can Muon use gradient-variance information to improve language-model pretraining efficiency?Muon orthogonalizes momentum updates but does not explicitly account for gradient variance. That omission may limit convergence speed and compute efficiency during language-model pretraining.
LLM Pretraining & Post-training
Machine Learning
Statistical Machine Learning
Latest papersRecent research connected to this question, newest first.Variance-Adaptive Muon: Pre-Orthogonalization Variance Modulation for Efficient Language Model PretrainingThe evidence concerns Muon-NSR and Muon-VS, which add variance modulation while retaining Muon’s spectral normalization and requiring one additional variance buffer. Experiments cover Llama-style and GPT-2 pretraining at 125M–1.2B parameters; on Llama-1.2B, Muon-VS reaches the well-tuned Muon baseline’s final validation loss in 1.33× fewer steps.research paper · Sep 1, 2026
Related questions
How can LLM pretraining avoid sudden gradient explosions when scaling to larger models?How can validation loss track language-model capabilities across changing training distributions without benchmark evaluation?How can large language models cut training and inference costs without materially harming accuracy?How can idle inference resources reduce scarce-GPU training cost without biasing gradient estimates?
Home
Topics
Search
Library