Get Started
Research questionHow can LLM pretraining avoid sudden gradient explosions when scaling to larger models?Pretraining can abruptly fail when gradients explode, wasting the computation invested before the collapse. The failure is preceded by declining weight-matrix stable rank and increasing alignment between adjacent-layer Jacobians.
LLM Pretraining & Post-training
Machine Learning
Latest papersRecent research connected to this question, newest first.MSign: An Optimizer Preventing Training Instability in Large Language Models via Stable Rank RestorationThe evidence concerns NanoGPT-style LLMs scaled with μP from 5M to 3B parameters. It provides theoretical analysis of the instability mechanism and experiments with MSign, which periodically restores stable rank and reports less than 7% computational overhead.research paper · Sep 3, 2026
Related questions
How can large language models cut training and inference costs without materially harming accuracy?How can split-LLM training prevent returned gradients from revealing which rows are real?How can reliable foundation-model scaling laws be constructed without training every configuration?How can LLM agents generalize to unseen tasks without directly fine-tuning their policies?
Home
Topics
Search
Library