Get Started
Home
Topics
Search
Library
Research questionHow can neural-network optimization make additive updates produce consistent relative changes across differently sized weights?Additive updates change parameters by similar absolute amounts, so a step can be negligible for a large weight but substantial for a small one. Optimization can therefore behave differently across weights because of their scale, even when update directions are comparable.
AI
LLM Pretraining & Post-training
Machine Learning
Neural and Evolutionary Computing
Research Paper
Technology
Latest papersRecent research connected to this question, newest first.Learning in Curved Weight Space:Exponential-Linear Weight Reparameterization for Improved OptimizationThe work evaluates a sign-aware exponential-plus-linear weight reparameterization during transformer training on OpenWebText across nine width-by-depth configurations, with small-model initialization ablations. It reports matched validation loss in 1.32–1.49 times fewer training steps, with larger widths showing the largest gains.research paper · Sep 3, 2026
Related questions
How can scientific machine-learning models overcome optimization plateaus when their local linearized subspace permits greater accuracy?How do stochastic optimizers progressively steer deep networks into simpler invariant subnetworks?How can LoRA fine-tuning optimize fixed-rank updates while respecting the induced weight-matrix geometry?How do architecture and optimization shape accessible representations and scaling at finite budgets on the same data?