Get Started
Home
Topics
Search
Library
Research questionHow can deep-network momentum adapt forgetting to unevenly sampled input directions?A single exponential decay treats frequently and rarely activated directions alike. As a result, stale gradient information may persist in some directions while other directions are forgotten too quickly.
AI
LLM Pretraining & Post-training
Machine Learning
Research Paper
Statistical Machine Learning
Technology
Latest papersRecent research connected to this question, newest first.Activation-Keyed Momentum: An Anisotropic Momentum Update via the Delta RuleThe source studies Activation-Keyed Momentum as a replacement for the momentum update in optimizers including SGD and AdamW. It provides theory for input-side curvature correction and reports results in FineWeb-Edu language-model pretraining, ResNet-18, and ViT-Tiny experiments; the supplied evidence does not establish broader architectures or deployment settings.research paper · Sep 3, 2026
Related questions
How can deep continuous-time recurrent networks avoid depth-induced gradient attenuation under truncated temporal backpropagation?How should momentum and batch size be tuned to preserve stability and data efficiency in one-pass training?How can multilayer RNNs learn nonlinear quadrotor dynamics without vanishing or exploding gradients?How can video deepfake detectors adapt to new forgery patterns without forgetting prior spatial and temporal cues?