Get Started
Research questionHow can LoRA initialization preserve full-rank training gradients despite its low-rank bottleneck?LoRA replaces full-rank updates with low-rank factors, so the initial factors may induce gradients that differ substantially from those of full-rank fine-tuning. This mismatch can make adaptation performance highly dependent on initialization.
AI
Machine Learning
Technology
Latest papersRecent research connected to this question, newest first.TaRA: Training-Aware Low-Rank Adaptation InitializationThe source concerns LoRA initialization for parameter-efficient fine-tuning. Its evidence centers on an initialization designed to approximate full-rank weight gradients, with reported results across diverse fine-tuning tasks and negligible additional computation.research paper · Sep 2, 2026
Related questions
How can LoRA fine-tuning optimize fixed-rank updates while respecting the induced weight-matrix geometry?How can multi-domain MoE fine-tuning prevent negative transfer when token-level routing separates domains but LoRA updates share subspaces?How can LLM pretraining avoid sudden gradient explosions when scaling to larger models?When do tied or untied attention parameterizations enable weak recovery under stochastic training?
Home
Topics
Search
Library