Get Started
Home
Topics
Search
Library
Research questionHow can LoRA fine-tuning optimize fixed-rank updates while respecting the induced weight-matrix geometry?LoRA trains two factors whose product defines a low-rank weight update. Treating those factors independently can produce steps that do not align with the geometry of the fixed-rank matrix changes they induce.
AI
LLM Pretraining & Post-training
Machine Learning
Natural Language Processing
Statistical Machine Learning
Latest papersRecent research connected to this question, newest first.LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style UpdatesThe evidence concerns LoRA fine-tuning of Llama-3.2-1B, Llama-3.1-8B, and Qwen3-32B on six commonsense and natural-language-inference benchmarks. It includes convergence guarantees for LoRA-Pro and LoRA-TSD under a tangent-projected-gradient stationarity measure, plus empirical comparisons with competing LoRA optimizers and adapter ranks.research paper · Sep 2, 2026
Related questions
How can LoRA initialization preserve full-rank training gradients despite its low-rank bottleneck?How can LoRA adapters be merged into native 4-bit microscaling checkpoints while preserving adaptation and the quantized code plane?How can multi-domain MoE fine-tuning prevent negative transfer when token-level routing separates domains but LoRA updates share subspaces?How can independently trained, frozen LoRA reasoning experts be combined without disrupting their learned update scales?