Get Started
Home
Topics
Search
Library
Research questionDoes reusing Transformer layers improve language-model quality when parameter, compute, and KV-cache budgets are matched?Layer looping increases effective computation by revisiting shared parameters, but comparisons can mistake extra computation or memory for an architectural improvement. The central difficulty is isolating the effect of reuse while holding major training and inference budgets constant.
AI
LLM Pretraining & Post-training
Machine Learning
Natural Language Processing
Research Paper
Technology
Latest papersRecent research connected to this question, newest first.SMELT: Scaling Laws for Compute-Matched MoE Looped TransformersThe evidence concerns sparse Mixture-of-Experts Transformers, comparing an unlooped baseline with a model that revisits the middle half of its layers twice. The study matches per-token FLOPs, total non-embedding parameters, and KV-cache size across four scales up to 54B parameters, and reports scaling-law and downstream results, including stronger effects on code and longer in-context examples.research paper · Sep 7, 2026
Related questions
How can complete transformer layers be removed from LLMs without sacrificing quality or standard serving compatibility?How can long-context language models reduce per-token reads from million-token KV caches without losing accuracy?How can large language models cut training and inference costs without materially harming accuracy?How can shared-prefix KV caching accelerate diffusion language model serving without corrupting context-dependent KVs?