Get Started
Home
Topics
Search
Library
Research questionHow can language models sparsely access a growing internal state without limiting it to fixed-size memory?Sparse computation can reduce the cost of processing long sequences, but fixed-size states or static routing may prevent models from reaching relevant information as their internal state grows. The central problem is making selective access scalable without discarding that expanding state.
AI
AI Memory
Inference Optimization
LLM Pretraining & Post-training
Machine Learning
Natural Language Processing
Research Paper
Latest papersRecent research connected to this question, newest first.Graph Machine: Towards Better Pretraining via EdgesThe evidence concerns Graph Machine sparse layers with an O(n)-sized state and differentiably updated, dynamic pointer-like routing. The study replaces 75% of the dense layers in Qwen3-0.6B, pretrains from scratch on 15.7B tokens, and reports results when each sparse layer retrieves 2 or 4 of 4,096 tokens per KV head.research paper · Sep 2, 2026
Related questions
How can large language models cut training and inference costs without materially harming accuracy?How can sparse autoencoders capture language-model features that persist across token sequences?How can language models personalize for many users without costly per-user model storage?How can autoregressive language models add recurrent cross-token memory without breaking KV caching or one-forward-per-token decoding?