Get Started
Home
Topics
Search
Library
Research questionHow can language-model attention remain reliable beyond its training context?Attention and positional mechanisms can become unreliable when inference contexts exceed the lengths seen during training. Their weighting can also lack a clear account of how information evolves across positions.
AI
LLM Pretraining & Post-training
Machine Learning
Mechanistic Interpretability
Natural Language Processing
Research Paper
Statistical Machine Learning
Latest papersRecent research connected to this question, newest first.Robust Filter Attention: Self-Attention as Precision-Weighted State EstimationThe source studies Robust Filter Attention, which treats tokens as noisy observations of a latent trajectory governed by a linear stochastic differential equation. Under isotropic-noise and decay assumptions, it retains standard-attention complexity; language-modeling benchmarks report lower perplexity than RoPE within the training window and stable zero-shot extrapolation to longer contexts.research paper · Sep 3, 2026Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture DesignThe source studies RoPE-based Qwen3-series and Llama3.1 models, along with controlled Transformers, using RFIS and RPD intervention metrics to analyze head functions. It reports a head-wise hybrid design using NoPE full attention for position-independent retrieval and linear attention for local positional modeling, with results showing improved retrieval and long-context extrapolation at a full-to-linear attention ratio below 1:3; generality beyond the studied models and configurations is not established.research paper · Sep 2, 2026
Related questions
How do attention and recurrent state divide factual retrieval from output style in hybrid language models?How should language-model robustness be evaluated when text perturbations affect hidden states and attention differently from outputs?How can sparse autoencoders capture language-model features that persist across token sequences?How can we measure and control an LLM’s reliance on token-frequency priors when context is sparse?