Get Started
Home
Topics
Search
Library
Research questionHow can causal transformers use task state discovered late in a long context to guide rereading?Causal transformers process tokens in one direction, so information discovered late cannot directly influence how earlier context was processed. This creates difficulty for long-context reasoning tasks that require newly inferred task state to guide another reading of the context.
AI
Evaluation & Benchmarks
Natural Language Processing
Reasoning
Latest papersRecent research connected to this question, newest first.Trace as State: Reasoning Traces as Conditional States for Long-Context TransformersThe evidence concerns long-context reasoning with causal transformers, using collected textual reasoning traces as a proxy for task state. It compares placing that trace before the long-context block on a fresh pass with appending it after the block, across three models and three long-context datasets; the reported results favor the former in 26 of 27 model-task-metric combinations.research paper · Sep 2, 2026
Related questions
How can language models compress conversational and document context without reconstruction while preserving task-relevant information?How do attention and recurrent state divide factual retrieval from output style in hybrid language models?How reliably can chain-of-thought text reveal which reasoning steps causally drive correct answers?How can retrieval-augmented generation reason causally over dynamic external information without losing context?