Get Started
Home
Topics
Search
Library
Research questionWhy do pretrained language models collapse continuous mixtures of hypotheses during latent-state reasoning?Latent-state reasoning can represent several candidate continuations as a weighted mixture rather than selecting one token. Architectural geometry and the model’s feedback dynamics can amplify small differences until one hypothesis dominates or contract distinct mixtures until they become indistinguishable.
AI
LLM Pretraining & Post-training
Machine Learning
Natural Language Processing
Reasoning
Latest papersRecent research connected to this question, newest first.The Dynamics of Continuous Mixture Collapse in Language ModelsThe evidence combines theoretical analysis with controlled empirical investigations across multiple language models. It identifies three independent failure sources, finds the observed contraction-to-amplification transition near the theoretical threshold, and reports that pretrained-model rollouts predominantly fall on the amplifying side; exact preservation for many components may require context-dependent corrections whose dimensionality grows with the number of components.research paper · Sep 2, 2026
Related questions
How can language-model attention remain reliable beyond its training context?How can we compare autoregressive and masked-diffusion language models without conflating formulation with architecture?How do attention and recurrent state divide factual retrieval from output style in hybrid language models?How can sparse autoencoders capture language-model features that persist across token sequences?