Get Started
Home
Topics
Search
Library
Research questionHow can we measure and control an LLM’s reliance on token-frequency priors when context is sparse?With few clues, a language model may fall back on token frequencies from its training corpus rather than context-specific evidence. The difficulty is determining when that fallback occurs and how strongly it influences predictions.
AI
LLM Pretraining & Post-training
Mechanistic Interpretability
Natural Language Processing
Latest papersRecent research connected to this question, newest first.The Geometry of Ignorance: LLMs Know When to Temper Bayesian PriorsThe evidence comes from internal analyses of Llama, Qwen, Gemma, and Pythia models ranging from 0.4B to 405B parameters, requiring access to their unembedding matrices and final prediction states. It identifies a measurable prior-loading factor that decreases with more informative context and shows that changing the final state can steer predictions toward or away from the unigram prior, assessed by KL divergence.research paper · Sep 2, 2026
Related questions
How should low-resource LLM fine-tuning use task-level language priors with ambiguous or incomplete data?How should LLM pre-training allocate a fixed token budget between repetition and auxiliary views when prior knowledge is incomplete?How can language-model attention remain reliable beyond its training context?How can input-adaptive sparse attention reduce long-context prefilling cost without losing retrieval-relevant context?