Get Started
Home
Topics
Search
Library
Research questionHow do pause tokens affect reasoning adaptation while preserving previously learned capabilities during LLM fine-tuning?Fine-tuning language models for reasoning can overwrite previously learned distributions and abilities. The training-dynamics role of pause tokens is less understood than their use as additional computation steps.
AI
LLM Pretraining & Post-training
Machine Learning
Reasoning
Latest papersRecent research connected to this question, newest first.Towards Understanding Pause Token Fine-Tuning Dynamics: A Mode Retention PerspectiveThe evidence concerns masked pause tokens placed at reasoning-step boundaries during fine-tuning, including synthetic continual-learning and math-reasoning probes. Experiments use 1B–8B Qwen and Llama models and also examine extension to GRPO; reported outcomes include mode retention, downstream-step information, math and code reasoning, and general language understanding.research paper · Sep 3, 2026
Related questions
How sparse can token-level supervision be in on-policy post-training without weakening language-model reasoning?How can large language models allocate reasoning computation to preserve accuracy under limited training and inference budgets?How should LLM pre-training allocate a fixed token budget between repetition and auxiliary views when prior knowledge is incomplete?How can reinforcement learning post-training prioritize useful reasoning prompts as learning signals shift?