Get Started
Home
Topics
Search
Library
Research questionHow should subtoken granularity be chosen to reduce masked diffusion language-model training loss?Masked diffusion language models operating on subtokens can incur higher cross-entropy loss with BPE-based tokenizers. The choice of subtoken granularity also lacks clear guidance, making it difficult to relate tokenizer structure to training and downstream behavior.
AI
Diffusion Models
Evaluation & Benchmarks
LLM Pretraining & Post-training
Machine Learning
Natural Language Processing
Research Paper
Technology
Latest papersRecent research connected to this question, newest first.MDM-Prime-v2: Binary Encoding and Index Shuffling Enable Scaling of Diffusion Language ModelsThe source studies MDM-Prime, which applies diffusion at the sub-token level, and analyzes token granularity and sub-token entropy in relation to its training objective and downstream performance. Evidence includes scaling to 1.1 billion parameters and results on eight commonsense reasoning benchmarks; the reported comparisons concern the listed language-model baselines.research paper · Sep 2, 2026
Related questions
How can masked diffusion language models retain generation quality with only a few denoising steps?How can high-resolution diffusion Transformers prune tokens without sacrificing image fidelity or predictable compute?How can factual knowledge be edited reliably in masked diffusion language models for multi-token targets?How can discrete audio tokenizers preserve semantics and acoustic fidelity for both understanding and generation?