Get Started
Home
Topics
Search
Library
Research questionHow can encoder Transformers preserve document-scale structure in positional encodings during masked language modeling?Transformers need positional signals to represent sequence order, but these signals can become entangled with semantic representations and lose document-scale structure. This complicates long-context understanding and retrieval.
Evaluation & Benchmarks
Machine Learning
Mechanistic Interpretability
Natural Language Processing
Latest papersRecent research connected to this question, newest first.Give it Space! Explicit Disentangling of Positional and Semantic Representations in EncodersApplies to encoder Transformers trained with masked-language modeling, focusing on absolute and relative positional representations and their behavior across layers. Evidence comes from mechanistic analyses and the Flash-Holmes linguistic probing benchmark, including results for 49 of 65 reported phenomena in the studied disentangled setup.research paper · Sep 2, 2026
Related questions
How can sparse autoencoders capture language-model features that persist across token sequences?How can sparse autoencoder features be shared across language models without per-model retraining?How can pixel-text encoders robustly represent grounded multilingual text across resolutions under tight visual-token budgets?How can language models compress conversational and document context without reconstruction while preserving task-relevant information?