UltraBERT trains one bidirectional encoder that is good at both classification and retrieval by mixing two Masked Language Modeling masking rates in each batch (5% and 50%) with a separate output head per rate, and distilling only the low-rate head from a 1.3B teacher.
If you build search, Retrieval-Augmented Generation, reranking, quality filtering for pretraining data, or reward scoring, you probably still reach for a BERT-style encoder, not a decoder LLM. They are small, fast, and cheap enough to run over billions of documents. The catch today: no single modern encoder is simultaneously strong at classification (think GLUE, NER, reward modeling) and retrieval (think BEIR dense search). ModernBERT and Ettin lead on one side, retrieval-focused encoders like LFM2.5-Encoder lead on the other, so teams often ship two different backbones.
The authors argue this split is not fundamental. It is caused by the pretraining recipe people inherited from the original BERT era, specifically the masking ratio. They revisit that recipe at trillion-token scale and build a single encoder that tops both sides.
The paper has three empirical findings and then a model that follows from them.
Finding 1: encoders saturate. Train a 150M or 1.3B encoder on up to 8.8T tokens and downstream scores (GLUE, SQuAD 2.0, BEIR) stop improving at roughly 1-2T tokens, even though training and validation loss keep dropping. Decoders behave differently and keep improving. So throwing more pretraining tokens at an encoder past a few trillion is wasted compute.
Finding 2: masking ratio splits the two task families. Once you are past saturation (so faster-converging recipes no longer have an unfair edge), classification wants a low mask rate (~5-15%) and retrieval wants a high one (~50%). This flips the common prior-work advice that bigger models should mask more. The retrieval benefit tracks a higher Effective rank of the pooled representations, meaning the embedding space uses more of its dimensions. Adding a spreading regularizer (KoLeo regularizer) at low masking raises effective rank but still underperforms simply masking at 50%.
Finding 3: distillation is not free. Pruning a 1.3B teacher down to 380M and distilling helps classification at low mask rates, but at 50% masking it actually hurts retrieval versus training from scratch. The authors suspect the teacher’s logits are near-trivial when half the tokens are masked.
The mechanism: LM head decoupling. The output projection that predicts masked tokens (the “LM head”) takes on very different geometries at 5% vs 50% masking, while input embeddings look nearly identical. Sharing one head across both regimes creates gradient interference. The fix is to keep the encoder body shared but give each mask ratio its own LM head, and apply distillation only on the low-ratio stream.
for batch in data:
# 4:1 mix inside one batch
low = mask(batch, ratio=0.05) # 80% of sequences
high = mask(batch, ratio=0.50) # 20% of sequences
h_low = encoder(low)
h_high = encoder(high)
loss_low = KL(lm_head_low(h_low), teacher_1p3B(low)) # distill
loss_high = MLM(lm_head_high(h_high), high.labels) # standard MLM
(loss_low + loss_high).backward()
Final recipe: start from a 1.3B teacher trained under this scheme, prune to 380M, distill, then prune again to 150M and distill from the same 1.3B parent. Train on 2T tokens with native 32K context.
Against strong modern baselines (ModernBERT, Ettin Encoder, NeoBERT, LFM2.5-Encoder, plus BERT/RoBERTa), UltraBERT is the first encoder the authors know of to lead on both sides at both sizes.
•
Classification: 91.0 on GLUE and best or near-best on SQuAD 2.0, MultiCoNER v2, Few-NERD, and RewardBench at 380M. The 150M model matches RoBERTa-large on classification with ~2.3x fewer parameters.
•
Retrieval: 47.7 on BEIR at 380M, best on code and math subsets of RAR-b with gains of roughly 7-8 points over the next model at each size.
•
Long context: native 32K context gives the biggest edge on LongEmbed; UltraBERT keeps ~99% passkey recall out to 32K tokens, while 8K-context baselines collapse past their window.
•
Ablation that isolates the mechanism (Table 1, 150M): single 6% masking gets 89.5 GLUE / 43.5 BEIR; single 50% gets 87.5 / 45.0; mixing {5%, 50%} with a shared head gets 88.9 / 44.7; adding head decoupling gets 89.4 / 45.1, essentially matching the best of each specialist on its own side.
The ablation is the load-bearing evidence that decoupling, not just “see both ratios,” is what resolves the trade-off. The authors propose that shared gradients through one LM head force the two regimes to compromise on representation geometry.
•
If you currently run one encoder for classification and a different one for dense retrieval, the released weights (UltraBERT-150M, UltraBERT-380M) are worth benchmarking as a single replacement, especially for pipelines that also need long context (RAG over long documents, long-form reranking).
•
If you are pretraining your own encoder, two concrete takeaways: do not budget past ~1-2T tokens expecting classification or retrieval gains, and do not pick a single masking ratio. The paper’s specific recipe (mix 5% and 50% at 4:1, decouple LM heads, distill only the low-ratio head) is directly implementable.
•
If you are considering distillation to shrink an encoder, test it per task family. The paper’s evidence is that logit distillation lifts classification at low mask rates but can hurt retrieval at high mask rates, so applying it uniformly is risky.
•
Worth testing, not established by the paper: whether the same head-decoupling trick helps other multi-objective pretraining setups (for example, mixing Replaced Token Detection with MLM, or mixing contrastive and MLM losses). The paper only validates it for two MLM mask rates.
•
All results are English-only and the pretraining mixture is fixed; multilingual behavior is not evaluated.
•
The retrieval numbers come from fine-tuning each backbone on MS MARCO with a specific loss and then evaluating zero-shot on BEIR. Different fine-tuning recipes (hard-negative mining, multi-stage contrastive) could reorder the leaderboard.
•
The paper excludes RTD-trained encoders like DeBERTaV3 from the main table on the grounds that they are known to be weak at retrieval; it reports them only in an appendix comparison.
•
The claim that saturation is “fundamental” to encoders rests on comparing one 1.3B encoder to one 1.7B decoder (SmolLM2); it is suggestive, not a scaling law.
•
CoLA is treated as a confounder for MLM-vs-RTD comparisons and excluded from that specific analysis; this is defended in an appendix but is a judgment call.