NeoMME processes text tokens and raw 32×32 image patches through a single bidirectional Transformer trained from scratch with Masked discrete diffusion, letting a 260M-parameter retriever match models 14× its size on visual document retrieval while indexing pages at roughly 2× the throughput of a comparable ModernVBERT-based baseline.
You’re building a search system over PDFs, invoices, or slide decks. The current playbook staples a pretrained vision encoder (usually SigLIP) onto a language model, then repurposes that generative stack as an embedder, like ColPali does. You’re paying for a decoder you never sample from and a vision tower whose training objective wasn’t retrieval. NeoMME’s argument: skip both. One bidirectional encoder, trained from scratch, eats text and raw pixel patches through the same layers, and hands you dense plus late-interaction embeddings from a single forward pass.
The model has no separate vision tower. RGB images get cut into 32×32 patches, each patch (3,072 raw pixel values) goes through a small MLP into the shared hidden space, and text tokens enter through a factorized embedding. From layer one, image patches and text sit in the same residual stream and attend to each other bidirectionally. Position information is a 2D rotary scheme: text keeps a diagonal coordinate, image patches get row/column coordinates, so the same Rotary Position Embedding (RoPE) mechanism handles both modalities without a separate visual position encoder.
Pretraining is text denoising, not next-token prediction. For each training segment the model picks a corruption rate, replaces that fraction of text tokens with a mask token, and predicts the originals. For multimodal segments the minimum corruption rate is bumped to 30%, so the model can’t lean entirely on surrounding text and has to actually read the image. No pixel reconstruction loss, no image classification head, no contrastive image objective. Images are just conditioning for text denoising.
for segment in packed_batch:
if segment.has_image:
rho = uniform(0.30, 1.0) # force image reliance
else:
rho = uniform(0.0, 1.0)
masked_text = mask_tokens(segment.text, rate=rho)
logits = encoder(masked_text, segment.patches)
loss += cross_entropy(logits[masked], segment.text[masked]) / max(rho, 0.05)
For retrieval, they fine-tune with two heads sharing the backbone: a dense head (mean pool, L2 normalize, trained at multiple Matryoshka Representation Learning widths) and a late-interaction head (128-dim projection per token, scored with MaxSim / late-interaction scoring). Both come out of one forward pass, so at serving time you can dense-retrieve a shortlist and rerank with late-interaction.
The prevailing recipe for a document retriever glues a pretrained vision encoder to a pretrained language model and fine-tunes the seam. This paper shows the opposite. A single bidirectional encoder trained from scratch on raw patches and text, with only a masked-text objective, learns retrieval-grade multimodal representations without a vision tower or a decoder. The load-bearing evidence is the image-gain probe: at 90% text masking, page patches lift masked-token recovery by ~40 points, which means the model genuinely reads images even though nothing in the loss ever asked it to describe one.
The finding that makes the thesis credible is the Image gain probe measurement during pretraining. When 90% of text is masked, giving the model the page patches versus a zero-tensor image improves masked-token accuracy by 38.4 points for the 260M model and 40.5 points for the 800M. Image reading is emergent from text denoising alone.
Secondary evidence that this translates to a useful retriever:
•
On ViDoRe v3, NeoMME-260M scores 0.523 nDCG@10, beating every evaluated model below 800M parameters and landing within 0.2 points of a 3.75B ColQwen2.5 model that’s 14.4× larger.
•
NeoMME-800M reaches 0.556, within 0.9 points of a similarly sized Vultron Flash baseline.
•
On BEIR-15 text retrieval, the same backbone’s late-interaction head hits 0.488 (260M) and 0.513 (800M), showing the encoder isn’t just a document-image specialist.
•
At 2048×2048 input on an L40S, the 260M model encodes 51.3 pages/sec, about 1.97× the throughput of ColModernVBERT at matched resolution.
•
Compression stacks well: hierarchical token pooling at factor 8 plus int8 queries and binary documents shrinks a ~1.5 MB late-interaction page embedding to 6 kB (255× smaller) while keeping over 95% of nDCG@10.
The qualitative interpretability check is also interesting: similarity maps show strong activations on chart titles, axis labels, and numbers for the query token “hour”, suggesting the model developed OCR-like behavior without an OCR pipeline.
Reach for this when you’re shipping visual RAG over a large PDF corpus and today you’re running ColPali or a Qwen-VL-based retriever whose per-page embedding is a megabyte-plus and whose indexer throughput is your bottleneck. NeoMME-Retriever-260M gives you late-interaction quality on par with 3B+ models, dense-plus-late from one forward pass so you can do multistage retrieval cheaply, and a compression story that makes billion-page corpora storage-feasible.
Artifacts are on Hugging Face under Apache 2.0: pretrained backbones (260M and 800M), retrieval-fine-tuned checkpoints, predecay checkpoints for continued pretraining on your own data, and a day-zero integration into Hugging Face Transformers. They also release Late-Interaction Kernels, fused MaxSim kernels that cut memory for late-interaction training and inference (useful even if you don’t use NeoMME itself).
A vision tower is a convenience, not a requirement. Raw patches into a bidirectional encoder, denoised against masked text, is enough to learn document retrieval. The efficiency win comes from not carrying a pretrained-elsewhere vision stack you never really needed for the retrieval task.
•
Frozen natural-image transfer is weak (13.2% average on 16-shot probes across 10 classification tasks). The pretraining objective never optimizes a global image representation, so if you want a general-purpose vision embedder rather than a document retriever, this backbone isn’t it without adaptation.
•
Pretraining and retrieval supervision are both modest by current standards. Text pretraining is ~290B tokens versus ModernBERT’s ~2T, and retrieval fine-tuning sees ~430K text queries versus mLateOn’s ~660M. The paper can’t say how much of the gap to larger retrievers is architecture versus data scale.
•
Visual retrieval training data covers only six languages (English, French, German, Italian, Spanish, Portuguese via query augmentation). The multilingual claim is stronger for text than for page images, so non-Latin-script document retrieval is untested.