Pixel Linguist II learns to read text-as-image by pairing rendered-text training with real photos, varying font sizes and canvas dimensions as spatial proxies, and using a two-stage multilingual curriculum. Retrieval quality holds even when 80% of visual tokens are dropped.
You’re building a document search system over a mix of scanned PDFs, screenshots of dashboards, and product photos with captions. Today you likely run OCR, then embed the extracted text with one model and the images with another (a CLIP-style dual encoder), then fuse. That pipeline breaks on layouts OCR mangles: tables, multilingual scans, charts with tiny labels.
The pixel-text approach says: render every text query as an image too, and use a single vision encoder for both queries and documents. Prior work like ColPali pushed this on documents; this paper asks what training recipe actually makes a general pixel-text encoder work, not just a document-specific one.
The core object is a vision encoder trained contrastively on two kinds of pairs: (a) natural photos with their captions, and (b) text-text pairs where both sides are rendered into images. The encoder itself is a NaViT backbone initialized from Qwen2.5-VL’s ViT, which accepts variable resolutions and aspect ratios rather than forcing everything to 224x224.
Four design choices carry the paper. First, spatial proxies: instead of training on expensive 4K document images, keep the canvas small but randomize font sizes (12-28pt) and natural-image aspect ratios. The model learns “text can appear at any scale,” which transfers to dense high-resolution documents at inference. Second, keep the natural images. Training only on rendered text collapses on real documents even at 26M-pair scale. Third, layout-aware rendering: sample from 393 fonts, apply rotation, blur, brightness jitter, and 5,000+ textured backgrounds from Describable Textures Dataset (DTD) so the model never sees the same visual instantiation of a string twice. Fourth, a two-stage curriculum: first pretrain on 62M multilingual Wikipedia documents using random-crop pairs as unsupervised positives (Contriever-style unsupervised pairs), then mid-train on 26M curated semantic text pairs. Both stages mix in 26M LAION-2B image-text pairs.
for batch in stream: # 280M examples total across 2 stages
text_pairs, image_caption_pairs = batch
# Render text on-the-fly, never cached
imgs_a = render(text_pairs.a, font=sample_font(),
size=uniform(16,28), bg=sample_dtd())
imgs_b = render(text_pairs.b, ...) # different render
# Single encoder, unified pixel space
embeds = vit_navit([imgs_a, imgs_b, image_caption_pairs.img,
render(image_caption_pairs.caption)])
loss = infonce(embeds, temperature=0.03)
The prevailing intuition is that if your target is text-in-images, you should train on more text-in-images. This paper shows the opposite. Natural photo-caption pairs are not optional flavor; they are a load-bearing regularizer that prevents the encoder from collapsing into pixel-level shortcut memorization, and no amount of synthetic text scale substitutes for them. The cleanest evidence is the text-only ablation, which matches full training on simple similarity tasks but degrades sharply on real document retrieval (ViDoRe), at both 7M and full scale.
The load-bearing finding is the shortcut-learning collapse. Training with rendered text on fixed fonts and plain canvases with no natural images yields an average nDCG@5 of 1.65 on ViDoRe subsets, essentially zero retrieval signal. Adding font-size variation and image aspect diversity brings it to 37.83 on the same 13M budget. Fixing only font size drops it to 30.97; fixing only image resolution drops it to 33.45. These aren’t marginal effects, they define whether the model works at all.
•
On English Visual STS, Pixel Linguist II with only mid-training scores 74.72 average Spearman, beating the strongest baseline (EVA02-CLIP-bigE-14-plus at 71.99) at roughly 1/7 the parameters and 1/87 the examples seen. Fine-tuning on AllNLI adds ~5 points to 79.80.
•
On cross-lingual Visual STS across 11 language pairs, mid-training alone hits 53.83 average vs 38.88 for the strongest SigLIP variant. Adding the unsupervised multilingual pretraining stage lifts this to 57.16, confirming the curriculum ablation showing a ~3.3-3.5 point boost from foundational pretraining.
•
On ViDoRe, forcing SigLIP-so400m into the same vision-only setup (rendering the query as an image too) costs it 24.2 nDCG@5 on average, showing how much dual-encoder baselines depend on their dedicated text tower.
•
Under token compression, Visual STS holds parity with full CLIP at 60% tokens dropped; ViDoRe still beats uncompressed CLIP at 80% tokens dropped.
Reach for this when you’re building unified retrieval over screenshots, PDFs, and images where OCR is unreliable or multilingual. Render every text query as an image, embed both queries and documents with the single encoder, and use standard cosine similarity. The 80%-compression result matters for MLLM inference budgets: you can feed heavily downsampled visual tokens and keep most of the retrieval quality, useful if you’re stuffing document pages into a limited context window.
Code and models are at GitHub. The training data recipe uses public sources: LAION-2B image-text pairs, multilingual Wikipedia dumps, and DTD textures for background augmentation. The on-the-fly rendering engine (393 fonts, jitter parameters in the appendix) is the reusable artifact if you want to train your own variant. Fonts, font-size ranges, blur probabilities, and background probabilities are all specified.
If you want a vision encoder to read text, don’t train it only on text. The natural photos are what stop it from cheating on pixels.
•
The recipe assumes you can afford 280M training examples across two stages on 64 GPUs. The design ablations transfer, but reproducing the headline numbers is not cheap.
•
Weakness on diagram-heavy content (ArxivQA) is acknowledged by the authors; the rendering engine covers layout diversity for prose and tables, not scientific figures, plots, or handwriting.
•
The unified vision-only setup means textual queries lose the semantic richness a dedicated text encoder provides. For pure text-to-text retrieval with no visual documents in the mix, a standard text embedder will still beat this.