ScriptMoE recognizes scene text across 10 writing systems with one model by routing each cropped image to two Mixture of Experts experts aligned with script families (Latin, CJK, Arabic, others), plus an always-on shared expert that carries symbols and geometry common to all scripts.
You’re building an OCR pipeline that needs to read street signs, receipts, and packaging in Arabic, Thai, Tibetan, Russian, and a dozen other scripts. Today you have two bad options.
Option one is what PP-OCRv5 MLT does: keep a separate recognizer per language and route each cropped word through a language classifier first. If the classifier picks the wrong language, the recognizer downstream has no way to recover, and you’re maintaining ten-plus models instead of one.
Option two is to throw a big vision-language model at it. Systems like Qwen3.5-9B or GOT-OCR 2.0 claim broad script coverage, but the paper shows several of them score near zero on Arabic, Bangla, or Tibetan. They’re also one to two orders of magnitude larger than a dedicated recognizer, which rules out edge deployment.
The paper wants a single Scene Text Recognition model that is smaller than a VLM, simpler than a per-language cascade, and more accurate than either on the long tail of scripts.
The core observation is that any single scene-text crop contains one script, occasionally two. So you don’t need a decoder that is fluent in every script at every decoding step. You need a decoder that can pick the right specialist once, per image, and stay there.
Concretely, ScriptMoE keeps a shared visual encoder (SVTRv2) that turns the image into a sequence of visual tokens. In the autoregressive Transformer decoder, the usual feed-forward layer is replaced with a sparse Mixture of Experts block. Four experts each specialize in one script family: alphabet (Latin plus Cyrillic), CJK, Arabic, and “others” (Hindi, Bangla, Thai, Tibetan). A fifth shared expert is always active and absorbs cross-script material like digits, punctuation, and text distortion patterns.
Routing happens at the image level, not per token. The router mean-pools the visual tokens into one vector per image, produces a softmax over the four experts, picks the top two, and uses that same pair for every output token of the image. This avoids token-level “ping-ponging,” cuts routing cost, and makes each expert directly readable as a script specialist.
To keep the router honest, a small auxiliary head predicts the script group from the same pooled visual vector, supervised by a label derived automatically from the Unicode ranges of the ground-truth transcription. The auxiliary loss is weighted at 0.1, enough to nudge experts toward scripts but not enough to override the router.
for image in batch:
F = visual_encoder(image) # visual tokens
f_bar = mean_pool(F) # one vector per image
if training: f_bar *= (1 + sigma * noise)
probs = softmax(W_router @ f_bar)
experts = top2(probs) # same 2 experts for all tokens
for t in decode_steps:
h = decoder_attention(h, F)
routed = sum(g_i * expert_i(h) for i in experts)
shared = shared_expert(h)
alpha = sigmoid(w_s @ h) # per-token blend
h = alpha * shared + (1 - alpha) * routed
On the data side, the authors also built TextMuSS-10M, a synthetic dataset of 1M images per script across 10 scripts and 229 languages, because real training data barely exists outside English and Chinese.
On TextMuSS-Bench (10,899 real scene-text images across the 10 scripts), ScriptMoE reaches 82.06% average word accuracy, +1.31 points over the strongest specialized STR baseline (SVTRv2-AR) trained on the same data. Gains concentrate on the visually hard low-resource scripts: +2.98 on Arabic, +2.40 on Thai, +1.96 on Tibetan.
Zero-shot VLMs trail by 18 to 70 points on average, and several collapse to near-zero accuracy on Arabic, Bangla, or Tibetan. This is the paper’s main negative finding about VLMs: broad training coverage does not translate into reliable per-script accuracy.
On end-to-end multilingual OCR (CC-OCR), swapping only the recognizer in PP-OCRv5 (keeping its detector) lifts F1 from 65.71% to 80.89%, just past the best zero-shot VLM at 80.73%, at roughly one percent of the parameter count. Because the detector is held fixed, the improvement is attributable to recognition alone.
Ablations isolate what matters. Four experts beats two or ten (ten spreads samples too thin per expert). Top-2 activation beats Top-1 or Top-4. Removing the shared expert costs 0.71 points on average, concentrated on Arabic (-3.62) and Tibetan (-3.65). Removing the script-classification signal costs 0.43 points. Token-level routing performs about the same as image-level but costs more.
One notable weakness: Russian is the paper’s worst script. Cyrillic and Latin share visually identical glyphs (homoglyphs) that map to different characters, and Latin has abundant real training data while Cyrillic relies on synthetic. ScriptMoE reduces but does not eliminate this confusion.
If you already ship a per-language OCR cascade and are tired of maintaining language-ID plus N recognizers, this is a plausible drop-in: the paper demonstrates it works when you keep an existing detector and swap only the recognizer. The full model is ~46M parameters with ~41M activated per image, which is edge-deployable. Code is on GitHub.
If you’re evaluating whether a big VLM can replace your OCR stack, treat this paper’s per-script table as a warning. Zero-shot accuracy on scripts outside the model’s dominant training distribution can be near-zero even for well-known systems, and the aggregate score hides this. Worth checking your target scripts individually before committing.
If you’re designing your own Mixture of Experts system for a task where each input has a clear discrete type (language, modality, domain), the image-level routing pattern here is worth borrowing. Routing once per input rather than per token gives you interpretable experts, stable training, and lower routing overhead, at the cost of losing token-level flexibility you probably don’t need.
Worth testing, not established by the paper: whether the script-classification auxiliary loss transfers to other one-input-one-type domains, and whether adding a new script really requires only training a new expert rather than the full model. The authors flag continual learning as future work.
Training data outside English and Chinese is almost entirely synthetic, and the authors acknowledge a synthetic-to-real domain gap they haven’t closed. The Cyrillic-Latin homoglyph problem is a concrete manifestation: even a balanced synthetic distribution can’t fully resolve visually identical characters that map to different Unicode points.
The end-to-end results depend on the PP-OCRv5 detector, so gains are bounded by upstream detection quality, which the authors note is uneven across scripts. Nothing in the paper establishes that a different detector would preserve the reported F1 numbers.
The expert grouping (4 script families) is hand-designed from character morphology. The paper shows 10 experts (one per script) actually hurts because each expert sees too few samples. This means scaling to genuinely new scripts requires deciding which existing family they join, or accepting reduced per-script capacity.