Scaling Native Multimodal Pre-Training From Scratch fits Chinchilla scaling laws separately for text and image losses in a from-scratch vision-language model, and finds the two objectives obey different allocation rules: language scaling ignores the image-to-text ratio, while multimodal scaling shifts more compute toward tokens, not parameters, as image data grows.
Say you’re planning the pre-training run for a vision-language foundation model your team will ship. Today the default recipe is Late-fusion vision-language model: take an existing LLM, bolt on a frozen CLIP-style vision encoder through a small projection layer, and continue-train on image-text data. It’s cheap because you reuse checkpoints, but vision and language were optimized separately on different data with different objectives, so the fused model inherits that seam.
The alternative, Native multimodal pre-training, trains one model from scratch on interleaved text and image tokens. Nobody had published a Chinchilla-equivalent recipe for it: how big a model, how many text tokens, how many image tokens, given a fixed FLOP budget? This paper is that recipe. Without it, teams doing native multimodal runs are guessing at the parameters-vs-tokens split that Chinchilla settled for text-only models three years ago.
The authors train a family of Mixture of Experts decoder-only transformers from 71M to 3B active parameters, feed images directly as patch embeddings (no separate vision encoder), and vary the multimodal data ratio r (image tokens / text tokens) across {0, 0.1, 0.2, 0.3}. Text budget is fixed at 250B tokens; image tokens go up to 75B.
The key move is to stop treating the joint pre-training loss as one number. They split it into a text loss and a multimodal loss, then fit compute-optimal scaling laws for each separately, using two independent estimators borrowed from Chinchilla: IsoFLOP profile and training-curve envelopes. Agreement between the two estimators is what lets them claim the exponents are real, not fitting artifacts.
For each objective they fit the standard power law: optimal parameters grow as compute^a, optimal tokens as compute^b, with a+b=1. Then they ask how a and b change as r changes. The headline finding is that the text exponents barely move with r, but the multimodal exponents drop sharply as r grows, meaning image-heavy mixtures want proportionally more tokens and fewer parameters.
Finally they combine both fits into a joint frontier that, given a total FLOP budget and a desired text/image loss trade-off, spits out a concrete (model size, text tokens, image tokens) triple.
for r in [0.1, 0.2, 0.3]:
# decouple compute per objective
C_text = 6 * N * D_text # D_text = D / (1 + r)
C_mm = 6 * N * D_mm # D_mm = D * r / (1 + r)
a_text, b_text = fit_isoflop(losses_text, C_text) # ~invariant in r
a_mm, b_mm = fit_isoflop(losses_mm, C_mm) # decreases with r
frontier = sweep_r(a_text, b_text, a_mm(r), b_mm(r), C_total)
The prevailing assumption when people extend Chinchilla to multimodal training is that one aggregate loss gives one set of scaling exponents, the way Shukor et al. native multimodal scaling modeled it. This paper shows the opposite. Language and vision objectives live in the same parameters but obey different allocation laws, and averaging them into a single loss hides the fact that adding image data pushes the compute-optimal frontier toward token scaling, not parameter scaling. The evidence that carries this claim is the divergence between the two per-objective exponent curves as r sweeps from 0.1 to 0.3, not any downstream benchmark score.
•
The load-bearing result: fitting per-objective, the language allocation exponent stays roughly flat across r, while the multimodal parameter exponent falls monotonically. At r=0.1 the joint-optimal parameter scaling is N_opt ∝ C^0.69; at r=0.3 it drops to N_opt ∝ C^0.66, with the token exponent rising to 0.34. Both IsoFLOP and envelope estimators agree, which is what rules out a fitting artifact.
•
Language capability is not taxed by multimodal data. Averaged across 16 text benchmarks, the 3B model scores 44.77 at r=0, 45.50 at r=0.1, 45.21 at r=0.2, and 46.03 at r=0.3. Deviations stay under one point at every model size from 71M to 3B.
•
Positive cross-modal transfer into text-only spatial reasoning. On the text-only spatial subtasks of SpatialEval (MazeNav, SpatialMap), models trained with r=0.3 beat r=0 baselines, and the gap widens with model size.
•
Multimodal in-context learning emerges with scale. At 71M, 1-shot and 3-shot prompting does nothing or hurts. At 3B, 3-shot lifts average multimodal accuracy by +2.43 points over 0-shot. The gain concentrates on spatial-reasoning and diagram tasks; OCR and recognition benchmarks flatten or regress with more shots.
Reach for this when you’re budgeting a from-scratch multimodal pre-training run and need to decide the (parameters, text tokens, image tokens) triple. The prescription: don’t reuse a text-only Chinchilla ratio. As you increase the share of image tokens, tilt the budget toward more tokens and a slightly smaller model than the text-only law would suggest. If you’re on the fence about whether to go native at all, the text-preservation result is reassurance that up to r=0.3 you’re not paying a language tax.
The paper releases no code, weights, or datasets that I can find in the text. All experiments cap at 3B active parameters, use a single (unnamed) image-text corpus, and rely on training loss as the scaling proxy rather than held-out validation, so the exponents themselves are the artifact you’d carry forward, not a checkpoint.
When you add image data to a from-scratch multimodal run, buy tokens, not parameters. Text scaling is stable enough to plan against with a text-only law, but the multimodal objective is data-hungry in a way that punishes over-parameterized, under-fed architectures at scale.
•
The whole story is fit on models up to 3B active parameters with an MoE backbone and a single image-text data family. Whether the exponents hold at 30B or with video, audio, or a different image distribution is untested.
•
Scaling laws are fit on smoothed training loss, not held-out validation, because the authors say solid multimodal validation metrics don’t exist. If training loss and generalization diverge for multimodal objectives, the frontier is optimistic.
•
The image-token range studied is narrow (r ∈ {0, 0.1, 0.2, 0.3}). The claim that language capability is preserved and that transfer helps spatial reasoning may not extend to image-dominant mixtures (r ≫ 0.3) where the shared-parameter competition looks very different.