When training a single model to do language, image understanding, and image generation together, start all three from step zero and skew the data toward language and understanding. Generation piggybacks on those priors and only needs ~5% of tokens to reach near-peak quality.
You’re building a product model that reads text, answers questions about images, and generates images. The default recipe today: take a pretrained LLM, bolt on a vision encoder later, fine-tune. This paper argues that bolt-on order actively hurts you, because the language side hardens and the vision pathway never fully engages. It’s an empirical physics-of-training study of the Transfusion family of unified models, with controlled sweeps rather than a new architecture.
The authors run controlled pretraining sweeps on a 1.5B Llama 3-style backbone, then validate at 13.5B with an Mixture of Experts setup on 2T tokens. Four questions, four setups.
First, knowledge flow. They hold one modality’s token budget fixed and scale another on top, measuring what transfers where. They repeat this on a synthetic CLEVR testbed where they can surgically delete a concept (e.g., “yellow”) from one stream and check whether the other stream teaches it anyway.
Second, synergy vs competition. They vary data complexity (from solid-color images up to real photos, from repeated-letter text up to web text) and watch when adding a second modality helps or hurts the first. Then they dissect the transformer block: which parts should be shared across modalities, which should be split.
Third, timing. They fix a 1T-token budget and sweep how many pure-language tokens come before vision is introduced (0B, 200B, … 800B). They also try all six sequential orderings of the three objectives.
Fourth, recipes. They grid-search the language/understanding/generation mix ratio at 1T tokens and pick a winner.
# Controlled sweep pattern used throughout
for lang_warmup in [0, 200, 400, 600, 800]: # B tokens
model = init_from_scratch()
train(model, data="language_only", tokens=lang_warmup)
remaining = 1000 - lang_warmup
train(model, data=mix(lang=0.5, vision=0.5), tokens=remaining)
eval(model, [language_bench, vqa_bench, geneval_bench])
The prevailing recipe treats vision as a module to attach to a finished LLM: pretrain language, then align vision, then maybe add generation. This paper shows the opposite. Start all three modalities together from step zero, share attention and normalization while splitting the feed-forward layers per modality, and heavily skew the token budget toward language and understanding. The load-bearing evidence isn’t the leaderboard, it’s the vision-laziness probe: the longer you delay vision, the quieter the vision feed-forward layers get at both training and inference time, and the less the model actually attends to image tokens.
•
Vision laziness is measurable, not just a metaphor. Across four independent probes (training-time activation norm, wrapper-token embedding norm, inference activation, and attention-mass on image tokens), all four monotonically drop as you extend the pure-language warmup from 0B to 800B tokens. The late-aligned model literally stops looking at the image and falls back on language priors.
•
Knowledge flow is asymmetric. Language boosts both understanding and generation universally. Understanding strongly boosts generation. Generation gives essentially nothing back to language or understanding, with a minor exception for object counting.
•
On the synthetic CLEVR setup, low-level attributes like color and shape do not transfer zero-shot in either direction, but structural concepts (spatial relation, size, count) transfer from understanding to generation.
•
Simple beats complex for cross-modal synergy. Pairing language with solid-color background images improves language perplexity more than training language alone. Pairing image generation with degenerate “repeated-letter” text improves generation loss more than pairing it with real web text.
•
Architecture verdict: fully shared weights (“dense”) make both modalities worse than unimodal baselines. Splitting only the Feed-Forward Network while keeping attention and normalization shared gives the biggest synergy. Splitting everything reduces to two independent networks.
•
Recipe verdict: a 70% language / 25% understanding / 5% generation mix wins the grid search. At 13.5B / 2T tokens, this beats a balanced 50/25/25 mix on language accuracy (54.3 vs 52.9), understanding average (43.1 vs 41.4), and text-to-image alignment (GenEval 0.482 vs 0.467), using 5× fewer generation tokens.
Reach for this when you’re planning a from-scratch multimodal pretrain, or deciding whether to “add vision later” to an in-house LLM. The paper’s concrete advice: don’t. Start joint from step zero, use a modality-split Feed-Forward Network (or Mixture of Experts with a couple of modality-pinned experts) on top of shared attention and normalization, and spend most of your token budget on language plus image-understanding pairs. Generation quality comes almost for free once those priors are strong.
The paper does not release code, weights, or the synthetic CLEVR-extended dataset in the text provided. The recipes themselves (mix ratios, sharing pattern, timing) are the deliverable. Public building blocks they use include DCLM for language, Cambrian-7M for Visual Question Answering fine-tuning, and SigLIP-2 as one of the visual tokenizer options.
Train modalities together from day one, then feed the hungry one and starve the lazy one. Language and understanding are the priors that do the work; generation piggybacks on them cheaply. The moment you let language finish first, the vision pathway learns to coast on language priors and never fully wakes up, no matter how much vision data you pour in later.
•
Everything is text and static images. Video, audio, and action modalities are explicitly out of scope, and the synergy/competition tradeoff could shift when a temporal axis is added.
•
The 5%-generation-tokens result is measured on GenEval and DPG-Bench style compositional prompts plus diffusion loss. Pure image fidelity (FID 5.23 vs 5.13 for balanced) is slightly worse, so if photorealism is your product bar, you may still want a dedicated generation mid-training stage.
•
Scale ceiling is 13.5B Mixture of Experts on 2T tokens. The authors flag that at frontier scale (>1T params), a bigger model might have the capacity to make generation give something back to understanding, which would break the asymmetric-mix recommendation.