Jointly training visual understanding and generation in one model helps both sides, but only if you specialize the visual computation for each objective. Routing all visual tokens through one path lets one objective dominate and starve the other by ~6 pp on the losing side.
You’re shipping a product that both reads images (answering questions, extracting layout) and generates or edits them (rendering diagrams, applying user edits). The current default is to bolt together two specialists: a vision-language model for the reading half, a diffusion model for the drawing half. A Unified Multimodal Model promises to fold both into one network so they can share knowledge, but nobody has cleanly shown whether that sharing actually helps or just papers over interference. This paper runs the controlled experiment and tells you what architecture choice determines which of the two you get.
The authors build a pixel-in, pixel-out model on top of Qwen3-1.7B. No pretrained vision encoder, no Variational Autoencoder on the generation side. Text is autoregressive as usual; image pixels are generated with Flow matching. They then compare three ways to route tokens through the backbone. In dense sharing, text tokens, clean image tokens (for reading), and noised image tokens (for generation) all go through the same LLM layers. In modality-decoupled Mixture-of-Transformers, text stays in the pretrained LLM, and all visual tokens (clean and noised) get shunted to a from-scratch parallel branch. Their proposed task-decoupled MoT splits differently: clean image tokens (understanding, or serving as edit context) stay with text in the pretrained LLM, and only the noised generation tokens go to a specialized visual branch. Cross-branch attention keeps them semantically linked.
The key diagnostic is Layer-wise Probing: freeze the trained backbone, then train tiny heads on top for ImageNet classification, ADE20K segmentation, and depth estimation. This measures what the visual features actually encode, independent of the training objective.
for tok in sequence:
if tok.is_text: route_to(llm_branch)
elif tok.is_clean_image: route_to(llm_branch) # understanding + edit context
elif tok.is_noised_image: route_to(gen_branch) # generation target
# all branches share global attention across the full sequence
loss = ce_loss(text_preds) + flow_matching_loss(image_preds)
The prevailing intuition about unified models is that either (a) more sharing means more transfer, or (b) understanding and generation fundamentally conflict and must be split. This paper shows both are wrong in the same way. Understanding and generation each produce useful learning signal for the other. Whether that signal shows up as mutual gains or as one side cannibalizing the other is decided by where you place the split in the architecture, not by whether you unify at all. The cleanest evidence is that dense sharing and modality-decoupled MoT fail in opposite directions on the same data, and task-decoupled routing recovers both.
The load-bearing result is the mirror-image failure between the two naive routings. In dense sharing, joint training lifts Vision-centric understanding to 61.77 (from 60.72) but drags GenEval2 down to 51.03 from 51.16 and hurts image quality metrics; the pretrained LLM anchors semantics and generation gets squeezed. In modality-decoupled MoT, the direction flips: GenEval2 jumps to 63.47 (from 57.55) but general understanding drops from 64.20 to 61.58; now generation dominates the shared visual branch. Same data, opposite winners. Layer-wise probing confirms both signals are real: generation supervision makes understanding features more semantically and geometrically informative, and understanding supervision raises text-image Centered Kernel Alignment alignment for generation.
The task-decoupled MoT avoids the trade-off, hitting 69.02 general understanding and 63.96 GenEval2 in a single model, matching or beating both specialists.
Task-level case studies then show transfer is real when domain knowledge overlaps:
•
Geometry: adding diagram generation lifts Geometry3K from 59.90 to 65.39 and MathVista from 71.80 to 74.30. Converting the same images into text descriptions instead of a generation target gives a smaller bump, so the generation objective itself matters, not just the extra pixels.
•
SVG (rendered icons vs. vector code): joint training improves both directions, and the model gets measurably better at a code-only “blind” VQA where it must mentally render the SVG without seeing the image.
•
3D spatial: eight of nine spatial-understanding benchmarks improve; view-prediction generation errors drop.
At the system level, on RISEBench and KRIS-Bench reasoning-heavy edits, an end-to-end UMM beats a matched planner-then-executor pipeline built from the same base model and data (18.88 vs 16.66 on RISEBench overall, 68.33 vs 66.48 on KRIS-Bench).
Reach for this when you’re designing a system that both reads and writes images, e.g. an editing product where the model must parse a user photo, decide what to change, and render the result. The takeaway for architecture: don’t force clean image tokens and noised image tokens down the same computation path. Keep the reading side anchored to your language model’s semantics; give the generation side its own parameters; let them talk through cross-attention on shared text. Also, if your two tasks share latent structure (geometry, vector graphics, 3D scenes), training them jointly will actually help both.
The paper does not release code, weights, or a named model checkpoint in the text provided. It’s a study paper, not a system release. What is directly reusable is the experimental setup: the three-way routing comparison, the layer-wise probing diagnostic for measuring representation quality, and the SVG blind VQA diagnostic for testing whether a model has internalized code-to-image simulation.
Unification is a routing decision, not a philosophy. Shared parameters transfer knowledge across understanding and generation only when you also give each objective its own place to specialize. Otherwise the stronger prior (the pretrained LLM, or the from-scratch visual branch, depending on your design) simply eats the other.
•
Results are on a single 1.7B base model with one specific generative formulation (autoregressive text + flow-matching images). The authors flag that fully-discrete or fully-autoregressive UMMs might behave differently, and don’t test them.
•
Task-level synergy was demonstrated only where the two directions genuinely share domain knowledge (geometry, SVG, 3D). Nothing here promises transfer between arbitrarily paired understanding and generation tasks.
•
The task-decoupled MoT is presented as a working solution, not the optimal one. MoE or other conditional routings might do as well or better; the paper doesn’t explore that space.