Intern Lumina U2 is a unified diffusion language model that represents images, videos, and 3D assets as groups of 8 discrete codes per visual position, letting one backbone handle understanding and generation under a single masked-prediction objective.
If you want one model that both reads images and generates them, today you pick a tradeoff. Discrete-token approaches (one code per image patch) are elegant because images and text look alike to the model, but a single code can’t carry enough visual detail. Hybrid approaches keep images in continuous space for better fidelity, but then image generation uses a different loss than text, and you often need two visual encoders bolted together.
The authors want the clean-shared-objective story of discrete tokens without paying for it in visual quality. The relevant baseline is LLaDA2.0-Uni, which shares this paper’s language backbone but uses single-code visual tokens: a useful apples-to-apples comparison.
The trick: give each spatial position in an image a group of 8 codes instead of 1, drawn from 8 separate codebooks of 4,096 entries each. One position still occupies one slot in the backbone’s token sequence, so sequence length doesn’t blow up, but it now carries ~8x more information.
At the input, each of the 8 codes has its own embedding table. The 8 embeddings for a position are concatenated and projected down to the backbone’s hidden size. At the output, a small 6-layer causal Transformer called the multi-codebook AR head predicts the 8 codes one after another, each conditioned on the backbone hidden state and the previously-sampled codes at that same position. The chain rule guarantees this never does worse than predicting codes independently.
The backbone itself is a masked diffusion LLM: it refines all spatial positions in parallel (bidirectional attention over the image grid), while text is generated block-by-block left-to-right. A task-aware attention mask routes who-sees-what among conditions, noisy answers, and clean copies in a packed sequence. Visual inputs come from AToken, a frozen tokenizer that maps images, videos, and 3D into a shared 4D code space using Finite Scalar Quantization (FSQ).
# per refinement step t, for an image being generated:
h = backbone(canvas_with_mask_tokens) # bidirectional, parallel
for pos in masked_positions: # all positions in parallel
prefix = [W_q @ h[pos]]
for k in range(8): # codebooks, sequential
logits = ar_head(prefix, codebook=k)
c_k = sample(logits)
prefix.append(embed_k(c_k))
canvas[pos] = (c_1,...,c_8)
# remask low-confidence positions per cosine schedule; repeat
For inference, they add PHV (Plateau-Halton sampling with V-verify): commit a position’s codes once its confidence stops changing (plateau test), spread commitments evenly over the canvas using a Halton sequence ordering, and refresh the backbone’s hidden-state cache only when a drift metric exceeds a threshold.
At 1.4B activated parameters (16.9B total, Mixture of Experts with 8 of 256 experts per token), the model is competitive across a wide range of tasks:
•
Vs. the same-backbone single-code baseline LLaDA2.0-Uni: better on 8 of 10 image-understanding benchmarks, with the largest gains on tasks needing fine visual detail. This is the cleanest evidence that multi-codebook representations help.
•
Image generation: ties or beats other unified models on GenEval, DPG-Bench, TIIF-Bench, and UniGenBench. Approaches Qwen-Image (which activates ~14x more parameters per token) on several of these.
•
Video and 3D understanding: leads on three video benchmarks and on 3D MM-Vet. No other unified-generation baseline in the comparison even supports video and 3D in one checkpoint.
•
Output-head ablation: dropping the autoregressive head and predicting the 8 codes independently costs ~14 points on GenEval (82.8 to 68.9), with the biggest losses on counting, position, and attribute binding. A Q-Former alternative does no better. This isolates the gain to modeling dependencies among the 8 codes at a position.
•
Inference: PHV sampling cuts text-to-image latency from 159.9s to 35.9s per 1024x1024 image (4.45x speedup) while GenEval goes from 86.6 to 89.3 on the same protocol. The authors attribute part of this to the Halton reveal order rather than the plateau test alone.
•
If you’re choosing a visual tokenizer for a unified model and you’ve been assuming you need continuous features or a separate generator for quality, this is evidence a multi-codebook discrete representation can close much of the gap. The output-head ablation suggests the dependency modeling is the load-bearing piece, not just “more codes.”
•
If you’re building masked-diffusion image samplers, the PHV result is worth reading closely: the authors show that for visual tokens, absolute-confidence commit thresholds (as in Fast-dLLM) don’t fire because confidences stay low (per-prompt median rose only from 0.046 to 0.127 across 96 steps). Temporal signals (plateau + drift) are a workable substitute. Worth testing on your own masked-generation sampler before adopting a universal threshold.
•
The paper doesn’t release code or weights in the supplied text, so applicability is at the level of design decisions, not drop-in use. The cross-platform run (Ascend NPU as primary, NVIDIA as reference with a smaller budget) suggests the recipe isn’t tied to one hardware stack, though the NVIDIA numbers are generally lower, which the authors attribute to the smaller token budget.
•
The NVIDIA reference run uses fewer training tokens (200B vs 500B+ for several stages), so the Ascend-vs-NVIDIA score gaps conflate hardware and compute. The paper is explicit about this, but it means you can’t read the NVIDIA numbers as the ceiling for the recipe on GPUs.
•
Image editing scores (ImgEdit, GEdit-Bench) trail the best unified baselines, and OneIG-Bench shows notably low sample diversity. The multi-codebook story helps understanding and text-to-image more than it helps editing in these results.
•
The multi-codebook AR head adds sequential decoding per position (K=8 steps), which is real overhead at high resolution. PHV helps, but the architecture inherently costs more per token than a single-code predictor.
•
Video and 3D generation are not supported, only understanding. The “omni-visual” claim applies asymmetrically across modalities.