In chat-templated text-to-image diffusion transformers, the formatting tokens like <|im_end|> (not the prompt words) are where object identity actually lives during denoising, and ranking heads by how much they read the prompt lets you prune 20% of attention FLOPs for a 1.4-point GenEval drop.
You’re shipping a text-to-image feature on top of a modern Diffusion Transformer (DiT) like Qwen-Image or FLUX.2. The user’s prompt gets wrapped in a chat template before hitting a vision-language encoder, and you probably treat those <|im_start|>/<|im_end|> delimiters as inert formatting. This paper says they aren’t inert. Inside the generator, those delimiter tokens are where the model actually stores “which object to draw,” which matters if you want to interpret, edit, or accelerate the model. The dominant baseline mental model comes from work on Attention Sinks in language models: sink positions absorb attention but carry no content. Here the sinks carry content, just not the content you’d expect.
The authors study Qwen-Image, a MMDiT that gets its text conditioning from a Qwen2.5-VL encoder run on a chat-formatted prompt. After discarding a fixed prefix, the retained token sequence splits cleanly into a semantic span (the user’s actual prompt words) and a structural span (trailing chat-template tokens like <|im_end|> and assistant\n). Their framework asks four questions with four tools: which tokens get attention (measure image-to-text attention mass), whether swapping a span changes the image (cross-prompt swaps), which heads causally carry identity (transplant per-head q/k/v from one denoising run into another), and where in depth this happens (mask attention layer-by-layer).
The surprising finding is that the structural tokens are attention sinks and semantic registers. Encoded-side, they carry almost no prompt information: you can average them across 100 prompts and paste that average onto any new prompt, and the object is preserved. But inside the DiT, they causally hold object identity. The path is indirect: prompt semantics flow first from the semantic span into the image latents in the very first block, and then the structural tokens read from the image latents to acquire object content.
# Progressive head transplant, A="apple" run, B="banana" run
heads = rank_by(m_S_attention) # image->semantic attention per head
for n in range(num_heads):
# ascending order: swap heads that IGNORE the prompt first
A_run.heads[heads[n]].qkv = B_run.heads[heads[n]].qkv
if identity_flipped(A_run): # apple becomes banana
break
# ~18% of heads suffice, and they are the low-m_S heads
The pruning rule falls out directly. Rank heads by descending average image-to-semantic attention, silence the top-K over the last 80% of the denoising schedule, and you get a prompt-independent speedup.
The intuitive assumption is that the tokens encoding a concept at the input are the tokens carrying that concept during generation, so “semantic heads” should be the ones that read the prompt most. This paper shows the opposite. The tokens that encode semantics at the encoder output are not the tokens that maintain semantics during denoising; identity is stored in content-free structural delimiters that read it back from the image stream. The clean evidence is the reversed head-swap: transplanting the heads that read the prompt hardest fails to transfer identity, while transplanting the heads that barely read the prompt flips apple into banana after only ~18% of heads.
•
Load-bearing ablation. In the progressive head-swap, swapping heads high-to-low by prompt attention collapses the image toward the model’s unconditional default (a generic human portrait) instead of transferring identity. Swapping in the reverse order flips the object cleanly after ~18% of the 1,440 heads. Reading the prompt and carrying the object are two different jobs done by two different sets of heads.
•
Structural tokens dominate attention. On GenEval, the five chat-template tokens absorb 76% of all image-to-text attention on Qwen-Image; per-token they get 6.4× the mass of prompt tokens. The single token <|im_end|> is the top sink at over half of all layer/head/step sites. This holds across prompt lengths, on Chinese prompts (Qwen-Image-Bench), on long dense prompts (DPG-Bench), on FLUX.2 with a Mistral encoder, on Krea-2-Turbo, on the editing checkpoint, and on a 2-step distilled variant.
•
Structural tokens carry almost no encoded semantics. Replacing the structural span with an average computed over 100 unrelated prompts leaves the generated object essentially unchanged (median DINOv3 CLS similarity 0.96–0.99). The rare failures collapse to the unconditional default, not to some other object.
•
Semantics enter the registers via the image latents. Masking the register tokens’ attention to the prompt span leaves the object intact; masking their attention to the image latents erases it within two blocks. The prompt-to-image injection completes in the first block.
•
Depth structure. The ~270 identity-carrying heads are bimodal: 89 in the first 10 blocks (commit identity), 96 thin across the middle 40 blocks (carry it), 85 in the last 10 (refine it).
•
Pruning result. Silencing 360 of 1,440 heads (top by prompt-reading attention) over the last 80% of steps removes 20% of joint-attention FLOPs at a 1.4-point GenEval drop (76.1 → 74.7). Ranking matters more than count: at K=288 the same budget scores 75.5 under this rule, 69.6 if you instead prune the register heads, and 51.3 if random.
Reach for this when you’re serving a modern DiT (Qwen-Image, FLUX.2, Krea-2-Turbo) at scale and want a prompt-independent attention speedup without retraining. The recipe is: profile image-to-text attention mass per head on a few hundred prompts, rank heads by how much they read the prompt content span, silence the top slice, and skip the pruning during early denoising steps because those commit object identity. Object-correctness metrics degrade slowly, perceptual quality degrades faster, so a smaller budget (they suggest K=216 on Qwen-Image-2512) is a better operating point than the headline K=360.
Code is released at DiT-Interpretability. The paper reports validation on Qwen-Image, Qwen-Image-2512, the 2-step distilled Wuli-art LoRA, Qwen-Image-Edit-2511, FLUX.2, and Krea-2-Turbo, so the diagnostic (“do my chat-template tokens dominate the I2T attention block?”) transfers to any DiT you’re already running. No new datasets are released; evaluations use GenEval, DPG-Bench, Qwen-Image-Bench, and GEdit-Bench.
In modern text-to-image DiTs, the tokens that encode a concept are not the tokens that carry it. If you want to intervene on generation, cache smartly, or prune, look at where attention lands during denoising, not at what the tokens ostensibly mean.
•
The register phenomenon is specific to DiTs conditioned on chat-templated VLM hidden states. Older CLIP/T5-conditioned models (SDXL, SD3, FLUX-Schnell) have no structural span of this kind, so the pruning rule as written doesn’t apply and the mechanistic story may differ.
•
The pruning gain is modest and one-dimensional: 20% of attention FLOPs, not 20% of end-to-end latency, and perceptual quality (LPIPS, HPSv3) degrades noticeably faster than object correctness. Treat it as a diagnostic-driven starting point, not a deployed acceleration recipe.
•
The causal experiments lean heavily on a single-step distilled checkpoint to keep the analysis tractable. The multi-step trajectory is spot-checked but not exhaustively traced, so the “early commit, middle carry, late refine” story is suggestive rather than proven across all schedules.