Representation Guidance (RG) trains a pixel-space image diffusion Transformer to build a compact internal representation and then forces the decoder to actually use it, by shifting the flow-matching target along the velocity change that representation induces.
Most high-quality image generators (Stable Diffusion, FLUX, SD3) don’t actually diffuse pixels. They diffuse in a compressed latent space produced by a separately trained VAE, then decode. That extra autoencoder is a liability: it caps detail, adds a training stage, and ties generation to a frozen representation chosen for reconstruction rather than for generation.
A cleaner alternative is to diffuse directly in pixel space with a plain Transformer. JiT showed this is viable. But pixel diffusion has a known weakness: the network has to learn semantics and paint every pixel from the same stack, and the pixel-prediction loss is easy to satisfy with local, high-frequency operations, so the model never bothers to form strong internal semantics. A common patch is REPA, which aligns an intermediate layer with features from a pretrained encoder like DINOv2. The authors show this helps early but the baseline catches up, and pushing the alignment deeper actually hurts Fréchet Inception Distance (FID). Another patch is to bolt on a specialized pixel decoder (PixelDiT, DeCo, DiP); the authors show these mostly reshuffle which block carries semantics vs. spatial detail, with no reliable quality gain and a parameter/FLOP tax.
So the diagnosis: injecting better semantics into an intermediate layer is not enough if later layers overwrite them, and splitting the architecture is not enough if nothing forces the generator to depend on the clean representation.
RG keeps the plain Transformer and changes two things.
Form the representation. Split the stack into an encoder and a smaller decoder (8 encoder blocks, 4 decoder blocks in the default). Between them, insert a linear bottleneck: project the encoder’s wide token features down to a narrow dimension (default 128), then back up. The decoder sees the encoder only through this skinny channel, used as a per-token conditioning signal. The bottleneck acts like a VAE latent in spirit, but it is computed fresh from the noisy input every step and trained end-to-end for generation. Diffusion itself still happens in pixel space.
Force the decoder to use the representation. This is the core trick. During training, run the decoder twice on the same noisy image: once with the real representation r, once with r zeroed out. The difference between the two predicted velocities, call it delta_v_rep, measures how much the representation is actually changing the decoder’s answer. RG then modifies the training target: instead of asking the model to predict the standard Flow matching velocity v, ask it to predict v + (s-1) * delta_v_rep, where s is a guidance scale (default 1.4) and the offset is treated as a constant (stop-gradient). This is structurally the same move as classifier-free guidance, but baked into the training target rather than applied at sampling, and the “condition” being amplified is the model’s own learned representation.
The effect is twofold. The decoder is pushed to depend more heavily on r (otherwise it can’t hit the shifted target). And because gradients flow through r back into the encoder, the encoder is shaped to put generation-useful information into that bottleneck. A REPA loss on one intermediate encoder layer provides an initial semantic scaffold from DINOv2.
# one RG training step (EMA teacher omitted)
z_t = t*x + (1-t)*eps # noisy image
r = encoder_with_bottleneck(z_t) # compact representation
v_with = velocity(decoder(z_t, cond=r))
v_without = velocity(decoder(z_t, cond=0)) # drop r
delta = stop_grad(v_with - v_without)
v_target = (x - eps) + (s - 1) * delta # shifted FM target
loss = ((v_with - v_target)**2).mean() + lam * repa_loss
RG is only applied when the noise level is moderate (t > 0.3); at very high noise the representation is unreliable. At inference, there is no extra cost: the second forward pass only exists during training.
On class-conditional ImageNet 256x256, RG-XL/16 (725M params) reaches FID 1.55, the best among reported pixel-space diffusion models, beating PixelDiT-XL (1.61), DeCo-XL/16 (1.69), and DiP-XL/16 (1.79). More striking for scaling: RG-L/16 at 459M params hits FID 1.79, below JiT-G/16 at 2B params (1.82). At 512x512, RG-XL/16 gets FID 1.74. A text-to-image variant, RG-T2I-L/16, scores 0.89 on GenEval and 87.1 on DPG-Bench using a 341M FLAN-T5 text encoder, competitive with much larger latent-space systems.
The ablation that most directly supports the mechanism claim: adding REPA to a decoupled B/16 gives FID 4.17; adding the bottleneck brings it to 3.79; adding representation guidance brings it to 3.28. If you remove either the bottleneck or REPA, guidance alone makes things worse (FID 7.85 to 8.99). The authors read this as evidence that guidance only helps when the representation is already compactly formed and semantically scaffolded. RG adds roughly 0.4M params and ~13% training FLOPs, and nothing at inference.
Layer-wise probing shows the intended specialization: in RG, encoder layers 3 to 8 keep high linear-probe accuracy (around 52 to 55%) and clear spatial structure, while decoder layers 9 to 12 shift to high-frequency pixel features. In plain JiT with REPA, probe accuracy peaks at the aligned layer and decays by the output. The authors frame this as: what matters is not peak representation quality but whether the representation survives to the point where the decoder consumes it.
•
If you are training a pixel-space diffusion Transformer and have been reaching for a specialized pixel decoder, the component ablation is the thing to read first. It argues the win comes from a cheap bottleneck plus a modified training target, not from decoder architecture.
•
If you already use REPA, note the paper’s claim that alignment helps early but is overtaken by baseline, and that aligning deeper layers hurts. RG uses REPA as a scaffold on an intermediate encoder layer, not at the output; the ablation shows aligning the final encoder layer blows up FID (8.97).
•
The guidance-as-training-target idea generalizes beyond this architecture. Worth testing: taking any conditional diffusion model that already supports classifier-free guidance dropout and shifting a fraction of that guidance into the training target. The paper only validates this with an internal learned condition, not an external one, so treat the generalization as a hypothesis.
•
Practical knobs the paper settles: bottleneck width 128 (narrower is better at longer training), RG scale s=1.4, apply RG only for t in (0.3, 1], an 8/4 encoder/decoder split, EMA network as the guidance teacher (an online teacher is slightly better in a one-off test). The 20-epoch warm-up before enabling RG is reported as basically optional.
No code or weights link appears in the supplied text.
All quantitative results are image generation on ImageNet and two text-to-image benchmarks; the paper explicitly flags video, 3D, and audio as future work. The reported text-to-image numbers come from a model trained on CC12M and fine-tuned on a 120K mixture that includes DALL-E 3 and ShareGPT-4o-Image data, so the T2I comparison to SD3 and FLUX is not a controlled training-data match. The derivation that RG converges to a CFG-style velocity with effective scale 1/(2-s) assumes the EMA teacher matches the online network at convergence, which the authors note explicitly as an assumption. Finally, the FID gains over pixel-specialized decoders like PixelDiT and DeCo are modest in absolute terms (around 0.06 to 0.15 FID at XL scale); the stronger story is the parameter efficiency and the mechanistic ablation, not a dominant leaderboard sweep.