GAE replaces the appearance-focused latent inside a video generator with a compact geometry-native latent distilled from a 3D perception model, so one flow model outputs RGB and consistent depth/cameras from the same tokens, roughly halving camera-trajectory error on RealEstate10K.
Suppose you want a video model that acts as a world model: give it a photo and a camera path, and it should return frames that look real AND correspond to one persistent 3D scene, so a downstream planner can trust the geometry. Today’s generators fail the second half. Frames look sharp, but if you run a 3D reconstructor over them, the recovered depth wobbles and the camera drifts off the requested trajectory.
The authors blame the latent space, not just the generator. Standard latent diffusion (a la Stable Diffusion) uses a pixel Variational Autoencoder that packs appearance well but throws away explicit geometry. Semantic latents like RAEv2 organize meaning but still don’t encode depth or cross-view structure. The common workaround bolts geometry on afterward as an extra head, a conditioning signal, or a fine-tuning reward. GAE’s thesis: perception models already produce features that encode geometry cleanly, so generation should evolve inside that same space rather than translating into it.
The starting point is DA3, a frozen geometry foundation model that reads an image and emits a four-level feature hierarchy that its own head decodes into depth, camera rays, and point maps. You cannot just feed those features to a diffusion model. Each level has 3,072 channels but only about 11 meaningfully active directions, with a covariance Condition number up to 10^16. That is a nearly-degenerate space, terrible for a flow model to learn transport over.
Stage 1: build a well-behaved geometry latent. GAE inserts a small convolutional autoencoder between the frozen DA3 encoder and the frozen DA3 geometry head. It fuses all four feature levels, squeezes them into a single grid latent with 64 or 128 channels, and is trained to reconstruct the full hierarchy so the original, unchanged geometry head still reads correct depth and cameras out of it. A separately learned RGB head decodes appearance from the same latent. Because the geometry head stays frozen, the codec cannot cheat by making the head compensate for lost information.
Reconstruction alone gives a latent that decodes but is still awkward to generate. So GAE adds two shaping losses on the bottleneck: a token-wise alignment to a frozen C-RADIO teacher, which smooths transport and organizes semantics, plus a relational term that matches pairwise token similarities to those from a frozen DINOv2 teacher. Token alignment on its own destroys spatial relationships between tokens; the relational term restores them. This shaping is called REPA-style but applied to the codec latent itself, not to intermediate denoiser features.
Stage 2: a standard conditional flow. Freeze the codec, standardize the latents, and train a Diffusion Transformer (DiT)-style Flow matching transformer that jointly denoises all target-view latents at once. Conditioning enters through three channels: text via cross-attention, metric Metric Plücker rays modulating self-attention for camera control, and clean reference-image latents prepended as extra tokens at timestep zero. Reference tokens are attended to but never denoised, which sidesteps a subtle train/test mismatch: DA3 is set-conditioned, so features of a reference encoded alongside target views differ from features encoded alone. Dropping each condition independently during training lets one checkpoint do text-to-image, camera-controlled video, and reference-conditioned novel views.
# Inference sketch
z_ref = standardize(encoder_phi(da3(ref_images))) # clean, t=0
z = sample_gaussian(shape=(V, C, H, W)) # noisy target views
for t in euler_schedule(1.0, 0.0, steps=50):
v = flow(z, t, cond=(z_ref, plucker_rays, text))
z = z + dt * v
z = destandardize(z)
rgb = rgb_head(z)
geom = frozen_dpt_head(decoder_phi(z)) # depth, rays, point maps
The headline comparison is controlled: same flow architecture, same training data (RealEstate10K and DL3DV), same schedule, same 9-view / 50-step / CFG=2 sampling. Only the latent encoder/decoder swaps. Competitors include SD-VAE, the WAN2.1 video VAE, semantic RAEv2, and raw single-level DA3 features.
•
Latent diagnostics. GAE compresses DA3’s 3,072 channels to 64 or 128 while raising effective rank from ~11 to the 30s or 50s and cutting the condition number from 10^8 down to the hundreds. Semantic neighborhood and spatial-structure scores also improve over every baseline that has a native geometry readout.
•
RGB generation. Against the strongest non-GAE controlled latent, GAE-64 cuts FVD by 12.7% on RealEstate10K and 23.1% on DL3DV, and also leads PSNR, SSIM, and LPIPS on both.
•
3D consistency, independently measured. When an external reconstructor (VGGT-Ω) recovers cameras from generated frames, GAE-64 reduces trajectory error ATE by 52.8% on RealEstate10K and 23.3% on DL3DV versus the best controlled competitor. The MEt3R cross-view feature-disagreement metric also improves. Because VGGT does not share GAE’s backbone, this is not a self-evaluation artifact.
•
Ablations. Removing the DINOv2-based relational loss collapses spatial structure metrics even while token-level scores stay high. Putting the reference latent into the noisy ODE state instead of as clean context inflates the reference-vs-target point-cloud gap by roughly 3x. Using per-scene normalized cameras instead of metric Metric Plücker rays wrecks scale-sensitive pose recovery. Text-to-image co-training substantially helps FVD.
•
Reconstruction sanity check. The 128-channel codec matches or beats raw single-level DA3 features on PSNR, LPIPS, and depth/pose reconstruction while using 24x fewer channels, so the compactness does not cost fidelity.
The authors’ claim is scoped: they show the latent representation explains a large chunk of the gap between generators that hallucinate geometry and generators whose frames survive an independent 3D reconstruction. They are not claiming a new state of the art against every camera-controlled video system in the wild.
•
If you are training a camera-controlled video or novel-view generator and you care about downstream 3D use (planning, simulation, reconstruction), the practical takeaway is that the choice of tokenizer matters more than most add-on tricks. Swapping a pixel VAE for a geometry-derived latent, holding the generator fixed, moved every 3D-consistency metric here. Worth testing on your own data before investing in geometry-aware post-training or reward shaping.
•
The reference-conditioning pattern (clean tokens at t=0, attended but never denoised, prepended rather than injected into ODE state) is portable to any set-conditioned encoder where the training-time and inference-time encoder contexts differ. Cheap to try in existing pipelines.
•
The two-teacher recipe (C-RADIO for token-wise semantics, DINOv2 for pairwise structure) applies whenever you are shaping a latent for flow modeling and find that token-alignment alone hurts spatial coherence. The paper does not claim this generalizes beyond geometry latents; treat it as a hypothesis for other representation autoencoders.
•
If you only want to evaluate a video generator’s 3D consistency, the paper’s protocol is directly reusable: reconstruct cameras from generated frames with an independent model like VGGT, Sim(3)-align, and report ATE/RPE plus MEt3R. This is what exposes drift that FID/FVD hide.
•
The paper does not mention releasing code or weights in the supplied text. A project page exists; whether artifacts ship there is not stated.
•
All controlled comparisons are 9 views at 252x252 on two indoor/scene datasets. The 81-view, higher-resolution model is shown only qualitatively, and its training mixture and settings differ from the controlled runs. Do not read the controlled deltas as guarantees for long rollouts or out-of-domain scenes.
•
GAE is tied to a specific geometry foundation model (DA3). Whether the recipe transfers to other backbones is untested here; the ablation only swaps DA3-LARGE for DA3-GIANT.
•
The DA3-GIANT cross-check for camera metrics shares a backbone with GAE’s encoder, so the authors correctly rely on VGGT as the independent evaluator. When reading Table 6, the VGGT columns are the ones to trust for GAE-vs-baseline comparisons.
•
‘Geometry’ here means depth, camera rays, and point maps derived from static-scene perception features. Dynamic scenes, non-rigid motion, and physical plausibility beyond geometry are out of scope.
•
Gen3R and GLD are shown as external references, not head-to-head competitors in the controlled ranking, because they are complete pipelines rather than drop-in latents. Comparisons against them should be read as system-level context, not apples-to-apples.