Latent-to-4D skips RGB decoding entirely by feeding a video diffusion model’s final denoised Variational Autoencoder latent straight into a pretrained 4D reconstructor, so one checkpoint trained on ~1K clips serves three different video generators that share the same VAE.
Say you’re shipping a product that turns a text prompt or a single image into a navigable 3D scene with motion: think a game asset preview, an AR try-on, or a robotics sim seed. The dominant recipe today is generate-then-reconstruct: run a text-to-video model, decode to RGB frames, then run a separate 4D reconstructor on those frames to recover per-frame cameras and dynamic point clouds. Two problems. First, the reconstructor was trained on real videos, so generator artifacts and out-of-distribution content leak into geometry as holes and fragmented surfaces. Second, if you swap the video generator (say you upgrade from Wan2.1-1.3B to Wan2.1-14B), the alternative fix, an integrated feed-forward model like 4DNeX, usually needs geometry-supervised retraining tied to that specific generator. You want modularity without the RGB round-trip.
The key observation: video diffusion models that share the same VAE also share a latent space. The last denoised latent, right before VAE decoding to RGB, already carries all the appearance and motion the generator produced. So instead of decoding to pixels and re-encoding into a reconstructor, the paper wires the latent directly into the reconstructor’s token grid.
The glue is a small network the authors call L4AR (Latent-to-4D Alignment and Refinement). The video latent and the 4D reconstructor’s expected tokens disagree on temporal resolution, spatial resolution, and channel dimension. L4AR fixes this in three steps: trilinear resample to the right grid shape, a learned 3D convolution to project channels and mix local neighborhoods, then a stack of alternating frame-wise and global self-attention blocks to add long-range spatial and temporal context. The 4D decoder, initialized from a pretrained reconstructor called 4RC, predicts per-frame cameras plus a dense world-space point map per pixel.
Crucially, training only touches the alignment module, low-rank (LoRA) updates to the refinement stack, and the prediction heads. The video generators, the VAE, and the reconstructor’s base weights all stay frozen. Training uses observed real-video latents from ~1K clips with 4D annotations. At inference, you swap in the generated latent from any compatible Diffusion Transformer (DiT) and the same pathway runs unchanged.
# Training: use encoded real video
z = VAE.encode(video).mean # frozen
# Inference: use generated latent from any compatible DiT
# z = DiT(text_or_image_condition, noise) # frozen
Q0 = conv3d(trilinear_resample(z)) # align grid + channels
Q = refine(Q0, cam_tokens, time_tokens) # frame + global attn, LoRA
cameras, point_maps = decoder_4d(Q) # frozen backbone, trained heads
The usual assumption is that the natural handoff between a video generator and a geometry model is RGB, because RGB is the universal interchange format and the reconstructor was trained on it. This paper argues the opposite. The final denoised VAE latent is a better interface than the pixels it would decode to, because it sits upstream of the artifacts the RGB decode introduces and it’s already shared across every generator built on the same VAE. The evidence that matters is not the headline benchmark lift; it’s that one checkpoint transfers unchanged across two text-to-video models and one image-to-video model without any per-generator tuning.
The load-bearing result is the transfer: a single L4AR checkpoint runs across Wan2.1-T2V-14B, Wan2.1-T2V-1.3B, and Wan2.2-I2V-A14B, and beats each generator’s own RGB-cascade baseline on the same latent. Concretely, on the controlled same-latent comparison against Wan+4RC:
•
On Text4D-200, DINO F1 improves by 2.88 to 3.45 points across the two text-to-video backbones.
•
On I4D-200 (image-conditioned), DINO-F1 improves by 5.81 points over the matched Wan2.2+4RC cascade, and beats the native image-to-4D model 4DNeX by a wide margin.
•
Human raters, comparing pairwise across 50 cases per benchmark with multi-view playback, preferred Latent-to-4D on geometry, completeness, temporal stability, and overall quality; every preference interval exceeds 50%.
•
A diagnostic that injects a controlled generator-derived residual perturbation into the latent shows Latent-to-4D’s point-map drift stays near 0.005 cm while the RGB baseline drifts to ~0.3 to 0.4 cm, evidence that the RGB boundary is where generator artifacts turn into geometry errors.
The authors are careful to note that off-axis DINO scores measure visible geometric coherence, not metric geometry, and that one baseline (CogVideoX+4RC) still wins on RGB-reference CLIP-I. This isn’t a clean sweep on every metric.
Reach for this when you’re building a pipeline that turns generative video into dynamic 3D, and you expect to swap or upgrade the video model over time. Instead of retraining a geometry stage per generator, you train one alignment-and-refinement bridge against your VAE and reuse it as new Diffusion Transformer (DiT) backbones come online in that VAE family. Upstream controls, camera trajectory, motion, pose, image conditioning, pass through unchanged, because they’ve already shaped the latent by the time L4AR sees it.
The paper doesn’t specify a public code release or model weights. Training used 1,143 clips from six reconstruction datasets; the two evaluation suites, Text4D-200 and I4D-200, are described as locked 200-case benchmarks but the paper describes construction only in an appendix. Practically, the constraint is stark: your generator has to share the exact VAE checkpoint, latent normalization, tensor layout, and compression convention as the one L4AR was trained on. Cross-VAE-family transfer is not demonstrated.
When two models in your pipeline share a latent space, that latent is the interface, not the pixels either side would decode to. Pixels are lossy re-encodings of information both models already have in a common representation, and the decode-then-re-encode round trip is where distribution mismatch and artifacts creep in.
•
The whole story depends on a shared VAE. If your video generators use different VAEs (say Wan vs CogVideoX vs HunyuanVideo), you need a separate L4AR per family. The paper only demonstrates transfer within Wan.
•
Evaluation leans on projection-based DINO similarity and human judgment, which measure geometric plausibility and completeness, not metric accuracy. The paper explicitly declines to claim metric geometry correctness for generated scenes.
•
The diagnostic showing robustness to generator artifacts uses one controlled residual construction, not arbitrary DiT errors, so real-world robustness across unseen generator failure modes is not established.