GeoNeXt turns a pretrained video diffusion model into a joint depth-and-normal estimator by casting geometry as the next frame after the input image, matching or beating discriminative baselines trained on ~100× more data.
You’re building a product feature that needs 3D structure from a single photo: think AR object placement, image relighting, or feeding a controllable image generator with depth and normal maps. The dominant recipe today is to fine-tune Stable Diffusion into a depth predictor, then train a separate Stable Diffusion into a normal predictor, then ship two ~2B-parameter models. Marigold pioneered this and works well but keeps depth and normals in separate checkpoints, missing the fact that these two outputs describe the same underlying surface. Alternatives that unify both tasks (like GeoWizard) bolt cross-attention onto image diffusion and need much more training data to compensate for the architectural surgery.
The key move is picking a different pretrained backbone. Instead of an image diffusion model, GeoNeXt starts from Stable Video Diffusion (SVD), which already knows how to generate a coherent sequence of frames from one input frame. The authors treat the RGB image as frame 1, the depth map as frame 2, and the normal map as frame 3. The video model’s built-in temporal attention then propagates structural information from the image to the geometry frames for free, no architectural changes needed to enable joint prediction.
A few practical details make this work. The RGB image is replicated into the depth and normal slots as conditioning, and its latent is concatenated with the noisy geometry latents at the U-Net input. Crucially, the model also re-denoises the image alongside geometry (an image reconstruction branch), keeping image and geometry aligned along a shared denoising trajectory. Depth is a single channel, so it gets tiled to 3 channels to fit the frozen Stable Diffusion VAE, then averaged back on decode. The CLIP cross-attention branch of SVD is disabled because it forces an image resize that distorts geometry. Fine-tuning touches only the U-Net, on roughly 59K synthetic samples from Hypersim and Virtual KITTI 2.
# training step (schematic)
z_img = vae.encode(rgb)
z_dep = vae.encode(tile3(depth))
z_nrm = vae.encode(normal)
z_seq = stack([z_img, z_dep, z_nrm]) # 3-frame latent
z_t = z_seq + sigma_t * noise # EDM noising
cond = concat(replicate(z_img, 3), z_t) # image as conditioning
pred = unet(cond, sigma_t) # denoise all 3 frames
loss = edm_loss(pred, z_seq) # supervise image + geom
At inference, all three frames start from Gaussian noise, five EDM steps run, and the image latent is thrown away. The authors ensemble 5 seeds.
The prevailing approach to adapting diffusion priors for geometry is to take an image generator and either train it per-task or graft new attention modules onto it for joint prediction. This paper shows the opposite: pick a backbone whose pretraining already contains the structure you need. A video model has learned frame-to-frame coherence, and that is exactly what image-to-geometry consistency requires, so the adaptation collapses to “just predict the next frame.” The load-bearing evidence is that removing the image reconstruction branch, which is what actually exercises the video model’s cross-frame prior, degrades every metric.
•
Image reconstruction is the mechanism. Ablation (a) drops the image-reconstruction frame and reverts to pure image-conditioned generation like Marigold does. Depth AbsRel on NYUv2 goes from 5.3 → 6.5, and normal mean angular error on iBims worsens from 16.4 → 18.5. This isolates the claim that jointly denoising the image alongside geometry, not just conditioning on it, is what buys the accuracy.
•
Joint beats separate. Training two independent GeoNeXt models, one for depth and one for normals, is worse than the unified model on every metric, evidence that depth and normals inform each other through the shared denoising trajectory.
•
Data efficiency vs discriminative SOTA. Depth Anything uses 62.6M training images. GeoNeXt uses 59K and reaches AbsRel 5.6 on ETH3D vs Depth Anything’s 12.7, though DA still wins on NYUv2 and ScanNet.
•
Against the closest unified generative baseline, GeoWizard (208K samples, image-diffusion backbone with cross-attention modifications): GeoNeXt wins with less data, e.g. AbsRel 8.2 vs 14.4 on KITTI.
•
Reconstruction order doesn’t matter. Swapping depth-then-normal for normal-then-depth changes numbers by <0.5, suggesting the model isn’t relying on an autoregressive ordering trick.
•
Inference is cheap. With 5 steps × 5-seed ensemble, one forward pass produces both maps in ~10s on an A5000; GeoWizard’s high-quality config needs ~272s for comparable quality.
Reach for this when you’re building a product that needs both depth and normals from a single image, e.g. an AR pipeline that places virtual objects on real surfaces, a relighting feature, or a ControlNet-style conditioner for image generation. Instead of shipping two ~2B-parameter checkpoints and paying the GPU I/O to swap them, you ship one ~1.5B model that emits both maps in a single denoising loop. The paper also shows the method is backbone-agnostic and can be moved onto DiT-based video models like WAN.
The project page is linked; the paper doesn’t explicitly state a code release or license in the provided text. Training uses only Hypersim and Virtual KITTI 2, both synthetic and freely available. If you already have an SVD checkpoint deployed, the fine-tuning recipe here is intentionally minimal: freeze the VAE, disable the CLIP branch, train the U-Net on RGB-depth-normal triplets with the standard EDM objective.
When adapting a pretrained generative model to a new prediction task, the biggest lever is choosing a backbone whose pretraining already contains the structure you need. A video model’s frame-to-frame coherence is a better prior for consistent image-to-geometry mapping than any cross-attention module you can bolt onto an image model, and it costs far less labeled data to fine-tune.
•
Training data is entirely synthetic (Hypersim indoor, Virtual KITTI outdoor). Depth is affine-invariant, not metric, so downstream systems that need absolute scale still need camera intrinsics or a separate metric-depth model.
•
The unified-model advantage over discriminative baselines like Depth Anything is not uniform. On NYUv2 and ScanNet the large-data discriminative models still lead on depth; GeoNeXt’s edge concentrates on higher-resolution or out-of-distribution sets like ETH3D and DIODE.
•
The comparison to GeoWizard is somewhat apples-to-oranges: GeoWizard uses image diffusion with architectural modifications, while GeoNeXt gets a stronger backbone (video pretraining) essentially for free. Some of the reported gain is attributable to the pretraining corpus, not the next-frame formulation itself, and the paper doesn’t isolate these two factors.