Marigold V2 turns an open image-editing Diffusion Transformer (DiT) into a sharp monocular depth predictor by fine-tuning with a Sinkhorn-based block-matching loss plus feature alignment against depth-derived semantic features, recovering fur, foliage, and hair-thin edges that prior diffusion depth models blur away.
Monocular depth estimation, predicting per-pixel distance from a single photo, feeds bokeh, relighting, novel-view synthesis, AR compositing, and single-image 3D reconstruction. The problem is fundamentally ambiguous: infinitely many 3D scenes explain one image, so models must learn strong priors.
Two camps dominate. Discriminative models like Depth Anything and MoGe regress depth directly and scale well, but ground-truth depth data is scarce (millions, not billions of examples) and mishandles transparent or reflective surfaces. Generative approaches repurpose diffusion image models, inheriting the world knowledge from billion-image pretraining. The original Marigold line showed this works with tiny fine-tuning datasets (tens of thousands of images), but the resulting depth maps are oversmoothed: thin structures dissolve, boundaries blur, and points near depth discontinuities scatter into “flying pixels” when projected to 3D.
Repurposing has also gotten expensive. Modern generative backbones are diffusion transformers, and even parameter-efficient fine-tuning of them costs many GPU-days on multi-GPU rigs. That prices out individual researchers and small labs.
The backbone is Qwen-Image-Edit, a pretrained image-editing diffusion transformer. Marigold V2 freezes its weights at 4-bit precision and trains small QLoRA adapters, so the whole fine-tune fits on one 32GB GPU.
The model is trained to do depth prediction in a single forward pass, not iterative denoising. Following Rectified Flow with a fixed timestep, the network learns to map the RGB latent directly to a depth latent. The depth target is log-scaled and clipped to the 2nd/98th percentiles so global scale and offset don’t matter, then encoded as a grayscale image the VAE can process.
Two custom losses do the real work:
iREPA-depth: an internal feature-alignment trick. During training, the ground-truth depth map is fed through a frozen DINOv3 encoder, and the DiT’s intermediate features are pushed to match those semantic features. Prior work aligned to features from the RGB input; the authors show aligning to features from the depth map itself gives better geometry and faster convergence. This runs only at training time, so inference cost is unchanged.
SinkLoss: a block-wise matching loss. Ground-truth depth for thin objects is genuinely noisy: a rendered hair strand might be assigned foreground or background depth almost randomly. Strict per-pixel L1 punishes the model for not reproducing that noise exactly. Instead, SinkLoss tiles the image into 5×5 blocks and, within each block, uses Sinkhorn-Knopp algorithm to find a soft one-to-one matching between the 25 predicted and 25 ground-truth depths. The loss is the transport cost of that matching. The model must produce the right set of depth values per block, not pin each one to a specific pixel.
Training runs in two stages:
# Stage 1: ~5 days, adapters only
for step in range(160_000):
z_I = vae_encode(rgb)
z_d_pred = z_I - dit(z_I, t=0.5) # rectified-flow, single step
d_pred = vae_decode(z_d_pred)
loss = (latent_mse + L1_pixel + L1_gradient
+ 0.2 * iREPA_depth(dit_features, dinov3(gt_depth)))
update(qlora_adapters)
# Stage 2: ~1 day, adds SinkLoss and unfreezes VAE decoder
for step in range(30_000):
loss = iREPA_depth + SinkLoss_5x5(d_pred, gt_depth)
update(qlora_adapters, vae_decoder)
On zero-shot depth benchmarks (NYUv2, KITTI, ETH3D, ScanNet, DIODE), Marigold V2 beats prior diffusion-based depth methods including Pixel-Perfect Depth (PPD) and Lotus-2 across all five, using the same 74K-image training set as the original Marigold. The abstract highlights 16–26% AbsRel improvement over the previous best on KITTI and ETH3D. On ETH3D, AbsRel drops from 3.8 to 2.8.
For edge quality on HyperSim, measured with the Soft Edge Error (SEE) metric that tolerates small spatial offsets near boundaries, Marigold V2 scores best across all three patch sizes tested, beating both Pixel-Perfect Depth and InfiniDepth.
The ablation isolates the two contributions cleanly. iREPA-depth beats iREPA on RGB features and beats a straight LPIPS perceptual loss, especially in cluttered regions like foliage. SinkLoss, added in Stage 2, sharply improves edge metrics (SEE) without hurting AbsRel or δ₁. The authors also swap the backbone to Stable Diffusion 1.5 and FLUX.2-klein-4B and show SinkLoss transfers: edge metrics improve on both without degrading average AbsRel. The evidence that gains come from the recipe rather than from Qwen specifically is reasonably direct.
The recipe extends to other dense tasks with modest retraining: surface normals (best or near-best on four datasets), albedo estimation (best PSNR and LPIPS on HyperSim), depth completion via a test-time LoRA fitted to sparse measurements, and see-through depth behind glass, where fine-tuning on the LayeredDepth-Syn dataset drops AbsRel from 13.7 to 8.2.
Inference is not free. On a 32GB GPU, Marigold V2 handles 2048×2048 in 9.6 seconds while Pixel-Perfect Depth, Lotus-2, and FE2E run out of memory. At 1024×1024 it takes 1.9 seconds. That’s faster than several diffusion competitors but far slower than discriminative models like MoGe.
If you need high-resolution depth maps where fine detail matters (relighting portraits, matting hair, 3D-lifting foliage, computational refocus), this is the model to try. The tradeoff over a discriminative predictor is roughly a second of latency for meaningfully sharper edges and fewer flying pixels. The project page is linked in the abstract.
If you want to adapt a diffusion transformer to a new dense prediction task on a modest budget, the recipe (QLoRA on a quantized DiT + iREPA against features of the target modality + SinkLoss in a second stage) is worth copying. The paper demonstrates it working on depth, normals, albedo, depth completion, and see-through depth from a single 32GB GPU in under a week per task. Two caveats worth testing before committing: the backbone-transfer ablation only measured SinkLoss’s effect, not the full recipe end-to-end on non-Qwen backbones, and every task in the paper is a dense image-to-image regression.
If you’re specifically working on noisy ground-truth (rendered thin structures, LiDAR sweeps with mixed foreground/background hits), SinkLoss is a general-purpose replacement for strict per-pixel losses. The block matching says “produce these values somewhere in this neighborhood,” which is often what you actually want. Worth testing on any task where pixel-perfect alignment is neither achievable nor desirable.
Real-time use is not viable with this backbone; the authors are explicit about that.
The headline 16–26% AbsRel improvement is on two of the five benchmarks (KITTI and ETH3D); gains on NYUv2, ScanNet, and DIODE are smaller. The comparison table excludes models trained on more than 5M images (like MoGe-2 and π³), which sometimes score better on individual metrics; the ranking is against methods trained on comparable data.
Reflections, motion blur, and defocus regions remain ambiguous, and the authors flag these as open problems. Depth output is affine-invariant (relative), not metric, unless you add the test-time LoRA fitting procedure with sparse depth anchors.
The SinkLoss backbone-transfer experiment shows edge metrics improve on Stable Diffusion 1.5 and FLUX.2 klein, but AbsRel is essentially unchanged or slightly worse on those backbones. The full “days on one GPU beats prior SOTA” claim is demonstrated only with the Qwen backbone.