InfiniSplat v2 turns noisy predicted point clouds into renderable Gaussian scenes by freezing the geometry branch and letting a separate branch learn only the Gaussians’ opacity, scale, color, and shape, cutting perceptual error by roughly a quarter versus the strongest baselines.
Suppose you want to drop a handful of photos into a system and get back a 3D scene you can fly a virtual camera through, at angles the photos never covered. The dominant approach today is 3D Gaussian Splatting: represent the scene as millions of fuzzy, colored ellipsoids and rasterize them. Classical 3DGS fits those ellipsoids per-scene with an optimizer, which is slow.
Feed-forward variants predict the ellipsoids in one network pass. They typically chain two jobs together: a pretrained geometry foundation model (like DA3 or VGGT-Ω) estimates per-pixel depth and camera poses, which get back-projected into a 3D point cloud; a second head then predicts each point’s Gaussian attributes. The predicted point cloud is roughly right but littered with floaters and view-inconsistent clusters. When prior work (DA3-GS and others) jointly trains geometry and attributes end-to-end against rendering loss, outputs still show blur, ghosting, and missing detail at wide-angle novel views. The authors argue the joint training itself is part of the problem.
The core intuition: if you let the network nudge point positions to make renders look better, it will “fix” bad attributes by moving geometry around, which corrupts the geometry. So lock the geometry down and force the attribute head to do the work honestly.
Concretely, three changes on top of a DA3-style pipeline:
•
Two independent branches from the same pretrained weights. The geometry branch (depth + camera heads) is frozen. A parallel attribute branch, with only its last few transformer blocks fine-tuned, feeds a DPT decoder decoder that outputs per-pixel opacity, scale, rotation, and color. Point centers are fixed at the back-projected positions. No learned positional offsets.
•
Scale-aligned ground-truth cameras during training. Predicted poses have small errors that, under wide-angle supervision, mislead the attribute head. The authors instead back-project with GT cameras, after rescaling GT translations to match the predicted depth scale via an Umeyama alignment fit on input-camera centers. Inference uses whatever cameras are available (supplied or predicted).
•
Large-angle target views. Supervising only on views near the input cameras leaves Gaussian shape under-constrained. They bias target sampling toward views whose optical axes differ most from any input view.
A pseudo-code sketch of the feed-forward pass:
depth, cam_pred = geo_branch_frozen(images, pose_hint)
feat = attr_branch(images, pose_hint) # last K blocks trainable
xi = dpt_decoder(feat) # opacity, scale, rot, color
cam = scale_align(cam_gt_or_pred, depth) # Umeyama scale only
centers = backproject(depth, cam) # positions FIXED
gaussians = voxel_merge(centers, xi)
gaussians = prune(gaussians, opacity < 0.01)
image_hat = render(gaussians, target_camera)
A second stage adds a learned refiner: it renders the scene back into the input views, feeds the rendering gradients plus DPT decoder features into a recurrent KNN transformer, and predicts small residual updates to attributes (not centers) over three steps.
The same trained model is evaluated across six datasets (DL3DV, ETH3D, NRGBD, Mip-NeRF 360, RealEstate10K, DTU) with 4, 8, or 16 input views, in both pose-conditioned and pose-free settings. Primary metrics are perceptual: LPIPS and DISTS.
•
Pose-conditioned feed-forward output achieves the lowest mean LPIPS and DISTS among evaluated baselines, with LPIPS down 25.8% vs. the strongest baseline (ReSplat) and DISTS down 32.9% vs. the strongest baseline (WorldMirror 2.0). Pose-free gains are 19.5% and 30.7% against WorldMirror 2.0.
•
Ablations on ETH3D and DTU support each design choice. Letting the geometry backbone fine-tune (w/o G) or adding learned positional offsets (w/ O) worsens both metrics; freezing geometry cuts LPIPS by 16.6% and 23.2% on the two datasets. Swapping predicted for scale-aligned GT cameras at training time cuts DTU LPIPS by 36.1%. Large-angle target supervision beats local-interpolation supervision on both metrics.
•
An unexpected side effect of opacity. Trained only against image reconstruction, the attribute branch assigns low opacity to points that are geometrically inconsistent across views. Using GT depth as an independent label on held-out Hypersim scenes, pruning at opacity 0.01 removes 51.8% of inconsistent points while discarding only 7.5% of all points and keeping 93.3% of surface-consistent points. Opacity acts as a learned reliability filter, though the paper presents this as an observation rather than a trained objective.
•
If you’re building a feed-forward 3DGS pipeline on top of a pretrained geometry model, the single most actionable finding is: don’t let gradients from rendering loss touch the geometry branch or the point positions. The ablations show that letting the attribute head “cheat” via position offsets measurably degrades both renders and depth. Worth trying as a cheap architectural fix even without the other changes.
•
The GT-camera-with-scale-alignment trick applies specifically at training time when you have ground-truth poses. At inference the method falls back to predicted or supplied cameras, so this is a training recipe, not a deployment requirement.
•
The opacity-as-reliability-signal observation is worth testing as a free pruning heuristic in your own feed-forward pipelines: threshold at low opacity, see what gets removed. The paper demonstrates this on Hypersim only, so don’t assume the exact 51.8% / 7.5% split transfers.
•
For evaluation choices, note that the headline gains are on perceptual metrics (LPIPS, DISTS). If your downstream task cares more about PSNR or geometric accuracy, read the appendix tables before committing.
•
Code and project page are linked from the paper: project page.
•
Fixed centers mean the method inherits whatever errors the frozen DA3 geometry produces. The authors explicitly flag this: opacity can hide bad points, but it can’t move them. Scenes with weak geometric priors will cap quality.
•
Pixel-aligned prediction creates redundant Gaussians that voxel merging only partly cleans up, and unobserved regions stay empty.
•
The 25.8% / 32.9% improvements are mean reductions across six datasets vs. the single strongest baseline per metric, in the pose-conditioned setting. Per-dataset and per-view-count rankings vary; the appendix is where you’d look to see if your scene type is in the favorable regime.
•
Comparisons are to the specific set of baselines the authors ran. The paper doesn’t claim improvement over methods outside that set.