WorldSculpt generates a cluttered 3D scene as a bag of individual object meshes by running a single-object generative prior once per object, conditioned on multi-view crops warped into a per-object canonical cube. The prior is trained only on isolated objects yet handles scenes with up to 701 objects and heavy occlusion without any scene-level training.
You’ve shipped a system that reconstructs a room from a phone video for a game engine or a robotics sim. Today you get back one fused mesh, so a designer can’t grab the chair and move it, and a physics engine can’t treat the vase as a separate rigid body. Either you re-run a single-object reconstructor per crop (which hallucinates because each crop shows only a fraction of the object), or you use scene-level generators like Marble that produce one monolithic representation. Compositional methods like MIDI exist but have mostly been shown on tabletops with a handful of items.
WorldSculpt targets the actual messy case: hundreds of mutually occluding objects, each of which needs to come out as its own editable mesh placed in the shared world frame.
Start with the outputs of an off-the-shelf perception stack: posed RGB frames, a per-object instance mask across views (from something like SAM 3), and a coarse 3D bounding box per object. WorldSculpt then handles each object independently.
For each object, pick one view as the anchor, the one where the object is most visible. Build a normalized cube around the object’s coarse box, oriented so the anchor camera looks at it from the canonical front. Grow the cube if any view’s mask spills outside it. Now every view can be cropped, masked, and projected into this shared cube using an adjusted intrinsic matrix that accounts for the crop.
Each cropped view is encoded with DINOv3 into a 2D feature map. For every voxel in the canonical cube, sample the feature at its projected pixel in each view. That gives you, per voxel, a stack of features from all views that saw that point. Fuse the stack with an IBRNet-style aggregator: compute the cross-view mean and variance, then let a small MLP predict a residual correction and a softmax weight per view. The final layer is zero-initialized, so training starts as plain averaging and learns view-dependent weighting on top.
The fused 3D feature grid is injected into the two geometry stages of Pixal3D, a single-image 3D generator, through zero-initialized projection layers added inside each transformer block. The frozen base model is adapted with LoRA. The anchor view still feeds the original single-image cross-attention path, so the pretrained conditioning convention is preserved. The output mesh sits in the canonical cube and gets pushed back into the world with the cube’s canonical-to-world similarity transform.
for obj in scene.objects:
anchor = pick_most_visible_view(obj)
cube = build_canonical_cube(obj.box, anchor)
grow_cube_until_masks_fit(cube, obj.masks)
crops = [mask_and_crop(v, cube) for v in obj.views]
feats = [dinov3(c) for c in crops]
grid = ibr_aggregate_per_voxel(feats, project_fn=cube.project)
mesh_canonical = pixal3d_geometry(grid, anchor_tokens=feats[anchor])
scene_meshes.append(cube.to_world(mesh_canonical))
Training uses only isolated objects rendered from random view counts (1–20), plus a curriculum that adds fake occluders, pose noise, mask errors, and downsampling to the conditioning views but never to the supervision. No scene-level training data is used at any point.
The prevailing move for compositional scene generation is to build scene-aware models: jointly generate multiple objects, or estimate scene layout and object shape together. This paper shows the opposite. A single-object generative prior, if you feed it multi-view features lifted into a per-object canonical cube, generalizes to scenes with hundreds of occluded objects without ever seeing a scene during training. The scene-level problem dissolves into N independent object-level problems once each object has its own coordinate frame. The evidence isn’t the headline scene numbers, it’s that the same weights trained on isolated objects work on 700-object interiors.
The load-bearing result is the single-object stress test on Toys4k with synthetic per-view occlusion. With one clean view, WorldSculpt roughly matches the single-view Pixal3D baseline (CD-ℓ2 of 2.59 vs 2.17), showing the multi-view adaptation didn’t damage the prior. As occlusion climbs to 75% per view, single-view baselines collapse (CD-ℓ2 above 40), while WorldSculpt with 16 views stays at CD-ℓ2 of 2.25, near its clean-input number. That’s the mechanism: extra views compensate for occlusion linearly, exactly as the design predicts.
•
On real tabletops (HouseCat6D) and synthetic clutter (Toys4k-Scene), WorldSculpt beats compositional baselines like SceneGen, SAM3D, and ShapeR, e.g. CD-ℓ2 of 0.28 vs 1.26 for the strongest baseline on HouseCat6D.
•
On the new UE-MeshyScene benchmark, WorldSculpt cuts CD-ℓ2 from ShapeR’s 7.42 to 2.48, and the per-instance median drops from 2.47 to 0.25, so the win isn’t a few easy objects dragging the mean.
•
The learned IBR aggregator’s edge over plain mean fusion grows with scene complexity: negligible on HouseCat6D, largest on UE-MeshyScene. Learned weighting matters most when views disagree.
•
As a demo, they run the whole pipeline on a Marble-generated Gaussian-splat world by rendering posed views from it, and get back a compositional mesh scene without retraining.
Reach for this when you’re building a pipeline that ingests casual video or a generated 3D world and needs to hand a game engine, physics sim, or AR editor a bag of separately manipulable object meshes. The concrete swap: instead of running a single-image 3D generator on each detected object crop (which guesses at occluded geometry from one view), give it all the views that saw the object, warped into a per-object canonical cube, and let a small IBR head fuse them. The upstream stack (poses, masks, coarse boxes) is standard off-the-shelf work.
The paper introduces UE-MeshyScene, a synthetic benchmark of six Unreal Engine environments (93–701 objects each, 2,299 objects and 5,964 views total) with per-object ground-truth meshes, camera poses, instance masks, 3D boxes, and depth. That’s directly useful for anyone evaluating compositional scene reconstruction under real clutter. The paper does not mention a code or model release; treat the method as a recipe to replicate rather than an artifact to download.
Give a single-object generator its own coordinate system per object and it will scale to scenes it was never trained on. The trick isn’t a bigger model or scene-level data, it’s an anchor-aligned canonical cube that turns “reconstruct this cluttered room” into N copies of “reconstruct this one object from partial views,” a problem the prior already knows.
•
The method inherits errors from upstream perception. Badly wrong masks or grossly misplaced 3D boxes will corrupt the canonical cube, and the augmentation curriculum only buys robustness to moderate noise.
•
Geometry only. There’s no texture or material output yet, so you get shape but not appearance; a game-ready asset still needs a separate texturing pass.
•
Static scenes only. Every view of an object is assumed to be the same rigid geometry in the same pose, so anything moving or deforming across the capture breaks the formulation.