PixWorld runs Diffusion model directly on RGB pixels (no Variational Autoencoder latent step) and unifies 3D scene reconstruction and generation by tagging some input views as clean and others as noisy, decoding both into 3D Gaussian Splatting (3DGS) primitives in one forward pass.
Say you’re shipping a product that turns a couple of phone snapshots of a room into a navigable 3D scene the user can fly through. Today you likely stitch two systems: one reconstruction model (like MVSplat-style feed-forward Gaussian prediction) when you have enough views, and a separate generation model (a video diffusion model conditioned on camera pose) when you only have one image and need to hallucinate the rest. They have different training objectives, different failure modes, and neither is great at the middle case of 2-3 sparse views.
The dominant recent approach for the generation half has been to piggyback on latent video diffusion, for example Gen3C, where an SDXL-style pretrained model denoises a video in a compressed latent space, and then a separate stage lifts frames to 3D. PixWorld argues the latent step is dead weight for 3D. It hurts geometric fidelity and forces the two-stage split.
The core trick is treating reconstruction and generation as the same operation with a different input mask. You give the model N posed views (N between 4 and 8). Each view is labeled either clean (a real observation you want to keep) or noisy (a slot to be generated at this camera pose). If every view is clean, the model reconstructs. If some are pure noise, it generates. Mixed is the interesting case.
The denoiser is a two-stream MMDiT transformer: one stream processes clean tokens, the other processes noisy tokens, and every block does joint attention across both. Camera pose enters through PRoPE, a positional encoding scheme for cameras. Instead of predicting pixels directly, the final layer emits per-pixel depth plus 3D Gaussian attributes. Gaussian centers come from unprojecting each pixel using its predicted depth, so the output is a pixel-aligned 3D Gaussian cloud that can be rasterized to any target view via differentiable rendering. The training loss compares rendered views to ground-truth views in pixel space.
On top of the pixel and LPIPS losses, PixWorld adds a geometry perception loss: it passes both the rendered view and the ground-truth view through a frozen 3D foundation model called \u03c03 and matches their features. This is the piece that supplies 3D structural signal that 2D image losses can’t see.
# One training step (schematic)
views = sample_scene(N=random(4,8))
clean, noisy = partition(views, bias_small_clean=True)
noisy_t = add_noise(noisy, t)
depth, gaussians = two_stream_dit(clean, noisy_t, cameras, text)
rendered = rasterize(gaussians, target_cameras)
loss = mse(rendered, gt) + 0.1*lpips(rendered, gt) \
+ 1.0*depth_loss(depth, gt_depth) \
+ 0.1*feature_match(pi3(rendered), pi3(gt)) # gated at t>0.3
The prevailing recipe in 3D scene diffusion is to reuse a pretrained image or video diffusion model that operates in a Variational Autoencoder latent, then bolt 3D on afterward. PixWorld shows the opposite. Skip the latent, diffuse in pixel space, and let differentiable rendering into a 3D Gaussian representation be the diffusion target. The load-bearing evidence is the geometry perception ablation, where removing the 3D feature-matching loss selectively wrecks pose accuracy and cross-view consistency while leaving per-frame image quality nearly untouched.
The ablation is the tell. Turning off the geometry perception loss drops PSNR by 1.13 dB, SSIM by 0.105, and strict pose accuracy AUC@5 by 0.080 (~12.5% relative). Meanwhile the VBench-style per-frame quality scores barely move. So 2D losses can make individual frames look fine while the underlying 3D quietly falls apart, which is exactly the failure mode a unified model needs to fix.
On top of that, the headline benchmarks land:
•
On single-image generation, PixWorld tops every metric on both RealEstate10K and DL3DV-10K, with PSNR +1.06 dB on RealEstate10K and +0.75 dB on DL3DV-10K vs. the next best. Strict pose accuracy AUC@5 rises from 0.546 to 0.614 and 0.420 to 0.485.
•
On WorldScore, the unified world-generation benchmark, PixWorld gets the best overall average (71.04), and leads on camera controllability (91.08), 3D consistency (91.39), and photometric consistency (93.84).
•
In the pure reconstruction setting (4 or 8 clean views, no generation), it beats specialized reconstruction baselines like DepthSplat on PSNR/LPIPS across the board.
•
Inference is 15 seconds per scene on an A100, roughly matching the distilled FlashWorld (10s) despite PixWorld running undistilled at 100 denoising steps, because it only needs to produce 8 key frames and then rasterizes the rest.
The 1.04B-parameter model is trained from scratch on ~67K scenes plus 10M single images as a 2D appearance prior.
Reach for this design if you’re building a photo-to-3D product where users can give you anywhere from one image to a handful, and you want one model instead of a routing layer. The clean/noisy partition is a genuinely nice interface: same weights, same forward pass, the user’s inputs just determine which slots are conditioned. If you have image editors or novel-view synthesis in your stack today running through a latent video model, the paper is a data point that pixel-space diffusion into a 3D Gaussian target is now competitive on speed as well as quality.
The paper does not link a code release or model weights in the provided text. If you want to reproduce, the training recipe is spelled out: 32 A800 GPUs, ~200K steps at 336x448 resolution, AdamW with linear LR decay from 1e-4 to 1e-5, mixture of RealEstate10K, DL3DV-10K, and BLIP-3o single images. The frozen critic is the released pi3 model.
When your output is 3D, supervise the 3D, not just its 2D shadow. A pretrained image VAE is trained to preserve pixels, not geometry, so latent-space diffusion for 3D leaks structure through a lossy channel that no downstream head can fully recover. Diffusing pixels directly and matching features in a 3D-aware backbone closes that gap.
•
Evaluation is entirely on indoor and building-facade scenes (RealEstate10K, DL3DV-10K, WorldScore’s static split). Outdoor, object-centric, or dynamic scenes are not tested; the paper flags this as a limitation.
•
Resolution is 336x448. The inference-speed comparison is not apples-to-apples because baselines like Gen3C run at higher resolution and produce many more frames per scene.
•
The geometry perception loss depends on a specific frozen 3D foundation model (pi3). If pi3 has systematic biases on a scene distribution, PixWorld inherits them, and no ablation swaps in a different critic.