Puffin-World is a single multimodal model that jointly perceives, generates, and reconstructs 3D scenes by conditioning every pixel on a 9-channel camera map that binds gravity-anchored absolute orientation to relative ray geometry, so generated trajectories stay upright instead of drifting.
Suppose you’re building a product that generates room walkthroughs or exterior fly-arounds from a single photo. Today you likely stitch three separate systems: a calibration model to guess the camera, a video or Multi-view diffusion model to hallucinate new views, and a Multi-view stereo (MVS) pipeline to recover geometry. The camera model has no idea which way is up in world coordinates, so as the virtual camera rotates the horizon tilts, walls lean, and the scene slowly drifts off its axis. Prior camera-controlled generators like CameraCtrl use Plücker embeddings that describe rays between views but carry no absolute reference to gravity. Puffin-World’s pitch is that one model, given a camera representation that knows where “down” is, can do all three jobs and stay physically consistent over long trajectories.
The core object is the Omni-Camera map: for every pixel, a 9-dimensional vector concatenating (a) a gravity-based Perspective Field (the up-vector and latitude angle at that pixel) and (b) a standard ray map (ray origin plus per-pixel direction). The first three channels tie the image to the real-world vertical; the last six describe camera motion relative to other views. This dense map is added into the diffusion latent before patch embedding, and re-injected at a handful of deeper transformer blocks, so every token stays aware of its camera geometry.
A single backbone handles three modes selected by a 4-channel role mask per view: understanding (predict camera parameters as text via an LLM), generation (denoise target views), and reconstruction (denoise depth as a bounded RGB image using an invertible Hilbert-curve color mapping, so the same VAE handles both appearance and geometry).
The key trick for long trajectories is physics propagation. The model perceives the absolute gravity direction of the reference view, then rotates that vector through the known relative camera rotations to give every future view its own gravity-anchored absolute map. Pseudocode:
g0 = perceive_gravity(reference_view) # LLM predicts roll/pitch/FoV
for t in range(1, T):
g_t = R_relative[t] @ g0 # rotate reference gravity
abs_field_t = render_perspective_field(g_t, intrinsics[t])
omni_cam[t] = concat(abs_field_t, ray_map[t])
latents = diffusion_denoise(target_views, cond=omni_cam, role_mask=m)
rgb, depth = decode_vae(latents) # depth via inverse Hilbert map
For sequences longer than the training window, chunks are chained in latent space (the last denoised latent becomes the next reference) to avoid VAE re-encoding drift.
The common practice in controllable video and multi-view generation is to describe the camera only by its motion relative to other frames, using ray-based embeddings like Plücker coordinates. This paper argues the opposite. A generative world model needs an absolute physical anchor, and gravity is the cheapest one to get: perceive it once from the first view, then propagate it through the same relative rotations the model already uses. The load-bearing evidence is the Physics propagation ablation on rotation-heavy trajectories, not the headline calibration leaderboard.
•
The ablation that matters: on rotation trajectories (roll, pitch, yaw) from Puffin-Traj-Bench, adding physics propagation cuts the roll error from 1.94° to 1.76° and pitch error from 5.14° to 4.75° on average, with PSNR rising from 19.87 to 20.54. Gains concentrate on roll and pitch, which are the axes directly coupled to gravity; yaw improves less because it rotates around the gravity axis.
•
Camera perception: on four calibration benchmarks (Stanford2D3D, MegaDepth, TartanAir, LaMAR), Puffin-World gets the best median error on every roll/pitch/FoV metric and best AUC at most thresholds, beating specialized calibrators like GeoCalib and its predecessor Puffin.
•
Camera-controllable generation: on the authors’ own Puffin-Cam-Bench, gravity direction error drops to 1.32° mean versus 28°+ for general-purpose generators such as GPT Image2, Nano Banana 2, and FLUX.2, with FID also lowest.
•
3D trajectory generation: on RealEstate10K Puffin-World edges MVGenMaster on PSNR/SSIM/LPIPS; on the harder Puffin-Traj-Bench it roughly halves camera-control errors versus the strongest baseline SEVA.
•
Emergent applications: “self-calibrated exploration” where the model detects its own gravity misalignment, predicts a corrective action, and imagines the resulting view, all in one loop.
Reach for this when you’re building novel-view synthesis, virtual staging, or embodied-agent simulation and today your generated pans and tilts visibly drift off-horizon. The recipe transfers even if you don’t adopt the whole stack: add a gravity-aware channel derived from a monocular calibrator alongside your existing ray or Plücker conditioning, and rotate that gravity vector through your known camera trajectory instead of re-estimating it per frame.
The authors released code, model weights (two variants built on Qwen2.5 + SD3.5, plus a caption model), and Puffin-16M (15M vision-language-camera image triplets plus 1M trajectories with challenging rotations). They also shipped absolute-camera annotations for ~44.5M images across 28 public datasets (ImageNet, COCO, Objects365, ScanNet, DL3DV, and others), which is directly useful even without touching their model, because most public 3D datasets restrict roll to ±5° and pitch to ±10° and have no absolute-orientation labels at all.
If your generative model has to move a camera through a scene, give it one absolute reference (gravity works) and let relative motion do the rest. Ray-only conditioning is fine for short clips near the training distribution; it breaks under long horizons and large rotations because nothing in the representation knows which way is up.
•
The gravity anchor is only as good as the first-frame perception. If the reference view is ambiguous (no horizon, no vertical structures, heavy tilt), propagation multiplies that error across the whole trajectory.
•
“Physics” here means gravity direction and latitude, nothing else. There is no rigid-body dynamics, no lighting model, no material or contact reasoning. Dynamic scenes are explicitly listed as future work.
•
The comparison against general-purpose generators (GPT Image2, Nano Banana 2, FLUX.2) uses camera errors re-estimated by Puffin-World itself, which biases the metric toward Puffin-World’s own perceptual model. The FID and qualitative gaps still favor Puffin-World, but the exact angular numbers should be read with that in mind.