GenCeption turns a pretrained text-to-video diffusion model into a single-pass perception model by feeding it clean video latents at timestep zero and negating the velocity output, matching task-specialist accuracy with 7× to 500× less training data than depth/geometry foundation models.
You’ve shipped a product that needs depth maps, segmentation masks, human keypoints, and camera trajectories from user video. Today that means four different models, four inference pipelines, four sets of training pipelines. Each is a specialist (a DINOv3-style backbone, a SAM 3 (Segment Anything 3) fork, a Depth Anything 3 head) and each gets swapped out yearly as a new SOTA lands.
The premise here: a single text-to-video generative backbone already contains the priors needed for all of these, because generating physically-plausible video forces the model to internalize 3D geometry, motion, and text-grounded semantics. You post-train it once, steer tasks with text prompts, and drop the four-model zoo.
Start with a frozen-then-finetuned WAN 2.1 text-to-video diffusion backbone. Normally this model would take noisy latents plus a timestep and iteratively denoise over 50 steps to hallucinate a video. GenCeption skips that whole loop.
The trick has three moves. First, feed the model the clean input video latents (not noise) and pin the timestep to t=0, telling the network it’s at the endpoint of denoising. Second, because the backbone was trained under a Rectified Flow objective to predict velocity (noise minus target), the authors negate the raw output so it directly approximates the target latent. Third, one forward pass, decode with the VAE, done. No iterative sampling.
To unify tasks, all dense outputs (depth, surface normals, segmentation, camera raymaps) are packed into 3-channel RGB space in [0,1]. Camera poses become a spatial “Rothko Raymap” laying ray origins in the center and ray directions around the edges. Sparse outputs (2D/3D keypoints) get handled by appending learnable tokens per frame, decoded by a small MLP. Text prompt picks the task. Training uses one L2 loss for everything; per-task quirks like depth scale ambiguity are absorbed by normalizing data (median-normalize depth, then a log-clip into [0,1]) rather than by custom losses.
# perception forward pass
latents = vae.encode(input_video) # clean, not noised
prompt_emb = text_encoder(task_prompt) # e.g. "predict depth"
velocity = dit(latents, prompt_emb, t=0) # single pass
target_latents = -velocity # negate: v = eps - x0
output = vae.decode(target_latents) # RGB-packed prediction
# for sparse tasks: append learnable tokens, decode via MLP
Training data is mostly synthetic: 7,500 Blender-rendered human videos (800 RenderPeople assets, 200 CMU Motion Capture Dataset motions), augmented with TartanAir and Virtual KITTI for depth and camera trajectories. Referring segmentation is the one task that uses real data.
The prevailing bet for a vision foundation model is a discriminative self-supervised objective: mask-and-reconstruct (VideoMAE), joint-embedding prediction (V-JEPA), or feature distillation. This paper argues the opposite. The generative objective of predicting future pixels conditioned on text is the one that forces a network to internalize geometry, physics, and language-grounded semantics all at once, and those are exactly the priors perception needs. The load-bearing evidence isn’t the headline benchmark wins, it’s the head-to-head where V-JEPA and Video MAE V2, finetuned on the same data, underperform the diffusion backbone.
Under matched finetuning data, the WAN 2.1 video-diffusion backbone beats the largest available V-JEPA and VideoMAE V2 variants on depth estimation. That’s the mechanism-isolating result: same downstream data, same task, different pretraining objective, generative wins. A second control shows that training the Diffusion Transformer (DiT) from scratch (no pretrained weights) yields a nearly flat learning curve, and performance scales monotonically with how many pretrained layers you transfer in.
•
Across depth, surface normal, camera pose, foreground segmentation, referring segmentation, and 3D keypoints, GenCeption matches or beats task-specialist SOTA models including Depth Anything 3, SAM 3 (Segment Anything 3), D4RT, VGGT-\u03a9, Sapiens, and Genmo.
•
Data efficiency is the standout: comparable depth performance to D4RT and VGGT-\u03a9 with 7× to 500× less training data.
•
Preliminary scaling: performance rises with both model size (1.3B → 14B) and data volume.
•
Trained only on synthetic humans in Blender, the model transfers zero-shot to real footage, multiple instances per frame, and non-human categories (animals, anthropomorphic characters). Fine detail in outputs (cat whiskers, hair edges) exceeds the fidelity of the training targets.
•
One negative: jointly training the sparse-token 3D keypoint head with dense tasks hurts both. The authors attribute this to the learnable tokens disrupting the pretrained attention patterns, and recommend minimal architectural surgery on the backbone.
Inference cost is real. On a v6e TPU at 480×832, 81 frames, the 1.3B model runs 5.92s (13.6 FPS effective), the 14B model 10.03s. Still, that’s one forward pass instead of 50 denoising steps.
Reach for this framing when you’re building a perception stack for video-native products (robotics telemetry, sports analytics, AR try-on, video editing) and you currently maintain a fleet of task-specific models. The recipe suggests you can start from an open video generation checkpoint, post-train on synthetic renders for your target modalities in RGB-packed format, and route tasks by text prompt. The synthetic-to-real transfer result matters here: you don’t need a large annotated real-video corpus to bootstrap.
A project page exists at genception.github.io. The paper builds on the open-weights WAN 2.1 backbone but does not, in the text provided, commit to releasing GenCeption weights, training code, or the synthetic dataset. Treat the artifact story as unconfirmed until the project page says otherwise.
If you want a model that understands the physical world, train it to predict the physical world, then repurpose the features. Discriminative pretext tasks are a detour.
•
The clean-latent-at-t=0 trick is specific to rectified-flow diffusion backbones. It’s not obvious this transfers cleanly to noise-prediction or x0-prediction formulations without redoing the output-negation math.
•
The synthetic training data is narrow: 800 human assets, 200 motions, Blender renders. Referring segmentation needed real data. Tasks that depend on textures, materials, or scene classes far from human-centric renders may not enjoy the same sim-to-real free lunch.
•
Sparse-output tasks (keypoints) don’t compose well with dense tasks in joint training. If your product needs both from one model, the current architecture forces a compromise, and the paper flags this as unresolved.
•
Comparisons to task specialists span many benchmarks but the paper acknowledges “mixed behaviors”: generalist joint training regresses on some dense benchmarks versus a specialist trained on that task alone. A single generalist isn’t strictly dominant.