4DAnyone turns a single casual phone video of a person into a free-viewpoint 4D avatar by generating dozens of consistent novel-view videos, using two tricks that keep a video diffusion model coherent when target views exceed what fits in one attention pass.
You want to let users film themselves once with a phone and drop the result into a game, VR chat, or product demo as a free-viewpoint avatar. Today, doing this well requires a calibrated multi-camera studio like the 48-camera rig used in DNA-Rendering. Monocular reconstruction methods exist but can’t invent plausible back-and-side appearance, so avatars look broken from any angle the phone didn’t see.
The standard workaround is a two-stage pipeline: use a camera-controlled video diffusion model to synthesize novel-view videos, then fit a 4D Gaussian Splatting representation to those videos. This works for a handful of views but breaks at the ~16+ views 4DGS actually needs, because the diffusion model can’t hold that many views coherent at once.
The root problem is that a Diffusion Transformer (DiT) can only fit so many view-tokens in one attention pass. Past ~4 views, you have to split targets into groups and denoise them separately, which creates two failures: (1) each new group ideally conditions on every previously generated view, but that reference set grows linearly and blows the budget, and (2) groups denoised in isolation can’t talk to each other, so global structure drifts between them.
4DAnyone fixes both. Reference Context Packing (RCP) takes the growing pile of already-generated reference views and squashes them into a fixed-size token budget using multi-scale patchify layer layers borrowed from FramePack. Nearby views are redundant anyway, so a mixed-resolution pack keeps global layout with a few tokens per view and reserves detail budget for the most informative references. Reference cost stays constant no matter how many views accumulate.
Target Context Routing (TCR) exploits a well-known property of diffusion: early (high-noise) steps decide global structure, late (low-noise) steps polish details. So at high noise, TCR cyclically shifts which views are grouped together each step, letting information leak across the whole view set over time. At low noise, it locks in fixed adjacent-view groups so neighbors refine details together without disturbing structure.
For pose control, they skip dense depth (unreliable in the wild) and condition on a 3D skeleton from an Human Mesh Recovery model, rasterized with a z-buffer so front-back arm ambiguity is resolved. The base model is Wan2.2-TI2V-5B fine-tuned in three stages on studio, synthetic, and in-the-wild data.
# TCR inference loop over one denoising trajectory
ref_ctx = pack_references(src_video, ref_views) # fixed-size RCP
z = {i: randn() for i in target_views}
for n, t in enumerate(schedule): # T -> 0
if t > t_switch: # high noise: rotate for global structure
groups = rotating_groups(target_views, size=4, step=n)
else: # low noise: adjacent groups for detail
groups = adjacent_groups(target_views, size=4)
for G in groups:
z[G] = denoise(z[G], cond=ref_ctx, skel=skeletons[G], t=t)
return decode(z)
The prevailing move when a diffusion model can’t fit all your views in one pass is to pick a fixed subset of anchor views or use a sliding window with overlap. This paper shows the opposite. Don’t shrink the reference set or freeze the grouping. Compress every reference into a fixed token budget, and rotate which target views share a group across denoising time so structure can propagate even when memory can’t. The evidence that matters is the ablation isolating Reference Context Packing and Target Context Routing, not the headline benchmark score.
The load-bearing result is the ablation. Removing Reference Context Packing degrades all three consistency metrics because the model loses appearance guidance for unseen viewpoints. Removing Target Context Routing alone degrades them too, and removing both is substantially worse than either, confirming the two mechanisms address genuinely different bottlenecks (conditioning vs. cross-group communication).
On the routing strategy itself: they tried random permutations, strided reordering, and sliding (cumulative one-position shift). Only sliding improved all three consistency metrics; random gave no gain and strided actively hurt, suggesting that preserving local view adjacency during routing matters. A sweep over the high-noise/low-noise switch point saturates at t_s/T = 0.2 (so ~80% of steps use rotating groups, final ~20% use fixed adjacent groups).
On the two evaluation benchmarks, DNA-Rendering (held out) and DyMVHumans (out-of-distribution for everyone), 4DAnyone beats three baseline paradigms across generated-video consistency, generated-video reconstruction, and downstream 4D Gaussian Splatting reconstruction PSNR/SSIM/LPIPS. Baselines fail in characteristic ways: MV-Performer distorts side and back views, TrajectoryCrafter accumulates depth errors and breaks on front-to-back changes, and a fine-tuned ReCamMaster (equipped with the same RCP and TCR for a fair comparison of the conditioning signal) produces plausible frames but imprecise camera control that degrades 4DGS.
Failure modes they own up to: loose flowing fabric (skeletons say nothing about it, so it varies across views), and inherited pose errors when HMR gets the source pose wrong (e.g., a dancer en pointe rendered as flat feet everywhere).
Reach for this when you’re building a phone-to-avatar feature and today you’re either forcing users into a rig or accepting that back-of-body views look melted. The recipe is: run an HMR model on the input to get a 3D skeleton track, render depth-buffered skeleton videos at your desired output cameras, then let 4DAnyone hallucinate photoconsistent RGB for those cameras before you fit a standard 4D Gaussian Splatting model. End-to-end inference for a ~5-second clip is roughly 2 min HMR prep + 7 min diffusion (on one H20) + 30 min 4DGS fit (on one RTX 4090). They note 10 denoising steps works nearly as well as 20 thanks to the strong conditioning.
The project page promises source code and video results. They also demonstrate a chained single image → animated video → 4D avatar pipeline by combining an off-the-shelf pose-driven animator (Wan-Animate) with 4DAnyone, which is a useful pattern if your input is a photo rather than a video. They introduce a new synthetic training set, MVGameHuman (38k synchronized 24-camera clips from an in-house game engine), but don’t state its release status.
When a diffusion model can’t hold your whole problem in one attention window, compress the context that grows and rotate the context that fragments. Sliding-window and fixed-anchor tricks are two ways of giving up; RCP+TCR is one way of not giving up.
•
Skeleton conditioning is the whole trick for reliability, so anything skeletons don’t describe (flowing dresses, capes, hair, held objects) will drift across views. This is a human method, not a general dynamic-scene method.
•
The output quality is upper-bounded by the HMR model. If it mis-estimates the source pose, every generated view faithfully renders the wrong pose. Nothing in this pipeline corrects it.
•
Training cost is not casual: 3 stages on 128 H20 GPUs over ~3 days. And inference is a fine-tuned 5B video diffusion model plus a 50k-iteration 4DGS fit per subject. It’s a studio-grade backend for a phone-grade frontend.