DreamX-Phi 1.0 predicts future robot-manipulation videos from a starting frame plus a bimanual action trajectory by injecting each arm’s rigid-body motion into attention as SE(3) transforms, so different commanded paths produce different rollouts instead of visually plausible but action-unfaithful ones.
Suppose you’re building a robot policy and want to score candidate action sequences without running them on hardware. A video world model lets you “imagine” the outcome of each candidate and pick the best one. The catch: modern video generators inherit a bias from their pretraining. They produce clips that look right but drift from the exact motion you asked for. A rollout that moves the wrong arm, or shows a grasp when you commanded a release, is worse than useless for planning.
Prior robot-conditioned video models like IRASim encode the action trajectory as low-dimensional tokens injected via cross-attention or modulation. That’s flexible but geometrically blind: the model has to infer from scratch that two nearby end-effector poses represent a small rigid motion. DreamX-Phi’s bet is that this structure should be handed to the model directly.
The backbone is Wan2.2-TI2V-5B, a pretrained video diffusion transformer, kept mostly frozen. Four things get bolted onto it.
First, the action interface. For each arm at each frame, the paper builds a 4×4 rigid-transform matrix from end-effector position and rotation, normalizes it so both arms sit in a shared coordinate frame, and rescales by the largest observed motion. These per-arm transforms are then plugged into a parallel attention branch using PRoPE. The attention heads are split into fixed groups, one group per arm, and the transforms multiply queries, keys, and values so any two tokens interact through their relative rigid motion rather than an absolute coordinate frame. Gripper open/close is a scalar and can’t fit into SE(3), so it’s added as a learned per-arm bias on the same heads. This whole branch is a residual: initialized to zero, it starts silent and only takes effect as it trains.
Second, an auxiliary depth branch. The last few transformer blocks are copied, and the copy predicts depth latents (from Depth Anything 3 targets) with one-way cross-attention that reads RGB features. Depth is supervised in latent space, and the RGB path never depends on it, so depth is optional at inference.
Third, object-centric supervision. A SAM 3 (Segment Anything 3) mask picks out the manipulated object, and the RGB loss is upweighted on masked tokens (a small object no longer gets drowned out by static background). On top of that, a frozen V-JEPA teacher provides features on the object region, and the student is trained to match the teacher’s Gram matrix alignment of pairwise feature similarities, not the features themselves. This keeps object identity and shape stable through contact without locking the student to the teacher’s basis.
Fourth, Distribution Matching Distillation (DMD) distills the multi-step diffusion generator into a few-step student, with an adversarial term on latents, for cheaper inference.
# Per transformer block, per arm k
A_k = build_relative_SE3(pose_k, gripper_k) # 4x4 per frame
Q, K, V = project(tokens) # standard attention
Q, K, V = apply_prope(Q, K, V, A_k, heads=H_k) # rigid-motion coupling
O_act = attention(Q, K, V)
O_act += gripper_bias(gripper_k) # scalar injected here
out = pretrained_self_attn(tokens) + O_act # residual, zero-init
The usual way to feed robot actions into a video model is to squeeze the trajectory into a token or a modulation vector and let attention figure out the geometry. DreamX-Phi argues the opposite: the model should never have to re-learn that end-effector poses form a rigid-motion group. Hand it the SE(3) structure explicitly through relative-transform attention, and separately supervise the two things pose-conditioning alone can’t fix, namely scene depth and object identity through contact. The paper’s own framing is that action fidelity and scene consequence are different problems needing different signals; the leaderboard results are system-level evidence, but the design case rests on this separation.
The paper reports system-level rankings rather than a clean per-component ablation, and the authors say so explicitly. On WorldArena 2.0 Track 1 (video prediction from an initial frame plus language or actions, 1,000 episodes), DreamX-Phi ranks first out of 31 entries with an EWMScore-P of 60.65, ahead of Alpha-World and FlowWAM-FiveAges. On Track 2, where the submitted world model is used as a training environment for a $\pi_{0.5}$ policy that’s then tested on held-out Adjust Bottle episodes in RoboTwin 2.0, it hits a 67.19% success rate, tying for second behind WOVR-PLUS. On the older WorldArena 1.0 Track 1, an offline evaluation reaches 76.88, which the authors note is 3.24 points above the leading pinned entry in that snapshot (though this was not a live submission). Qualitatively, arms, grippers, and objects stay coherent through rollouts even under domain randomization of backgrounds, lighting, and distractors. The paper does not isolate how much of this comes from PRoPE versus depth versus V-JEPA. That’s flagged as a limitation.
Reach for this design if you’re building a learned simulator for a robot with a known kinematic structure. The pattern to steal: don’t tokenize the action trajectory generically. If your control signal has group structure (SE(3) poses, camera extrinsics, articulated joints), inject it as relative transforms into attention and split heads by controllable entity (per arm, per camera, per finger). Then separately supervise the things the control signal doesn’t constrain, like scene geometry via depth and object persistence via a frozen self-supervised teacher.
Model weights and inference code are promised at github.com/AMAP-ML/DreamX-Phi after the WorldArena 2.0 IROS challenge concludes. Training data is a curated mix of egocentric video (for visual priors), real-robot demonstrations, and simulated RoboTwin 2.0 clips upsampled with the authors’ DreamX-Refiner. The training set totals aren’t fully broken down beyond a 178.7-hour AgiBot filtered split and 25,000 RoboTwin clips.
When your control signal has geometric group structure, wire that structure into attention directly and supervise scene consequences separately, rather than hoping a generic token adapter will infer both at once.
•
No matched ablations. The paper is a challenge-submission system report, so we don’t know how much PRoPE alone contributes versus the depth branch, the SAM 3 (Segment Anything 3) reweighting, or V-JEPA alignment. The Core Insight is plausible but not isolated.
•
Evaluation is narrow. Track 2 uses only the Adjust Bottle task, and everything runs in RoboTwin 2.0 simulation. Real-robot transfer, other embodiments, and other tasks are unverified.
•
This is a forward dynamics model, not a policy. It predicts video from externally supplied actions; it doesn’t propose actions. Using it for closed-loop control requires a separate planner or policy on top, which the paper does not evaluate end-to-end.