Latent Interface Training (LIT) trains a robot policy to generate actions from a goal pose before it ever sees an image, then adds vision through a narrow bottleneck supervised to predict that same goal pose, cutting the shortcuts that break policies when the camera or lighting changes.
You’ve trained a robot manipulation policy on a few hundred demos. It works when the camera, lighting, and table clutter match training. Move the camera 20 cm, add a mug on the table, and success rate collapses. The policy learned to associate task-irrelevant pixels (a shadow, a background texture) with the demonstrated action, because those pixels happened to correlate with the right move in the small training set. This is a Shortcut learning problem, well documented in imitation learning.
Modern robot foundation models like Vision-Language-Action model systems and World-Action Model systems all share the same structure: a pretrained vision-language or video backbone feeds rich visual features into an action expert that outputs motor commands. That direct pipe from raw visual features to actions is exactly where shortcuts get baked in. Prior fixes either enrich the visual features (add depth, motion traces, 3D position) or pretrain the action head on language alone before turning vision on. The authors argue neither actually constrains how the action head consumes vision once vision is connected, so the shortcut can re-form during the visual training phase.
LIT is a two-stage recipe that slots into an existing VLA or WAM without changing its backbone or action head architecture.
Stage 1: learn to act without ever seeing an image. The action expert is trained to produce the next chunk of ~H motor commands given only the language instruction, current robot state, and one extra input: the end-effector pose at the end of that chunk (position, orientation, gripper). Think of it as “given that your hand needs to end up here holding this, what’s the trajectory?” This forces the action head to become a goal-conditioned motion generator, with no opportunity to memorize pixel-to-action correlations.
Stage 2: bolt on vision, but only through a supervised bottleneck. A small set of 100 learnable tokens (the latent interface) is inserted between the backbone and the action expert. These tokens cross-attend to the backbone’s visual and language features and are the only path visual information can take to reach the action expert. Crucially, an auxiliary MLP head is trained to reconstruct the same terminal end-effector pose from these tokens. That reconstruction loss pushes the bottleneck to encode goal-relevant spatial content (where’s the object, where should the hand go) and drop the rest.
# Stage 1: image-free action prior
g = terminal_pose(chunk) # 8-dim: pos, axis-angle, gripper
cond = concat(backbone(lang, state), encode(g))
loss = flow_matching(action_expert(chunk, cond), chunk)
# Stage 2: visual conditioning only via supervised latent tokens
Z = learnable_tokens # K=100 tokens
for layer in range(L):
Z = Z + self_attn(Z)
Z = Z + cross_attn(Z, backbone_lang_state[layer])
Z = Z + cross_attn(Z, backbone_visual[layer])
loss = flow_matching(action_expert(chunk, Z), chunk) \
+ 0.3 * mse(pose_decoder(Z[-1]), g)
At inference, the pose encoder and decoder are dropped. The policy runs on image, language, and state, just like the baseline.
The authors drop LIT into four different robot foundation models (π0.5, MolmoAct2, Fast-WAM, ImageWAM) and evaluate on LIBERO (in-distribution) and LIBERO-Plus (seven zero-shot perturbations: camera angle, sensor noise, lighting, backgrounds, robot start pose, object layout, instruction phrasing). They train from scratch, no fine-tuning of released checkpoints, so their baseline numbers are lower than published leaderboard scores.
•
In-distribution is preserved. Average LIBERO success matches or slightly beats each baseline (e.g., π₀.₅ goes from 87.75% to 91.80%).
•
Out-of-distribution improves across the board. LIBERO-Plus overall success rises by +3.87 to +10.70 percentage points depending on architecture. The biggest gains are on the perturbations most likely to trigger shortcuts: camera viewpoint (+22 pp on π₀.₅, +27 pp on FAST-WAM) and sensor noise (+22 pp on MolmoAct2). LIT wins 26 of 28 architecture-by-perturbation cells, and no loss exceeds 2.11 pp.
•
Real robot, three tasks (LEGO pickup, wiping trash, egg transfer), single MolmoAct2 policy trained on 300 demos. Aggregated success under unseen lighting, a top-only camera view, and added distractors improves by 13.3 to 16.7 pp. On Transfer Egg specifically, distractor-condition success jumps from 20% to 90%.
•
Ablations isolate what matters. Removing Stage 1 costs 3.7 pp OOD. Removing the pose-reconstruction loss costs 3.1 pp. Letting the action expert see backbone visual features directly (bypassing the bottleneck) costs 4.2 pp. Learnable tokens alone, without Stage 1 or pose supervision, give only 2 pp over baseline. All three ingredients contribute, and none replaces the others.
•
Attention and counterfactual probes. Baseline attention maps drift when the scene is perturbed; LIT’s stay locked on the robot-object region. When the authors swap the language goal but keep the scene fixed, baseline keeps executing the original goal (a classic shortcut tell); LIT redirects.
The ablations support the authors’ causal story: the gains come from the combination of goal-conditioned pretraining, the exclusive bottleneck, and the pose reconstruction, not any one piece.
•
If you’re training a manipulation policy from scratch and OOD robustness matters more than squeezing the last point of in-distribution accuracy, this recipe is worth trying. It doesn’t change your backbone or action head, just the training procedure and adds ~100 latent tokens plus a small pose MLP. The evaluated architectures span two coupling styles (shared self-attention in π₀.₅, layer-wise cross-attention in MolmoAct2) and both VLA and WAM designs, so the pattern is not architecture-specific.
•
You need terminal end-effector poses per action chunk in your training data. For most teleoperated demos this is trivial (it’s just the recorded state at the chunk boundary), but the method assumes you can express a goal as an 8-dim SE(3) + gripper vector. Tasks where the “goal” isn’t well captured by a single terminal pose (long-horizon reasoning, continuous interaction) are not evaluated.
•
Don’t read this as evidence that LIT beats fine-tuned production VLA checkpoints. The authors explicitly train from scratch and say their baselines are not comparable to published fine-tuned numbers. What the paper shows is a controlled, matched-budget comparison against the same architecture without LIT.
•
The real-robot evaluation is small (10 rollouts per OOD condition per task, three tasks, one architecture). Treat it as directional evidence that simulation gains transfer, not as a deployment benchmark.
•
Project page: magiclab-nus.github.io/LIT.
•
All simulation results use policies trained from scratch on LIBERO demonstrations only. Whether LIT’s benefit stacks on top of large-scale cross-embodiment pretraining is untested.
•
The pose supervision assumes a single meaningful terminal pose per action chunk. The paper doesn’t explore chunks where the “goal” is ambiguous or where the terminal pose is a poor summary of intent.
•
Real-world evaluation covers three tasks and only MolmoAct2. The 13-17 pp aggregate gains are encouraging but the sample size per condition is small.
•
One perturbation (language instruction rephrasing) shows small regressions on two of four architectures. The method is aimed at visual shortcuts and doesn’t claim to help with language variation.