VLAct changes how you continue-pretrain a Vision-Language-Action model backbone: freeze the shallow layers, mix in image captions, and supervise with three continuous action heads at once, so the backbone learns action features that survive being paired with a new head or a new robot, including a humanoid the model never saw during pretraining.
Suppose you’re shipping a robot manipulation policy. You take a strong vision-language model, tack on an action head, and fine-tune the whole stack on your teleoperated demos. It works on the arm you trained on. Then someone asks you to swap in a diffusion-style action head, or run the same backbone on a bimanual rig, and performance falls apart. You retrain from scratch.
That’s the pain point here. Web images and text scale by scraping; robot trajectories don’t, because a human has to physically drive the robot. So the field’s default (crank up trajectory count, fine-tune end-to-end on one action head like π0 (pi-zero) does) leaves the backbone over-fit to one decoder geometry and one embodiment. VLAct argues the backbone itself, not just the data, is a design variable.
Start from Qwen3-VL-4B as the base vision-language model. Continue-pretrain it on a mix of open robot datasets (DROID, InternA1, RoboCoin, MolmoAct) plus captioning data. Three ideas do the actual work.
First, don’t wreck the VLM prior. Robot data is visually narrow. If you update every layer, the backbone forgets the general perception it learned from web data. So VLAct freezes the vision encoder and the lower half of the LLM layers during pretraining, and mixes in caption data (LLaVA-ReCap, ShareGPT4V) as a semantic anchor. Both are unfrozen at downstream fine-tuning time.
Second, supervise with multiple action heads in parallel, not one. A pilot study shows that pretraining with a single continuous head like OFT action head gives great numbers when you fine-tune with that same head, but transfers badly to other continuous heads. The backbone has quietly reorganized its features around one decoder’s geometry. VLAct attaches three heads simultaneously (OFT action head, PI action head, and GR00T action head) to the same shared latent and sums their losses:
z = backbone(image, instruction) # shared latent
a_oft = head_oft(z)
a_pi = head_pi(z, noise, tau) # flow matching
a_gr00t = head_gr00t(z, state, noise, tau)
loss = L_oft(a_oft, a) + L_pi(a_pi, a) + L_gr00t(a_gr00t, a)
# at fine-tune time: discard all three, attach a fresh head
Third, share action dimensions across robots only where they physically match. Different robots have different action spaces. Rather than giving each robot its own head (isolates learning) or padding everything to a common vector (aligns coordinates that mean different things), VLAct uses one shared head with a partially unified layout: gripper open/close is shared across embodiments, arm joints stay embodiment-specific and get masked when inactive. A wrap-aware L1 loss handles the fact that -179\u00b0 and +179\u00b0 are 2\u00b0 apart, not 358\u00b0.
At downstream fine-tuning, you throw away the three pretraining heads and the caption stream, attach a fresh task-specific head, and train normally.
The prevailing move in Vision-Language-Action model work is to scale trajectory count and fine-tune the whole backbone end-to-end against one action head. This paper shows the opposite pays off. Treat the VLM backbone as the first-class design variable, and force it to serve multiple decoders and multiple embodiments during pretraining so it can’t overfit to any one of them. The most telling evidence isn’t the headline benchmark score; it’s that a backbone pretrained only on Franka and AgileX arms, fine-tuned on 20% of the data for a humanoid it never saw, beats full-data baselines on that humanoid.
The load-bearing result is the cross-embodiment transfer on RoboCasa-GR1. VLAct pretraining never saw the GR-1 humanoid. With just 20% of the downstream trajectories, VLAct hits 49.5%, edging past the full-data GR00T-N1.6 baseline at 47.6%. With full data VLAct reaches 54.0%. This is the finding that makes the thesis true: the backbone genuinely learned reusable action structure, not just a fit to the training embodiments.
Secondary evidence stacks up around this:
•
On LIBERO-Plus (robustness perturbations to camera, lighting, layout, noise), VLAct scores 82.6%, beating the same-backbone Qwen3VL-OFT baseline by 7.6 points with identical downstream fine-tuning. It also edges Alibaba’s ABot-M0.
•
On RoboTwin 2.0 bimanual, VLAct-OFT hits 92.5% clean / 90.8% random in the data-scaling setting, competitive with industrial systems like LingBot-VLA and InternVLA-A1. Swapping the downstream head (OFT vs PI vs GR00T) keeps results within 3.4 points, supporting the head-transfer claim.
•
On RoboDojo (an ARX X5 leaderboard, another unseen embodiment), VLAct ranks 6th of 35 by success rate and beats every entry explicitly labeled a world-action-model.
•
Real Franka experiments: 92.5% vs 77.5% on short-horizon in-domain, and 72.0% vs 44.0% on dual-arm tasks, despite pretraining only on single-arm data.
All of this uses fully open-source data and a 16-GPU training setup.
Reach for this recipe when you’re building a manipulation policy stack that you expect to redeploy: new robot next quarter, different action head next year, more demos landing every week. The concrete move is to insert a continued-pretraining stage between your off-the-shelf VLM and your task fine-tune. Freeze the vision encoder and lower LLM layers, mix caption data (~equal loss weight with actions), attach OFT + PI + GR00T heads in parallel over your aggregated robot data, and share only physically-comparable action dimensions across embodiments. When you fine-tune for a specific product task, throw the three heads away and attach whatever head your deployment needs.
The authors say they will release training scripts and checkpoints; the project page is the entry point. The pretraining data is DROID, InternA1, RoboCoin, MolmoAct plus standard caption datasets, all open. If you don’t already have Qwen3-VL-4B fine-tuning infrastructure, most of the value here still transfers to other VLM backbones, though the paper only validates the recipe at 4B scale.
Pretrain the backbone against decoders and embodiments you don’t plan to ship, so the features you ship stay reusable. Head diversity during pretraining is cheap (one backbone forward pass, three lightweight heads) and it buys you a backbone that survives being redeployed. Single-head pretraining looks great on the matched-head benchmark and quietly locks you in.
•
Validated only at 4B backbone scale. The paper explicitly flags that the optimal recipe may shift with larger VLMs, and none of the freezing or head-diversity ratios were retuned at scale.
•
The “unseen embodiment” transfer story rests on morphologies (GR-1 humanoid, ARX X5 bimanual) that still share the manipulation-with-grippers regime with the pretraining robots. Legged locomotion, mobile bases, or dexterous hands are outside what was tested.
•
RoboDojo and RoboTwin leaderboards don’t normalize training compute. VLAct’s rank-in-top-quartile framing is real, but comparisons against industry systems with unknown training budgets should be read as “competitive,” not “better per FLOP.”