More multimodal training environments isn’t always better. This paper picks environments by the abilities an agent uses in them, and schedules difficulty by first giving text scaffolds then removing them, yielding ~143% average relative gain over a base Qwen3-VL agent.
Suppose you’re training a vision-language agent to click through GUIs, play browser games, or solve visual puzzles. The current playbook is: scrape together as many interactive environments as you can, verify each one runs, then throw them all into a reinforcement learning loop. This paper shows that pouring more environments in can actually hurt the model. Prior work on multimodal environment pools like VisGym and Gym-V focuses on whether each individual environment is well-formed (executable, correct rewards). This paper argues that’s necessary but not sufficient: the distribution of environments matters as much as their individual quality, and multimodal training is more fragile here than text-only training.
The authors first document the problem. On a pool of 200 multimodal environments, mixed training drops accuracy 10.7% versus training on each environment alone, whereas the text-symbolic version of the same environments only drops 1.3%. Gradient cosine similarity between environments is more polarized (more strongly negative) in the multimodal case, meaning environments actively fight each other during optimization. Error analysis blames two multimodal-specific bottlenecks: extracting state from pixels, and modeling how actions change the world.
The fix has two pieces. First, Ability-aware Environment Selection picks a subset of environments by looking at what the agent does inside them, not what they look like on the surface. They roll out Qwen3-VL and Gemini-3-Flash in each environment, use GPT-5 to segment trajectories into “atomic abilities” (things like identify target position, verify constraints), and build a per-environment ability profile. A greedy selector then maximizes ability coverage while penalizing profile overlap (redundancy) and negative gradient cosine (conflict).
Second, Hierarchical Difficulty Curriculum organizes training difficulty along two axes. A harness is textual scaffolding, like a symbolic description of the current visual state, or explicit rule text, that helps the model get past the perception and world-modeling bottlenecks early. Training starts with harnesses on, then peels them off across five levels. Within each harness level, the classic knob (grid size, entity count) advances as an inner curriculum.
for step in training:
e = sample_environment()
h = sample_harness(current_harness[e]) # earlier levels mixed in
s = sample_scale(low[e], current_scale[e])
instance = build(e, h, s); rl_update(instance)
if recent_success(e) > tau_scale:
current_scale[e] += 1
if current_scale[e] == target and recent_success(e) > tau_harness:
current_harness[e] += 1; current_scale[e] = 0
The prevailing view is that scaling multimodal agent training means adding more verified environments to the pool. This paper shows the opposite. When environments are picked by shared underlying abilities and ordered by removing perception scaffolds before scaling state complexity, a small curated subset beats the full pool, and multimodal-specific failure modes get directly targeted. The load-bearing evidence is that a 30-environment Ability-aware Environment Selection subset outperforms training on all 170 environments, and that the conflict-control term (gradient-based) is what drives most of the gain in the ablation.
•
The single most load-bearing finding: in the Ability-aware Environment Selection ablation, removing the gradient conflict term collapses OOD relative gain from 40.3% to 2.8%. Removing the redundancy term drops it to 25.3%. Gradient-level conflict detection is doing most of the work, not surface similarity.
•
Naive environment scaling is non-monotonic. Even with a per-environment budget held constant (256 samples each), success rate peaks around 40-60 environments and drops as more are added, all the way down to 23.6% OOD at 160 environments.
•
On the headline setup with Qwen3-VL-4B, AES + HDC gets 223.9% relative gain on in-distribution environments and 68.5% on held-out ones, versus 80.1% and 7.5% for training on all 170 environments.
•
Selection transfers across scales: the subset chosen using 4B trajectories still helps the 8B model, giving 144.3% ID relative gain there.
•
Selection also transfers across model families: applied to InternVL3-8B (never used during selection), AES still beats random selection by roughly 5 points on OOD single-turn success.
•
General multimodal benchmarks (MMMU, MathVision, MMStar) barely move, so this training doesn’t obviously trade off general perception.
Reach for this when you’re training a multimodal agent by RL across a heterogeneous pool of interactive tasks: browser navigation, visual puzzle solving, GUI control. Instead of training on every environment you have, roll out a strong model plus your target model in each, have a stronger LLM tag the trajectories with reusable ability labels, then pick a subset that covers those abilities with low gradient conflict. When you train, start each environment with text descriptions of the visual state, then remove those scaffolds as the model gets competent.
Code is at GitHub. The 200-environment pool is built on top of existing multimodal environment works (VisGym, Gym-V) rather than newly synthesized, so reproducing the full setup depends on those upstream releases. The paper doesn’t specify a released model checkpoint.
Diversity that matters is diversity of abilities the model exercises, not diversity of task descriptions or screenshots. A gradient-cosine check between environment pairs is a cheap, blunt tool that catches optimization conflicts surface embeddings miss entirely, and it’s what separates a curated subset from a random one.
•
The ability annotation pipeline leans on GPT-5 plus manual inspection to produce trajectory segmentations and merge synonymous labels. If your ability taxonomy is noisier, or you can’t afford a strong annotator model, AES’s redundancy term degrades.
•
Gradient cosine similarity between environment pairs requires actually computing gradients on batches from each environment, which is extra offline compute the authors acknowledge as a limitation.
•
Everything is validated at Qwen3-VL-4B/8B scale on a 200-environment pool sourced from two prior works. Whether the same non-monotonic scaling curve and the same 30-environment sweet spot hold at 70B, or on a pool of 2000 environments, is untested.