MAI-Gym is one open platform where GUI agents on phones, desktops, and browsers share a single rollout path across evaluation and Online reinforcement learning, scaling to 20,000 parallel environment instances and letting one 9B policy train jointly on mobile and computer tasks.
If you want to train an LLM agent to actually use a phone or desktop, you hit two walls that text-only RL does not. First, every attempt needs its own real device or VM reset to a known state (a GUI agent changes files, settings, and screens, so attempts cannot share one instance like a code sandbox can). Second, your training stack and your evaluation stack usually end up as two different codebases, so a model’s benchmark score depends partly on which harness ran it, not just on the weights.
Prior open GUI gyms pick a side. Infrastructure-focused ones (BrowserGym, SCALECUA, and others) each cover a single platform and release training code that runs at most a few hundred environments in parallel. Data-focused ones curate new apps and tasks but ship no training loop. MAI-Gym tries to be both at once, across mobile, desktop, and web.
The core design choice is boring and important: evaluation, data collection, and RL training all call the same rollout path and the same agent harness. The harness is the glue code that builds the prompt from screenshots and history, calls the policy, parses the output into actions, and executes them. If training keeps the last 5 screenshots and eval keeps 10, you are measuring a different agent than you trained. MAI-Gym makes that drift impossible by construction.
An environment manager sits between the agent and the backends (Android via ReDroid containers, desktop VMs, headless Chromium). It exposes five operations (request, observe, reset, step, evaluate) and hides the mess of provisioning, resets, failure recovery, and reward computation. Resets are fast enough for RL scale: median 165s on Android, 37s on desktop, 6s on browser, with 1,000 fresh instances coming up in roughly 90-330 seconds.
For long multimodal trajectories, MAI-Gym uses window-wise history so the inference engine can actually reuse its KV cache. The common “keep the last k screenshots” trick breaks caching every step because the prompt’s prefix keeps changing. Instead, steps are grouped into overlapping windows of K=10 with O=2 overlap; within a window, new content only gets appended, and the prefix only changes once per window.
Training uses Group Relative Policy Optimization (GRPO) on top of the slime trainer. The scheduler applies policy-adaptive sampling: tasks the policy solves sometimes get the full group of attempts, while never-solved or already-mastered tasks get small budgets (GRPO extracts no gradient from a group where every attempt gets the same reward).
for round in training_rounds:
tasks = sample_by_success_rate(pool, eta=0.8)
for task in tasks:
instances = env_manager.request(n=group_size)
trajectories = [run_harness(policy, inst, task)
for inst in instances] # same harness as eval
rewards = [verifier(t.final_state, t) for t in trajectories]
advantages = (rewards - rewards.mean()) / rewards.std()
update_policy(trajectories, advantages) # window-wise packing
sync_weights_to_inference_engines()
On the data side, MAI-Gym synthesizes mocked apps (self-contained apps with resettable backends), including real-world clones of seven Chinese consumer apps (12306 train tickets, DiDi, Meituan, etc.) and 42 novel apps invented from scratch. For task generation, it first builds a coherent cross-app world (a persona plus scenario populates emails, spreadsheets, calendar entries that share people and facts), then writes tasks that force the agent to combine evidence across apps. Verifier code and a golden solution are synthesized independently from the spec, then validated by executing initial-complete-reset on a live instance.
They train MAI-Gym-9B from Qwen3.5-9B with an SFT warm-up followed by one joint RL run over mobile and desktop tasks. Reported results on the two benchmarks whose environments (but not tasks) appear in training:
•
78.4% on MobileWorld and 76.2% on OSWorld-Verified, the best among open GUI agents under 10B parameters. For scale, UI-Venus-2-9B reports 75.2/70.8 and the larger GUI-Owl-1.5-32B reports 43.9/56.5.
•
On five environment-OOD benchmarks (environments never seen in training), RL improves over the SFT warm-up on all five, suggesting the policy learned something beyond memorizing training environments.
•
Joint RL on both domains beats training two per-domain experts and merging them by weighted parameter averaging, in both domains simultaneously (no merge weight wins both).
•
Scaling synthesized tasks from 500 to 5,000 lifts MobileWorld 69.2% to 78.4% at a fixed rollout budget.
•
Sim-to-real: RL only in the 12306 train-ticket clone, evaluated on 52 paired tasks on a physical phone, improves real-app success from 28.8% to 48.1%. The authors note transfer is incomplete: the clone-to-real gap actually widens from 7.7 to 25.0 points after training.
•
On novel apps the model has never seen, RL from verifier rewards alone (no demonstrations) lifts held-out success from 25.8% to 45.2% and cuts mean trajectory length from 33.3 to 16.4 steps.
•
Window-wise history gets an 85.9% prefix-cache hit rate versus 42.5% for step-wise, cutting training update time from 2166s to 811s per batch, at the cost of ~19% lower rollout throughput.
The SFT warm-up actually accounts for most of the raw score gain over the Qwen3.5-9B base; RL on top adds roughly 4-8 points of average improvement across benchmark groups.
•
If you are building a GUI agent and currently maintain separate eval and training code, the shared-harness design is the main architectural lesson here, independent of whether you adopt their stack. Train/eval drift is a silent accuracy tax, and they measured it in their own runs.
•
If you want to try GUI RL without building the environment layer yourself, the GitHub release includes the framework, synthesis skills, 49 mocked apps, 2,000 verified tasks (1,000 each for MobileWorld and OSWorld), and the MAI-Gym-9B checkpoint. The RL training code is released, which distinguishes it from several data-focused gyms that ran RL internally but did not publish the trainer. •
If you need agents for apps whose real versions involve transactions (money, orders, bookings), the real-world-clone recipe is worth testing: build a resettable clone, run RL there, then evaluate on the real app. The 12306 result is encouraging but is one app, one language, and 52 tasks, so treat it as evidence that transfer is possible rather than a general reliability claim.
•
Window-wise history is a drop-in idea for any long-horizon multimodal rollout where you want KV-cache reuse. The win is in training-step time more than in rollout throughput.
•
The large real-app transfer number comes from one app with 52 tasks evaluated on a physical phone, and the clone-to-real gap grew after RL, so clone performance overstates real-app ability.
•
Several baseline comparisons use published numbers rather than the authors’ re-runs, and MAI-Gym-9B has the natural advantage that its training environments overlap with two of the seven benchmarks (the “Task OOD” setting shares environments but not tasks).
•
The SFT warm-up contributes most of the absolute score; the RL stage adds a smaller delta on top. Framing MAI-Gym purely as an RL win understates how much of the result is supervised data.
•
The 20,000-instance scale number is a demonstrated capacity, not the configuration used for the main 9B training run (that used 512 GPUs with up to 20,000 concurrent environments over ~96 hours).
•
The diagnosis-driven synthesis loop (failures feed back into new tasks) is described but not evaluated across repeated rounds in this paper.