Qwen-UI-Agent is a foundation GUI agent trained on real phones, not just simulators, that unifies GUI clicks with shell commands and batched actions in one action space. On computer-use tasks, over 40% of its actions are emitted as batches and CLI makes up 40-55% of actions, cutting steps per task substantially versus GUI-only baselines.
You’re building an assistant that clicks through a mobile app to book a flight or drives a desktop to reformat 200 receipts into a spreadsheet. Today most GUI-agent research trains against sandboxed emulators where UIs are clean, network is stable, and no pop-up ever asks for a CAPTCHA. Your users’ real phones have expired logins, ads, permission dialogs, and third-party apps that behave nothing like the sandbox. The dominant baseline here is the UI-TARS-2 line of native multimodal GUI policies, plus proprietary systems like Claude Opus 4.8 and GPT-5.6 Sol. This paper argues the gap that matters isn’t model scale, it’s that these agents rarely trained on live devices and rarely had CLI as a first-class action.
The agent operates over a unified action space: GUI primitives (click, drag, type, system_button), a cli_command action that runs bash inside the target machine, api_call for structured services, and ask_user for confirmation on sensitive steps. Critically, one model turn can emit a batch of these actions to run consecutively without another screenshot in between, useful when the next few steps are predictable. Training is a three-stage pipeline: supervised fine-tuning on trajectories, then Action RL which targets six recurring local errors (mis-grounding on look-alike icons, wrong sort order, off-by-N target counts, premature terminate, action loops, and failing to use rare actions like long_press), then Online RL using a variant of Group Relative Policy Optimization (GRPO) over complete rollouts scored by executable verifiers. For mobile, they built a fleet of ~100 physical Android phones with a health-aware scheduler and used virtual-display mechanism (scrcpy) to run multiple app sessions concurrently on one device, reportedly ~20x rollout throughput. Data comes from an agent-driven flywheel that generates tasks, synthesizes initial environment state, writes verifiers, runs trajectories, and does failure analysis to pick the next iteration’s targets.
# One decision step in Qwen-UI-Agent
obs = env.observe() # screenshot, cli stdout, api response
reasoning, actions = policy(instruction, obs, history)
# actions is a list; K=1 is single-action, K>1 is batched
for a in actions:
if a.type == "cli_command": run_bash(a)
elif a.type == "ask_user": pause_for_user(a)
else: dispatch_gui(a)
if a.needs_fresh_observation: break # end batch early
history.append((obs, reasoning, actions))
The prevailing default in GUI-agent work is to scale up sandboxed benchmark training and treat GUI clicks as the universal interface. This paper shows the opposite. Real-device experience and a hybrid GUI+CLI action space are what unlock reliability. Sandboxes don’t teach recovery from pop-ups, stateful placeholders, or scroll-wheel overshoot, and GUI-only execution turns bulk file work into hundreds of unnecessary steps. The cleanest evidence is the failure-mode breakdown of a strong GUI-only baseline on real phones, plus the finding that CLI absorbs roughly half the actions on desktop when the model is allowed to choose.
The load-bearing finding is behavioral, not a leaderboard number. Reviewing every failed Qwen 3.7 Plus trajectory on real Android, 52.0% of failures come from real-world scenario challenges (UI misreading of placeholders 24.7%, pop-ups and CAPTCHAs 18.2%, physical widgets like sliders and date pickers 9.1%) and 40.3% from execution-capability limits (exploration failure in deep app hierarchies, action loops when a tap does nothing, lost state in long horizons). These are patterns simulators structurally don’t produce. The mechanism payoff shows up in two places:
•
Hybrid action use is not a fallback but a preference. On OSWorld-Verified, CLI is 40.7% of actions and appears in 92.0% of tasks; on OSWorld-v2 it’s 55.1% and 98.2%. Batched actions are ~40% of all actions with mean batch size 3.1, and Qwen-UI-Agent-27B needs 135.8 steps/task on OSWorld-v2 versus 173-327 for GUI-only open baselines.
•
Headline scores follow. 82.1% on MobileWorld, 92.2% on MobileWorld-Real (their new 409-task real-device Chinese benchmark), 97.5% on AndroidDaily, 79.5% on OSWorld-Verified, 40.0% partial / 13.9% binary on OSWorld-v2, 73.6% on WebArena, and 81.5% on ScreenSpot-Pro zoom-in. On MobileWorld-Real they beat Gemini 3.1 Pro, Opus 4.8, GPT-5.6 Sol, and Seed 2.1 Pro by 3.5-7.5 pp.
Ablations for Action RL show +6 to +10 pp on the six targeted error patterns, and Online RL raises the fraction of trajectories that verify their own work before terminating by 14.7 pp and cuts the false-stop rate by 11.2 pp.
Reach for this design when you’re shipping an agent that acts on a real user’s device against apps you don’t control. The concrete change versus a GUI-only baseline: give the model a bash channel and let it decide when to use it. On the receipt-organization example the paper walks through, the agent stitches candidate images into one montage via ImageMagick, inspects them in a single screenshot, then goes back to GUI clicks. Also let one model turn emit several actions when the next dialog is predictable; that’s most of the step-count savings.
The paper does not link a GitHub repo, a model checkpoint, or a dataset release in the text provided. MobileWorld-Real (their 409-task Chinese real-device benchmark) is described but no download URL is given here. They mention plans to open-source their environment-synthesis methodology later. If you want to reuse ideas today, the reusable parts are conceptual: the six-category taxonomy of recurring GUI action errors, the AutoJudge trajectory-level VLM evaluator (reported 92.8% exact-match against expert labels on 666 trajectories), and the virtual-display trick for parallel mobile rollouts.
Give a GUI agent a shell and real devices to fail on, or it will keep re-clicking placeholders forever. GUI-only execution and sandbox training both hide the failure modes that dominate on real phones and desktops. The hybrid GUI+CLI action space matters most when tasks involve bulk file work or verification; the real-device runtime matters most when your users hit pop-ups, sliders, or expired logins.
•
Real-device success rates are graded by AutoJudge, a VLM-based majority-vote judge, not by deterministic verifiers. Reported accuracy against experts is 92.8%, so a few points of the mobile-benchmark deltas sit inside judge noise.
•
The strongest mobile-use claims are on Chinese-ecosystem apps (Dianping, Meituan, Amap, RedNote, Douyin); the paper doesn’t quantify how the real-device gains transfer to Western app ecosystems.
•
Several system pieces are described but not released or fully evaluated in the text: the 35B-A3B variant’s computer-use and DeepSearch training was unfinished at report time, and the higher-fidelity synthetic environments and full autonomy of the data flywheel are called out as still requiring human oversight.