OSWorld-Pro grades Computer-Using Agent trajectories at the subgoal level instead of only on final deliverables, using an LLM Judge aligned with 67k human annotations to expose where agents stall, misclick, or waste steps on long tasks the top model only finishes 75.7% of the time.
If you ship an agent that drives a desktop (clicking menus, typing in LibreOffice, running shell commands), today’s standard way to score it is OSWorld: run the agent for up to hundreds of steps, then check whether the final file matches a reference. That’s a single pass/fail at the end.
That framing has three concrete problems the authors call out. An agent that dies on subgoal 1 gets the same zero as one that dies on subgoal 9, so you can’t tell whether a new model is actually making progress. Stronger agents also start reward-hacking the outcome check (the paper cites a real OpenAI incident where an agent reached restricted answer keys on a third-party server). And you get no signal on where time was wasted, which is exactly what you need to fix a harness or pick a cheaper model. OSWorld-Pro is the authors’ attempt to score the process, borrowing the idea of step-level grading from math benchmarks like PRM-800k but adapted for GUI tasks where the “right” path is open-ended.
The core move: decompose each task into an ordered list of subgoals that must be completed in sequence (open the file, then add the slide, then rename it, then save as). Tasks average 9.2 sequentially dependent subgoals across 3.45 apps, versus roughly one subgoal and 1.34 apps in OSWorld. 305 tasks cover three buckets: Diversity (unusual apps like Archive Manager, LibreOffice Draw), Coordination (≥4 apps per task), and Robustness (non-Ubuntu Linux distributions and desktops like Fedora, AlmaLinux, Xfce, LXQt).
Subgoals are drafted by DeepSeek-V4-Pro from a task goal plus required app list, then human-validated. For each step in a recorded agent trajectory, annotators label four fields in order: which subgoal(s) the step targets, whether that subgoal is feasible in the current environment, whether the step makes progress, and whether it completes the subgoal. Later fields only count if earlier ones are correct. To keep labels reliable, both an annotator and an independent reviewer label the task blind, then the platform (SuperAnnotate) surfaces disagreements for discussion. Inter-annotator Cohen’s kappa lands above 0.8 on every field.
Because humans at 5–20 hours per task don’t scale, the authors also build an LLM judge and benchmark it against the human labels. The final judge design puts an entire trajectory (reasoning, action, screenshot per step, up to hundreds of steps and ~500 MB of images) into a single API call:
for task in tasks:
steps = load_trajectory(task) # reasoning, action, screenshot each
subgoals = task.subgoals
# ONE call, all steps + all subgoals in context
labels = gpt_5_6_sol(prompt(steps, subgoals), reasoning="max")
# labels[i] = {targeted:[...], feasible, progressed, completed}
score_task(labels, human_ground_truth)
Only the GPT-5.6 family tolerated the payload size in the authors’ tests; Claude, Gemini, and open models hit payload, image-count, or context errors.
The judge gets close to humans on the easy fields, not the hard ones. GPT-5.6-sol at max reasoning matches humans within ~3 points on identifying which subgoal is targeted (97.0 vs 98.4) and on completion, but trails badly on feasibility (61.9 vs 91.6) and progression (74.7 vs 94.1). Those two fields are also where humans agree with each other less, so part of the gap is genuine subjectivity. The authors note that higher reasoning effort actually hurts feasibility accuracy, because models increasingly predict “feasible=Yes” (dropping from 9.7% “No” to 5.1% “No” against a 15.3% human rate), apparently talking themselves into believing any subgoal can be done.
OSWorld-Pro is harder than OSWorld, and the gap is biggest for open-weight models. Top closed model Claude Opus 4.8 reaches 77.7% overall vs 83.4% for the top Opus on OSWorld. The best open-weight entry, Qwen 3.8 Flash Next at 125B, reaches 55.1% on OSWorld-Pro while open-weight models generally clear 80% on OSWorld. For a matched model, MiniMax-M3 drops from 75.2% on OSWorld to 28.9% here. Open-weight models do best on rare apps, worse on multi-app coordination, worst on non-Ubuntu Linux, which the authors attribute to training-data scarcity for exotic desktops.
Process signals reveal failure modes outcome scores miss. Minimax M3 and Kimi K3 succeed on click actions only ~39–42% of the time (vs 72–93% for Claude and GPT-5.6), with concrete failures like clicking the wrong coordinates to close a window. Claude Opus 4.8 shows a jump on both Click (86.3→92.5%) and Drag+Move (41.0→74.3%) over Opus 4.7, which the authors read as deliberate pointer-behavior training. Even top models waste steps: Claude Opus 5 Max was once stuck on a single subgoal for 59 steps of goal-irrelevant actions despite eventually completing everything.
Model size alone isn’t the story. Within GPT-5.6, Luna does better than Terra at judging but worse than Sol; Sonnet 5 beats Opus 5 but loses to Opus 4.7/4.8. Smaller models partly compensate by taking more steps (Sonnet 5: 108.7 steps vs Opus 5: 76.0). Cost per task spans 20× for similar quality (GPT-5.6-Luna at $0.51 vs GPT-5.6-Sol at $9.56).
•
If you’re picking a judge for your own agent eval, the paper’s judge-model table is a direct cost/quality tradeoff: GPT-5.6-Terra Max slightly beats GPT-5.6-Sol Medium at 22% lower cost, and Luna Max roughly matches Terra Low at one-sixth the price. Worth testing the same pattern on your task mix before defaulting to the biggest judge.
•
If you’re evaluating a CUA harness, the action-category breakdown (click, drag, keyboard, scroll, execution) is the kind of diagnostic that tells you whether to invest in better grounding/pointer logic versus better planning. The paper demonstrates this on recorded trajectories, not as a training recipe, so treat it as a diagnostic lens rather than a direct training signal.
•
For model selection on long-horizon desktop work, step-efficiency numbers (Successful Effort, Failure Persistence, Infeasible Persistence) argue against assuming “more steps = more thorough.” Some models grind on infeasible subgoals for 40+ steps. If your product pays per step or per token, this matters more than headline accuracy.
•
Don’t read the 77.7% top score as “CUAs almost work.” OSWorld-Pro is pre-recorded trajectories graded post hoc, not a live deployment test, and the Robustness category (non-Ubuntu desktops) collapses open-weight scores to the single digits.
The benchmark grades trajectories generated separately, then judged; it is not a live agent loop with environment feedback into the judge. Judge coverage is limited to the GPT-5.6 family because of payload constraints, so results could shift as other providers raise image/context limits. Each model and judge is run once, justified by low observed inter-run variance (<0.6% on judge Macro-F1) and by following OSWorld’s single-run convention, but single-run numbers still carry noise. Subgoals come from an LLM decomposition that humans then correct; tasks where subgoals aren’t sequentially dependent were filtered out, which keeps the benchmark coherent but means it does not cover genuinely parallel workflows. Finally, the authors flag that the judge’s weakest field (feasibility) overlaps with the field humans themselves disagree on most, so some headroom here may not be recoverable with a smarter judge alone.