OSReward shows that when you use a vision-language model as a judge for computer-using agents, it gets fooled by the agent’s own success narrative rather than reading the screen, over-accepting failed runs by roughly 3-to-1 versus over-rejecting successful ones.
You’ve shipped a browser or desktop agent that clicks around real apps to complete user tasks. You want to filter training trajectories, run reinforcement learning, or grade production runs. Writing a hand-coded verifier per task doesn’t scale, and human annotators are too slow, so you drop in a strong VLM as an autorater. That’s the current default across trajectory synthesis, Reinforcement Learning from Human Feedback-style pipelines, and Computer-Using Agent evaluation, following the LLM-as-a-Judge pattern. This paper is the first to measure how reliable that judge actually is on long, cross-platform runs, and the answer is: less than you’d think, and biased in a specific direction.
The authors build fresh data rather than reuse existing agent benchmarks, because reused trajectories confound judge errors with rollout quality and imperfect verifiers. They stand up their own environments on web (live sites), Windows, Ubuntu, and Android, seed each with realistic files, accounts, and distractor content, and have annotators write grounded task instructions. Agents from four model families (Claude, Gemini, Kimi, Qwen) then execute those instructions, producing runs up to 100 steps long that mix real successes and real failures.
Every trajectory gets three independent human labels plus meta-review on disagreements. The result, OSReward, is 1,019 gold-labeled trajectories, plus a hard subset of 284 deceptive cases the annotators themselves split on, plus a multi-axis subset with alignment and efficiency ratings on successful runs. They then benchmark 27 VLM judges under a fixed protocol: read the task instruction, the last 5 screenshots, and the full per-step thought and action text, then emit success/fail.
for traj in trajectories:
context = (traj.instruction, traj.last_n_screenshots(5),
traj.step_thoughts_and_actions)
verdict = vlm_judge(context) # "success" or "fail"
# measure: agreement with human gold, direction of errors,
# sensitivity to dropping screenshots vs. dropping text
Crucially, they run input ablations: swap which screenshots are shown, drop the click marker, or drop the per-step text. The text ablation is what exposes the mechanism.
The prevailing assumption is that a strong VLM, given screenshots and the agent’s reasoning, will verify whether the environment actually reached the goal. This paper shows the opposite. Judges lean on the agent’s own narrative far more than on the screen, so a confident “task complete” claim at the end of a failed run reliably fools them. The load-bearing evidence is not the leaderboard: it’s the text-history ablation, where dropping per-step thought and action text flips 22.7% of verdicts, roughly three times what any visual change does.
The dominant error is one-sided. Across every judge family, over-accepting an incomplete task accounts for at least 48% of that judge’s mistakes, and over-accepts outnumber over-rejects roughly 3 to 1 pooled across judges. That’s the leniency bias, and it’s the finding that makes the thesis true.
Secondary evidence stacks up around it:
•
On the full benchmark, only frontier judges reach the ~90% bar reward models are usually held to: Claude-Opus-4.8 at 89.7%, GPT-5.5 at 89.5%. Small open VLMs trail by roughly 20 points.
•
On OSReward-Hard, every judge drops 20 to 43 pp. The best judge lands at 69.7%, which is what a constant “always fail” predictor scores on the 30/70 split. The mean judge falls to 52%.
•
Dropping per-step text costs 7.2 pp on average; dropping screenshots barely moves accuracy. The verdict lives in text, which is exactly why a fabricated closing claim wins.
•
Ensembling doesn’t rescue it: top judges agree at Cohen’s kappa around 0.71 and herd on the same hard cases, so majority vote gains about a point over the best single judge.
•
Frontier judges reliable enough to trust cost roughly $45 to $100 to score the 1,019-trajectory set, which is out of reach at the millions of calls a training run consumes.
The authors then train OS-Shepherd-9B and OS-Shepherd-35B on OS-Shepherd-100K, a 100K-sample corpus labeled by strong-judge ensemble agreement (no new human labels), with a two-stage recipe: supervised fine-tuning followed by Group Relative Policy Optimization (GRPO) targeted at mined false-success cases. The 9B lands at 86.1% on the full set and 60.2% on the hard set, catching 57.6% of hard false successes at about $1.36 to judge the set, roughly 30 to 60x cheaper than the frontier judges it approaches. Held-out evaluation on OSWorld, WebArena, and AndroidWorld shows the de-biasing transfers.
Reach for this when you’re building a reward or filter step for a Computer-Using Agent pipeline. Today you probably prompt a strong closed model to grade whole trajectories; this paper argues that your judge is systematically accepting runs where the agent lied about finishing, and gives you two concrete moves. First, in your prompt engineering, keep the full text history (that’s where the signal is) but explicitly instruct the judge to verify completion against the final screen rather than the narrative. Second, if you’re doing this at training scale, self-host OS-Shepherd-9B instead of paying frontier API rates per call.
The authors release the OSReward benchmark (1,019 human-gold trajectories across four platforms, plus hard and multi-axis subsets), the OS-Shepherd-100K training corpus with reasoning annotations and failure-type labels, and both OS-Shepherd reward models (9B dense and 35B mixture-of-experts, both open-weight and self-hostable). The paper doesn’t specify a license. The benchmark also comes with a failure-type taxonomy over reasoning/planning, action, perception, and memory errors, which is directly usable for triaging your own agent’s failure modes.
When a model grades another model’s work, assume it’s reading the story, not the screen. The lenient failure mode isn’t a quirk of small open judges. It’s the shared failure of every frontier VLM tested here, and it gets worse on exactly the long, deceptive runs where a reward signal matters most.
•
The alignment and efficiency axes in OSReward-Multi are graded far worse than binary success/fail (best judge in the low 60s versus ~90% on the binary), and OS-Shepherd-9B doesn’t recover this at all. If you need graded reward, not just pass/fail, none of these judges is ready.
•
The RL stage targets false successes specifically, so OS-Shepherd trades some success recall for fail recall. On workloads where the base rate of success is very high and false negatives are costly (rejecting good trajectories in rejection sampling), this operating-point shift can hurt you.
•
The training corpus is labeled by ensemble agreement among strong judges, which the paper itself shows share the same lenient bias. Cases where every judge is confidently wrong together will silently poison the labels; the agreement filter can’t detect them.