UniClawBench evaluates proactive agents on 400 bilingual real-world tasks running in live Docker containers, using a three-agent closed loop where a hidden supervisor with rubrics grades progress and a firewalled user simulator provides follow-up feedback without leaking answers. Even top models pass under 50% of tasks.
You’re building a personal-assistant agent that browses live sites, drives desktop apps, and coordinates files across a laptop. You want to know: is my agent actually failing at vision, at long-context memory, at cross-app coordination, or at picking the right tool? Existing agent benchmarks like WebArena and OSWorld use sandboxed mirrors and single-turn scoring, and they bucket tasks by scenario (“office,” “shopping”), so a failure tells you nothing about which capability broke. They also assume a fixed ground-truth answer, which decays the moment a real Amazon price changes overnight.
The benchmark reorganizes 400 hand-written tasks (English and Chinese, half each) around five capabilities: skill usage, exploration, long-context reasoning, multimodal understanding, and cross-platform coordination. Each task lives in a fresh Docker container with real browsers, files, and services. Instead of comparing the agent’s final answer to a stored answer, the authors write a hidden rubric of step-by-step checkpoints per task. Grading is done by three cooperating LLM agents. The executor is the system under test. A hidden supervisor sees the full trajectory plus the rubric and emits pass/fail/continue with a score. A user simulator sees only the executor’s visible trajectory plus a coarse status flag (not the rubric, not the rationale), and writes the next user message. A deterministic rewriter sanitizes that message before it reaches the executor. This is the “information firewall” that lets multi-turn feedback happen without leaking the answer key.
for cycle in range(max_followups + 1):
trajectory = executor.run(task, workspace) # in Docker
verdict, score, rationale = supervisor.judge(
trajectory, hidden_rubric, hidden_refs)
if verdict == "pass" or verdict == "fail": break
# only 4 fields cross the firewall, no rubric, no rationale
handoff = {verdict, state, recoverable, score}
msg = user_sim.write_followup(trajectory, handoff)
task = rewriter.sanitize(msg)
Each task ships a YAML declaring public inputs and a hidden eval_rule.md with a nine-section rubric; the supervisor scores against fixed rubric sections. Runs use a per-turn and global wall-clock timeout, and up to two follow-up cycles.
The usual way to build an agent benchmark is to freeze a scenario, freeze the expected answer, and score the final output in one shot. This paper argues the opposite. In a live environment where ground truth drifts and users iterate, the benchmark itself must be an agent system: hidden rubrics replace stored answers, and a firewalled user-simulator loop replaces single-turn scoring. The evidence that pins this down is not the headline pass rate but the reliability study showing the automatic supervisor agrees with a human majority vote 92% of the time, and the cycle-by-cycle progression showing scores rise across follow-up turns.
The load-bearing finding is a wide, systematic “halfway failure” gap: models routinely earn high checkpoint-based average scores while failing the final pass criterion. For example, Claude Opus 4.8 under OpenClaw reaches an average score of 0.702 but a pass rate of only 0.475 overall. Every model shows this pattern. Secondary findings:
•
Best overall pass rates come from Claude Opus 4.8 (0.475) and GPT-5.4 (0.407), but no model clears 50%. Open-source Qwen 3.5-Plus and Kimi K2.6 land competitive with mid-tier closed models.
•
Capability breakdown: skill usage and exploration are relatively tractable; long-context, multimodal, and cross-platform tasks are much harder. Multimodal pass rates sit near or below 0.2 for every model tested.
•
Framework matters more than the authors expected. Same model, three frameworks: OpenClaw consistently wins on pass rate; EDICT burns 2–3× the tokens with lower pass rates because sub-agent handoffs drop context; Nanobot is token-efficient but too thin on context to close out long trajectories.
•
Multi-turn feedback measurably helps: scores rise across the two allowed follow-up cycles, which is the whole point of the closed loop.
Reach for this when you’re evaluating a proactive-agent product that has to survive live websites, real desktop apps, and users who correct it mid-task. Instead of maintaining a stale golden-answer file, port the three-role pattern: write a hidden checkpoint rubric per task, grade with a supervisor LLM against those checkpoints, and let a firewalled user-simulator LLM push follow-up turns. The 92% supervisor-human agreement is the number to point at when someone asks whether LLM-as-judge is trustworthy for this.
Code and tasks are at GitHub. The framework is designed to swap in your own agent binary via the YAML task package, so it’s usable as an evaluation harness, not just a leaderboard. The 400-task set is directly runnable if you have Docker and API budget for a supervisor model (the authors use GPT-5.4 with high reasoning effort for both supervisor and user simulator).
When ground truth drifts and users iterate, replace stored answers with hidden checkpoint rubrics and put a firewall between your grader and your user simulator. The firewall is the load-bearing part. Without it, the simulated user leaks the rubric back to the agent and your benchmark quietly becomes an open-book test.
•
400 tasks is small, and all of them are hand-written by the authors. Coverage bias is real, and one bad rubric contaminates a whole capability score.
•
The supervisor and user simulator are both GPT-5.4. If your executor is also GPT-family, shared-model bias in judging is a live concern the paper does not fully rule out beyond the 50-trajectory human study.
•
Live environments are non-reproducible by design. A task that hits a real Amazon page can score differently next week because the page changed, not because the agent did.