CLASHarness generates verifiable multi-agent conflict tasks for computer-use agents by pre-committing visual ground truth before rendering and compiling graders from executed reference trajectories. Even the best model solves only 28.7% of 600 tasks, with closing coordination failing >58% of the time.
Most computer-use agent benchmarks like Osworld or Webarena assume one agent in a cooperative environment where every tool output tells the truth. Real multi-agent deployments break that assumption. A sub-agent reports a deployment is healthy from a stale cache, while a dashboard shows failing health checks. A peer holds a shared lock and claims it was released. The main agent cannot just trust peer text. It has to open the dashboard, verify what the system actually looks like, then take the correct state-changing action (rollback, revoke, hand off).
Building tasks like this by hand does not scale. Letting an LLM freely invent them produces three recurring defects: the story drifts across turns, the answer leaks into the task prompt, and the grading rubric asks for things no reference solution can actually achieve. The authors want a pipeline that rules out those defects by construction.
The core idea: never let the generator freelance on the facts that matter. Decide the decisive ground-truth value first, bake it only into a rendered image, then compile the grader from a reference run that actually executed in a sandbox. If the reference run passed its own checks, the task is provably solvable.
Concretely, generation is split into three tiers and eight stages. Tier 1 sets up the shared world: scenario outline, sub-agent roles and permissions, initial files and UI. Tier 2 synthesizes 10 rounds in a package. Every 3-4 rounds it refreshes milestones (so prompts don’t balloon), then builds each round through eight narrow sub-steps: write instructions, draft peer dialogue, update files, inject visual assets, commit the decisive value v* with an automated leakage scan, draft a plan, execute it in a sandbox, and compile checks from the resulting state diff. Tier 3 does a cross-stage consistency audit, renders the images, and runs an exclusivity gate that scans all text for v* and fails if the answer leaks outside the image.
Two repair mechanisms keep this cheap. Schema and formatting bugs get fixed deterministically in code with zero model calls. Semantic mismatches (e.g., instruction contradicts the grader) retry only the broken sub-step with the error message attached, rather than regenerating the whole 10-round package. Total API calls stay linear in round count, averaging ~95 per package.
for package in seeds:
world = init_tier1(package) # outline, roles, workspace
for phase in chunks(rounds, size=3-4):
plan_phase_milestones(phase, world)
for r in phase:
spec = draft_round(r, world) # instructions, dialogue, files
v_star = commit_fact(spec) # scan prompt for leaks
traj = execute_in_sandbox(spec) # reference trajectory
checks = compile_from(traj) # exact + semantic
world = apply(traj, world)
audit_and_render(package) # exclusivity gate on v*
Grading is hybrid: deterministic code checks for things like “did file X get written with value Y” and scoped Llm As A Judge semantic checks for free-text deliverables, scored by 3-sample majority vote.
The benchmark has 600 rounds across 20 operational domains, 3,760 checks, and 7,970 rendered visual assets. 346 of the 600 rounds require inspecting an image because v* lives only there. 15 frontier models were evaluated.
•
Overall difficulty: top model GPT-6 Astra reaches 28.7% pass rate; Claude Opus 5 leads the 20-step budget cohort at 25.2%. Mean per-check scores sit at 55-67% for frontier models, so failures come from missing one or two steps in a long chain, not from being clueless.
•
Blind trust vs. visual grounding: when visual assets are withheld, pass rates collapse to 11-13% of their sighted values across all models tested (p<10^-11), even on rounds where no rubric check explicitly requires image inspection. Visual evidence is load-bearing, not decorative.
•
Resolution vs. detection: on a controlled comparison between rounds that require acting on a conflict versus rounds that only require reporting it, pass rates drop from 28.1% to 11.0%, a difference-in-differences of -18.2 pp against a control arm that stays flat. Spotting the contradiction is easy; executing the rollback, lock release, or permission revocation is where models fall apart.
•
Closing-action bottleneck: even GPT-6 Astra and Claude Opus 5 fail over 58% of closing actions (terminal protocol steps like releasing shared locks or handing off to sub-agents). Time-window synchronization fails 83-97% of the time. These are evaluated by deterministic code checks, so no judge noise.
•
Reasoning effort inverts: across the 20-step API models, reasoning-token share correlates negatively with pass rate (ρ = -0.697). Gemini 3.5 Flash at high effort drops to 3.0% because thinking loops consume the step budget; it hits the 20-step cap on 89% of rounds.
•
Horizon matters more than effort: giving Gemini 3.8 Flash 60 steps instead of 20 raises pass rate from 12.8% to 30.7% while cutting cost per solved task by 21%, because the extra steps go to productive verification rather than loops.
•
Robustness: rankings are stable across an alternate generator model (Spearman ρ=0.96) and across three different semantic judges (ρ≥0.98, Cohen’s κ≥0.90), and no model moves more than one rank under judge swaps.
•
If you’re building or evaluating multi-agent computer-use systems, the released GitHub repo gives you a conflict-focused test set that your current benchmarks probably don’t cover. The failure modes it surfaces (blind trust, missed lock release, missing handoff) are the ones that bite in production. •
The resolution-vs-detection split is the single most actionable finding. If you only measure whether your agent flags an inconsistency, you’ll overestimate reliability by roughly 17 pp relative to whether it fixes system state. Worth adding state-change assertions to your own evaluation harness.
•
The effort inversion is a useful prior: for interactive conflict-resolution agents, cranking reasoning effort can hurt. Worth testing medium effort against high before paying for the latter. The paper shows this on hosted frontier models, so it is directly transferable.
•
The pipeline itself (pre-commit the answer, scan for leaks, compile graders from an executed reference) is a reusable recipe for anyone generating LLM evaluations. Even if you don’t need multi-agent conflict, the exclusivity gate pattern prevents a common silent failure where models pass benchmarks by exploiting leaked text.
•
If you control a model (e.g., fine-tuning), the authors frame CLASHarness as a data engine for post-training. That is a plausible next experiment, not something the paper evaluates.
•
Conflicts are evaluated through logs and workspace traces, not live chat with other live agents. The paper calls this out: it mirrors ticket and review workflows, but real-time negotiation dynamics are out of scope.
•
65% of scored checks go through an LLM judge. The authors validate stability across judges and show primary diagnostic findings (visual grounding, terminal actions, deadlines) use only deterministic checks, but semantic scoring noise still exists.
•
The benchmark contains only conflict rounds, no conflict-free baselines. Isolation of “conflict handling” relies on the resolve-vs-investigate gate (p=.052, right at the significance boundary), not on a conflict-vs-no-conflict contrast.
•
The paper’s model lineup includes names like “GPT-6 Astra” and “Claude Opus 5” that may be preview or internal labels at time of writing; absolute pass rates will shift as models update, but the paper’s rank-stability analyses suggest the ordering and the failure modes should hold.