SWE-Duel is a coding-agent benchmark where two agents take turns planting hidden bugs inside new features they wrote, then try to find each other’s bugs. Tasks and tests are made at evaluation time, so none can leak into training data in advance.
Static coding benchmarks have a short shelf life. The tasks get scraped into the next model’s training set, and the fixed difficulty ceiling gets saturated once agents improve. SWE-bench is the canonical example: OpenAI stopped reporting on its Verified split after frontier models could reproduce the gold patches. The usual response, release a new benchmark, buys time but repeats the cycle, because the new tasks come from the same distribution of real GitHub issues and have a fixed difficulty too.
Recent work tries to patch this. SWE-bench-Live and SWE-rebench keep pulling fresh GitHub issues after model cutoffs, but they’re still bounded by whatever bugs human developers happen to file. BugPilot and SwingArena bring LLMs into task creation, but either generate tasks ahead of time as a fixed pool, or draw task prompts from human-written issues and only auto-generate the oracles. None of them scale task difficulty with the agents being measured. SWE-Duel’s pitch is that if you want a benchmark that keeps up with frontier models, the agents themselves should be writing the problems.
Two agents play a duel on a real open-source repository, swapping roles each turn. The Red agent authors a task; the Blue agent tries to solve it blind.
Red does its work in three isolated sessions, each gated by executable checks:
•
Session A (feature): Red picks a module, writes a new feature, and ships feature tests. Gate: the repo’s existing tests and the new feature tests must pass.
•
Session B (bug): A fresh Red session gets the feature spec and feature diff, embeds a subtle bug in the feature code, and writes sealed bug tests that fail only because of the bug. Gate: the full repo + feature tests still pass on the buggy diff (so Blue sees no red flag), and the sealed bug tests pass on the clean version and fail on the buggy one.
•
Session C (blind self-repair): Another fresh instance of the same Red agent gets only what Blue will get (repo, spec, PR diff, feature tests) and must fix the bug. If it can’t, the task is thrown out. This is the solvability floor.
Blue then reviews the pull request without being told a bug exists. It wins the turn only if its revised diff passes all three suites: the repo’s existing tests, the feature tests, and the sealed bug tests. Reverting the PR loses, because that kills the feature. The scoring is a hard conjunction: in plain terms, Blue needs a clean sweep, no partial credit.
for repo in repos:
for i in range(N):
task = red.author(repo) # 3 sessions + 3 gates
if not task.admitted: # exhausted slot
score[blue] += 1; continue
patch = blue.review(task.public_bundle)
if run(existing, feature, sealed_bug, patch).all_pass:
score[blue] += 1
else:
score[red] += 1
Tournaments are round-robin, and ratings use Bradley-Terry loss. A newcomer only needs to play about log(N) incumbents (chosen by Chatbot Arena’s active-sampling rule) rather than everyone, so the leaderboard grows cheaply.
The authors ran 1,500 turns across 11 (model, harness) participants on 15 repos in 5 languages (Python, Java, Go, JavaScript, C/C++) for about $2,109 in API spend. Participants paired frontier models like GPT-5.5, Claude Opus 4.8, DeepSeek V4 Pro, Kimi K2.7 Code, and Gemini 3.1 Pro with one of three harnesses (mostly Mini-SWE-Agent, plus Codex CLI and Claude Code).
•
Solving vs. authoring are different skills. GPT-5.5 with Codex CLI was the top solver, repairing 98.3% of defects it reviewed. Claude Opus 4.8 with mini-swe-agent was the top task author, with a 27.3% Red win rate, more than double either GPT-5.5 configuration’s Red rate. Being great at fixing bugs doesn’t mean you’re great at hiding them.
•
Difficulty tracks author capability. Holding the solver fixed (DeepSeek V4 Pro) and only varying the author across three successive GLM versions, the solver failed on 4.0% of tasks from the two older versions pooled vs. 42.9% from the newest, with a significant upward trend (p = .015). This is the paper’s cleanest evidence that the benchmark actually self-scales.
•
Blue wins most of the time overall. Across contested turns, Blue succeeded 85.1%, meaning most tasks that pass admission are solvable. Logic errors were the hardest category (81.9% repair rate); input-validation bugs the easiest (95.5%).
•
Late entry is cheap and doesn’t disturb rankings. Adding Gemini 3.1 Pro took 5 matches and $243, and refitting Bradley\u2013Terry on the incumbent-only subset reproduced the full ordering exactly.
•
Oracle soundness audit. A manual check of 50 Blue losses found a 6.0% false-negative rate on the sealed bug tests. And rerunning the Session-C self-repair gate three extra times on 44 sampled tasks showed a 91.7% pooled repair rate, so admitted tasks are mostly genuinely solvable rather than lucky.
•
If you’re evaluating coding agents and worried about contamination on SWE-bench-style suites, SWE-Duel’s protocol gives you a template: generate tasks at evaluation time, gate them with blind self-repair, score with execution. The toolkit is open source and the dataset is on OSF. •
The author-vs-solver split is a useful lens for picking models. A model that’s a strong reviewer (high Blue win rate) is what you want for CI or PR review; a model that’s a strong author (high Red win rate) tells you something about depth of code understanding, since hiding a plausible bug requires grasping the feature well enough to break it subtly. The paper shows these don’t correlate perfectly.
•
The security framing is worth noting: a 27.3% rate of “plausible, test-passing, CI-evading defect” against frontier reviewers is a disclosure-relevant finding for anyone relying on automated review as a safety net. Worth testing on your own review stack.
•
For your own replications, generate a fresh task bank. The paper is explicit that releasing a bank publicly retires it for future unseen-task claims, since it then becomes training-data candidate.
•
Task difficulty is capped by the author’s own bug-finding ability (the Session-C gate). As models get more uniformly strong, this ceiling could compress differences between top participants. The authors flag this and sketch a future privileged-review gate as a fix.
•
Only 15 repos, all relatively small so that test suites run fast. Larger, messier codebases are an obvious extension but weren’t tested.
•
Harness choice matters and confounds model comparisons. GPT-5.5 with Codex CLI and Opus 4.8 with Claude Code are “native” pairings in the main table; the authors run a separate harness ablation but note its intervals are too wide to resolve a clear harness ranking.
•
The self-scaling evidence is one three-rung ladder in one model family (GLM 5.1 \u2192 5.2 \u2192 5.3) with 12\u201314 tasks per rung. Suggestive, not conclusive.
•
Sealed bug tests have a measured ~6% false-negative rate, so individual Blue losses aren’t always genuine misses.