SWE-bench Science asks coding agents to repair real scientific repositories where a broken patch corrupts not just outputs but the evidence behind a claim, and even the best agent lands below 50% Pass@1 across 119 tasks in 20 domains.
You’ve shipped a coding agent that fixes bugs in a customer’s Python codebase. If it silently makes a unit test pass by hard-coding a value, you notice in code review. Now imagine the codebase is a molecular-dynamics simulator, and the “fix” makes the tests pass by breaking an energy-conservation invariant nobody wrote down. The bug ships. The paper published on top of that simulator is now wrong.
Existing agent benchmarks like SWE-bench measure whether a patch passes the project’s tests. That’s necessary but not sufficient for scientific software, where correctness is defined by physical or mathematical contracts that live in papers, not in the test suite. This benchmark targets that gap across 98 real repositories spanning chemistry, physics, biology, materials science, and 16 other fields.
The authors hand-curated 119 tasks and grouped them into three flavors of scientific repair work. Issue-driven tasks are classic bug fixes rolled back from real GitHub issues and PRs. Expert-exploratory tasks give the agent a symptom (weird convergence, biased results) with no confirmed root cause and ask it to reason its way there. Engineering-integration tasks require wiring together multiple modules to complete an end-to-end scientific workflow.
Each task has a public test suite the agent can iterate against and a hidden private suite the evaluator runs after submission. The private tests are the interesting part: they include reverse checks designed to catch hard-coded solutions, heuristic pseudo-fixes, and repairs that only work on the visible example. So an agent that overfits to the public tests gets caught.
The headline number is Pass@1, which is 1 only if every private test passes. The paper also reports Fail2Pass (fraction of previously-failing private tests now passing) and Pass2Pass (fraction of previously-passing tests still passing, i.e. regression check).
The second contribution is an ablation the authors call scientific auxiliary information separation. For 91 of the 119 tasks, they can cleanly strip out the scientific rationale (paper excerpts, equations, expected properties, domain-specific diagnoses) while leaving the repository, tests, and runnable environment intact. They run the same agent twice, with and without that scientific context, on the same task.
for task in tasks_91:
env = load_repo_and_tests(task) # identical both runs
result_with = agent.run(env, sci_context=task.rationale)
result_without = agent.run(env, sci_context=None)
# compare pass@1, tokens, and which tasks flip
This isolates the marginal value of scientific guidance from the repository engineering context.
The prevailing assumption when building scientific coding agents is that more domain knowledge in the prompt is straightforwardly good: hand the model the relevant equations and it does better. This paper shows scientific context is a two-sided lever. Well-grounded information constrains the search and improves both accuracy and token efficiency on weaker models, but poorly aligned information causes anchoring on a plausible-but-wrong explanation and can lower exact-repair success on stronger ones. The evidence is the paired ablation where the same context helps DeepSeek-V4-flash and hurts GPT-5.6-sol, not the leaderboard.
The load-bearing finding is the ablation split. On the 91-task separable subset, giving scientific context raised DeepSeek-V4-flash pass@1 from 16.48% to 23.08% but lowered GPT-5.6-sol pass@1 from 36.26% to 31.87%. Token use moved in opposite directions too: the weaker model spent more tokens with context, the stronger model spent fewer. Task-level overlap shows GPT solved 8 tasks only-with-context and 12 only-without, so this isn’t noise, it’s real anchoring on some cases and real help on others.
•
Nobody clears 50% pass@1. Best is Claude Code with Opus at 47.9%, then Codex with GPT-5.6-sol at 46.2%. Meanwhile public-test scores sit at 96-100%. That gap is the whole reason private tests exist.
•
No single winner across metrics. GPT-5.6-sol leads private score, fail-to-pass, and pass-to-pass. Claude-Opus-5 leads overall pass@1 and both Issue-driven and Expert-exploratory. DeepSeek-V4-Pro leads Engineering-integration.
•
The manual failure audit sorts errors into four buckets: scientific-knowledge/abstraction deficits, misguided exploration or surface-level repair, incomplete repair coverage across modules, and failure to generalize the scientific principle beyond the observed case. Claude-Opus-5 has the fewest categorized errors (58) and stands out for making almost no surface-level repairs (2), suggesting it more often traces symptoms to root causes.
Reach for this benchmark when you’re building a coding agent aimed at technical or scientific customers (simulation vendors, computational-biology platforms, EDA tools) and you need to know whether your agent will silently corrupt a domain invariant to make a test pass. The private-test design with reverse checks against hard-coded fixes is the transferable idea, you can copy that pattern into your own eval harness even if you never touch the benchmark.
Code is on GitHub, data on HuggingFace, and there’s a leaderboard. Task 5 in Appendix B is a good place to start reading, it captures the flavor: reconcile two periodic representations of the same crystal while preserving physical energy.
More domain knowledge in the prompt is not free. On a weak model it constrains the search; on a strong model it can anchor a wrong story that the code then rationalizes. Test both directions before you decide your retrieval pipeline is helping.
•
The ablation is two models on 91 tasks. The authors explicitly note it’s descriptive, not causal, and doesn’t establish statistical significance. Treat “strong models get hurt by scientific context” as a hypothesis worth testing on your own stack, not a settled result.
•
Domain coverage is uneven. Chemistry has 24 tasks, six domains have one task each. Cross-domain claims about which agent is best at, say, geophysics are not supported.
•
The failure taxonomy is a manual audit, so the counts depend on annotator judgment. Useful as a diagnostic vocabulary, less so as a precise measurement.