SWE-Bench ProMax is a 170-instance, seven-language code refactoring benchmark where gold patches touch 11.4 files and 261.6 lines on average, and the best frontier agent resolves only 41.2%, versus 75%+ on the benchmark it aims to replace.
You ship a coding agent that takes issues and edits a repo. Today you probably measure it on SWE-bench Verified, where 86% of tasks touch a single file and frontier agents already clear 75%. That number stopped separating good agents from great ones, and an audit found ~60% of unsolved instances had broken tests or leaky specs, which is why OpenAI deprecated it. This paper argues the honest next test isn’t harder bug fixes but refactors: behavior-preserving edits that ripple across dozens of files. The prior refactoring benchmarks (RefactorBench and SWE-Refactor) are single-language and, in one case, unverified. ProMax is the multilingual, expert-curated version.
ProMax is a benchmark, not a method, so the contribution is the construction pipeline. The authors scraped GitHub for post-January-2025 commits whose messages contain “refactor” but not “bug fix”, touch both source and test files, and live in repos with 500+ stars in one of seven languages (Python, Java, TypeScript, Go, C, C++, Rust). That yielded 29,782 candidates.
Each survivor gets a Docker environment rebuilt at the pre-refactor commit, using SWE-Factory for automation. If the gold patch doesn’t make the original test suite pass in that container, the instance is dropped.
Then humans (with LLM assistance) do the load-bearing work. For every remaining instance they read the diff, throw out tasks that are too small or single-file, and audit the test suite for two specific defects: overly narrow tests that lock in implementation details a valid alternative solution would fail, and overly broad tests that check behavior the issue never asked for. Both get deleted. Then they rewrite the issue description from scratch so it is a necessary-and-sufficient spec for the gold patch: a correct solution must satisfy it, and it must not admit unintended solutions. 170 instances survive.
for commit in github_search(langs=7, after="2025-01", kw="refactor"):
env = build_docker(commit.parent)
if not env.tests_pass(apply=commit.patch):
continue # env or gold broken
if commit.files < 2 or too_simple(commit):
continue # not a real refactor
tests = drop_narrow_and_broad(commit.tests) # human review
spec = rewrite_issue(commit, gold=commit.patch, tests=tests)
if is_necessary_and_sufficient(spec, commit.patch):
yield Instance(env, spec, tests, gold=commit.patch)
Evaluation runs each model under two agent harnesses, Mini-SWE-Agent and OpenHands, with a 300-step and $10 cap per instance. Pass@1 is the only headline metric.
The prevailing move when a benchmark saturates is to mine harder issues from the same distribution: trickier bugs, longer tickets. This paper takes a different route. The saturation problem isn’t that tasks are too easy, it’s that tasks are too local and their tests are too loose; refactoring forces both wide file scope and behavior-preserving specs, so it stresses the exact skill (cross-file coordination) that current agents are worst at. The evidence isn’t the 41.2% ceiling on its own, it’s the file-count analysis showing agents systematically edit fewer files than the gold patch, regardless of budget.
The load-bearing finding is a behavior pattern, not a leaderboard number. Across Claude Sonnet 4.6 and Kimi-K2.5, the gold patch distribution needs about 20 files to hit 90% coverage, but both agents plateau near 10 files. Agents locate and edit the core files correctly, then fail to propagate the change to peripheral call sites, config, docs, and test fixtures. This is what the authors call incomplete refactoring, and it’s the dominant failure mode.
Secondary findings, in decreasing order of how much they should update your priors:
•
Best resolve rate is 41.2% (GPT-5.2 under OpenHands), far from saturation, versus 75%+ on SWE-bench Verified.
•
Failed attempts consume more interaction rounds than successful ones, not fewer. Agents enter edit-revert loops (the paper calls this unproductive exploration) instead of expanding scope.
•
Cost does not track capability. Claude Sonnet 4.6 spends $4.77/instance for 38.8%. GLM-5 spends $0.24 for 36.5%. Open-weight models come within roughly five points of proprietary ones at a fraction of the spend.
•
Scaffold matters as much as the model. Moving from mini-swe-agent to OpenHands lifts GPT-5.2 from 21.8% to 41.2%. Richer file-editing and sandboxed execution tools disproportionately help large refactors.
•
No model wins every language. Claude leads Rust (63.6%) and TypeScript (53.6%); GPT-5.2 leads Python (48.3%) and C (75.0%); GLM-5 leads Java. Gemini-3-Pro scores 0.0% on TypeScript under OpenHands, which the authors attribute to training-data variance rather than language difficulty.
Reach for this if you’re evaluating a coding agent you actually plan to point at a large repo. The scenario it targets: a customer asks your agent to rename an API, migrate a header, or unify a scattered pattern across the codebase. Your existing SWE-bench-Verified score won’t tell you whether it will finish the job or edit five files and declare victory. ProMax will, and it will do so in seven languages so you catch the case where your agent is quietly Python-only.
The benchmark spans 70 repositories with permissive-to-copyleft licenses (Apache-2.0 and MIT dominate; some GPL and AGPL). Instances ship as pre-built Docker containers, so evaluation needs no per-repo setup. The paper doesn’t include a public artifact URL in the text provided, so the release location isn’t stated here. The construction pipeline itself, especially the narrow-vs-broad test taxonomy and the necessary-and-sufficient spec check, is also directly borrowable if you’re building an internal eval set from your own commit history.
When a coding-agent benchmark saturates, don’t hunt for harder bugs. Widen the blast radius of the task and tighten the tests, and the same models will start failing in informative ways. The refactoring frame does this almost for free: cross-file scope is intrinsic, and behavior-preservation makes overly narrow tests obvious.
•
The dominant failure mode (incomplete file coverage) is measured on two models. It’s plausible but not proven that agents with better retrieval or repo-map tooling would close this gap without any core reasoning improvement, which would make the benchmark measure scaffolding more than model capability.
•
Instances come from post-January-2025 commits in public repos. Contamination is reduced but not eliminated, and the paper does not run a contamination probe.
•
Language coverage is uneven at the repo level. TypeScript’s 28 instances come from just 2 repos (25 from Angular alone), so per-language numbers should be read as “this model on this codebase” more than “this model on this language.”