CheckerBench asks coding agents to synthesize working static-analysis checkers for real CVEs across 300 tasks and five languages, with independent rebuild-and-rescan verification; the best model-harness combo passes only 45.33% and the mean is 32.30%.
Static analyzers like Clang Static Analyzer and CodeQL catch bugs without running code, but each new defect pattern needs a hand-written checker: a rule, written in the analyzer’s own extension API, that encodes what the bug looks like and how to detect it across a whole repository. Writing one is expert work. You read the vulnerable code, decide what analysis is needed (syntactic match, path-sensitive, dataflow, cross-procedure), code it against the analyzer’s framework, register it in the build, and iterate until it fires on buggy code and stays quiet on fixed code.
The natural question: can a coding agent do this end-to-end? Existing benchmarks don’t answer it. SWE-bench and its successors score patches to fix issues. Vulnerability benchmarks like ReposVul score detection or classification. The closest prior work, KNighter, synthesizes Clang Static Analyzer checkers from Linux kernel patches but is tied to one analyzer and one harness. Nothing measures whether a general-purpose agent can produce a reusable checker that an independent party can rebuild, run on both the buggy and fixed revision, and trust.
The benchmark has two pieces. CheckerBench is 300 tasks drawn from 297 real CVE records across 167 repositories, 85 Common Weakness Enumeration categories, and five languages (C/C++ via Clang Static Analyzer, plus Java, Python, JavaScript, and Go via CodeQL). Each task hands the agent a pinned repo environment, the vulnerable and fixed revisions, the fixing patch, a checker scaffold, and a shared skill library (analyzer docs, example checkers, utility functions, but no task-specific solution). The agent gets a 50-minute budget to inspect code, implement a checker, compile it, scan both revisions, and refine based on feedback.
CheckerLab is the evaluation harness. Crucially, it throws away whatever the agent built in its workspace and rebuilds the submitted checker source from scratch in a clean environment. It then runs a frozen verifier that checks several non-compensating gates: the checker compiles, it fires on the vulnerable revision in a patch-modified function, it goes quiet on the fixed revision, residual noise stays bounded, and a broader scan over mainline code shows at most 25% false positives on sampled alerts.
serves as judge for process quality, semantic fidelity, and false-positive triage, with human validation giving Cohen’s kappa of 0.85.
The rollout loop looks roughly like this:
for attempt in range(budget_steps):
plan = agent.analyze(patch, vuln_code, fixed_code)
checker_src = agent.write_checker(plan, skills)
build_ok, diag_vuln, diag_fixed = run_analyzer(checker_src)
if build_ok and diag_vuln > diag_fixed and in_patch(diag_vuln):
agent.submit(checker_src); break
agent.observe(build_ok, diag_vuln, diag_fixed)
# verifier later rebuilds checker_src in a clean workspace
Across 21 model-harness configurations with three repeats each, mean Pass@1 is 32.30% and the top configuration (Claude Opus 4.8 with OpenCode) hits 45.33%. More than half of tasks defeat even the best setup.
The gap between gates is informative. Overall Build rate is 78.25% and differential detection (fires on buggy, quiet on fixed) is 73.65%, but full acceptance is 32.30%. The 41.35-point drop from differential detection to Pass@1 comes from semantic fidelity, false-positive control, and process quality, not from compilation failure.
Harness choice reshuffles rankings. OpenCode averages 36.95% Pass@1, Claude Code 33.81%, Hermes 26.14%. The leading model changes per harness: Claude Opus 4.8 wins on OpenCode and Hermes; DeepSeek-V4-Pro wins on Claude Code. Kimi K3 drops from 38.67% to 24.33% depending on harness.
C/C++ tasks (analyzed by Clang Static Analyzer) are hardest for everyone; the top model there reaches only 24.53%, while Python with CodeQL goes above 65%. The ablation on DeepSeek-V4-Pro shows Pass@1 falls from 39.67% to 8.33% without the skill library, to 13.33% without execution feedback, and to 0.33% for one-shot generation with no tools. The authors caution that execution-condition differences prevent clean causal attribution.
Against KNighter on its own 61 Linux-kernel tasks (both using DeepSeek-V4-Pro), the paper’s skill-guided OpenCode method produces valid checkers on 80.3% vs KNighter’s 67.2%; removing skills collapses it to 29.5%.
If you’re evaluating a coding agent for security engineering, don’t infer checker-writing ability from SWE-bench scores. The gates these results expose (semantic adequacy, false-positive control, analyzer-API fluency) aren’t exercised by issue-fix benchmarks. Worth running your candidate configuration on CheckerBench’s smaller language slices before trusting it on your own analyzer rules.
Treat the complete model-plus-harness pair as the unit of selection. Picking a strong model and dropping it into whatever harness you have handy will leave double-digit Pass@1 on the table, and the best harness depends on the model.
If you’re building an agent system for this kind of long-horizon synthesis, the ablation suggests budget is better spent on a curated skill library and surfacing real compiler/analyzer output than on prompt tweaks. The one-shot baseline is nearly zero.
For efficiency, Kimi-K3 under Claude Code matches DeepSeek-V4-Pro’s Pass@1 with roughly 38% fewer tool calls, worth testing when inference cost matters.
All results are conditional on a 50-minute budget; the final 10 minutes add only 2.25 points to aggregate Pass@1, but shorter deadlines were not separately tested. The judge for process, semantic, and false-positive stages is a single model (Claude Opus 5), so systematic judge bias cannot be ruled out despite high human agreement. The Linux kernel contributes 107 of 300 tasks, so aggregate numbers over-weight one codebase; the paper reports repository-equal averages separately. The contamination audit found only small Pass@1 shifts (2-3 points) when identifiers were paraphrased, but the audit covers two models on 100 tasks, not the full matrix. Finally, mainline false-positive rates use sampled alerts (at most ten per checker), not exhaustive adjudication; re-ranking under the all-alert gate changes the leading harness-model pair in an archived cohort.