Doc2Harness measures how well an LLM can build the surrounding agent harness itself (the loop, tools, permissions, session state) by giving it a spec plus an interface-only skeleton, then driving the finished harness with a scripted mock model and checking four observable channels. Even the top model fails nearly half of 328 test cases.
When you ship a coding agent, the model is only part of what ships. The Agent harness around it decides which tools exist, when they run, what the agent is allowed to touch, how session history is compacted, and how failures are recovered. Vendors like Anthropic and OpenAI pair their models with their own harnesses (Claude Code, Codex CLI), and swapping harnesses can shift scores by double-digit points. So “how good is this model at agent work” is really “how good is this model-plus-harness.”
Existing coding benchmarks don’t test harness-building. SWE-bench and its successors give the model a mature repo and ask it to fix an issue, judged by the project’s own unit tests. That doesn’t work for a harness, because (a) the same feature can be built many legitimate ways, and (b) a harness doesn’t finish a task on its own, it needs a model driving it to show behavior. The authors also want to study whether models can eventually improve their own harness, a prerequisite for recursive self-improvement.
The benchmark has two stages, each holding one piece fixed to avoid confounding who gets credit for what.
Stage 1, construction. The model under evaluation is dropped inside a fixed builder harness (OpenCode) and given two things: a development document (a de-duplicated spec distilled from the user docs of twelve real harnesses like Claude Code, Gemini CLI, Aider, Cursor), and a contract skeleton (a stripped-down Mini-SWE-Agent of roughly 70 lines that declares three interfaces: a CLI, a headless entry point, and an API for automated evaluation). The model works in a fresh container with no network except the model gateway, has up to 24 turns and two wall-clock hours, and must explicitly submit. A contract check verifies the submission compiles and exposes the required interfaces but does not run behavior tests. Pass the check and the harness is frozen forever. A typical finished harness is ~30 files and ~16,000 lines, of which ~99.6% was written by the build model.
Stage 2, evaluation. The frozen harness is launched in a clean container, one per test case. A mock model feeds it canned responses from a script. There is no real LLM in the loop here, which makes scoring deterministic. The grader watches four channels: requests the harness sends to the model, files on disk, process exit code, and command output.
Positive controls. A test case might pass for the wrong reason. Example: you check that a symlinked file outside the workspace does not leak into the model’s requests. That could mean the harness correctly blocked the escape, or it could mean its read tool is just broken and reads nothing. The authors call this a Vacuous pass and gate 130 of the 328 cases with controls that confirm the observation would have been visible if the behavior occurred. A failing control forces the gated case to FAIL regardless.
# Stage 2 per test case
spin_up_container(fixture)
status = run(candidate_harness, mock_model=script)
for ctrl in positive_controls_for(test_case):
if run(candidate_harness, mock_model=ctrl.script) != PASS:
return FAIL # observation precondition not met
return status # PASS only if assertion holds and controls hold
Test cases come from three sources used in rotation: mining public GitHub issues on four real harnesses (21,083 issues ranked and reviewed top-down), comparing a candidate harness module-by-module against a mature one like Kimi Code, and single-variable expansion of known-discriminating seeds (varying the tool used, the input form, the erroneous parameter, or where the result surfaces).
Nineteen frontier models, three independent builds each, 328 test cases. Results reported as pass rate.
•
Not saturated, still discriminating. Top model GPT-6 Astra reaches 53.05%, weakest (Hy3) reaches 27.44%, a gap of roughly 26 points. Even the best model fails about half the suite.
•
Where the top two diverge. On ordinary scenarios Opus 5 leads (80.68% vs 72.46%), but on boundary scenarios GPT-6 Astra leads (48.97% vs 37.02%). The headline gap is really about edge cases.
•
Category rank is unstable. A model’s rank across the six main categories can swing by up to 16 positions. Grok 4.7 is first on file operations but eleventh on session state. Despite this, all 15 pairwise category correlations are positive (mean +0.521), and overall KR-20 coefficient internal consistency is 0.913, consistent with roughly one underlying capability plus category-specific noise.
•
Architecture matters, and model identity sets it. Mean code size per model ranges from ~1,300 to ~16,700 lines, and 95.9% of the variance in code length is between models, not between a single model’s three builds. Code length correlates +0.614 with pass rate. More importantly, after controlling for code length, module count has partial correlation -0.423 with pass rate and lines-per-module has +0.504. The authors’ reading: splitting cross-cutting functionality across many small modules hurts. Treat this as an observed association across 57 builds, not a proven causal rule.
•
If you’re picking a model to build or evolve agent infrastructure, the ranking here is more relevant than ordinary SWE-bench scores, because it stresses generalization to software (harnesses) that barely existed during pretraining. The strong-weak gap is wide enough to matter.
•
If you’re evaluating an agent system you built, the positive-control idea travels. Any assertion of the form “forbidden thing did not appear” should be paired with a check that the observation channel is actually live. This is cheap to add and catches a class of silent false passes.
•
If you’re designing a harness yourself, the architecture finding is worth testing on your own codebase: at matched code size, higher fragmentation across modules associated with lower pass rate. Worth checking before you refactor a working harness into many small files.
•
Limits on what this tells you. The protocol scores a single build with up to three submissions and no behavioral repair loop, so it does not measure iterative self-improvement, only one-shot construction. A model that would do well with evaluate-and-fix feedback could look worse here than it would in production.
Project page: geyuyao.com/doc2harness. The paper doesn’t specify a code or data release beyond that.
•
The contract check is structural, not behavioral, so models that submit early are rated on an under-revised harness. Pass rates are a lower bound on what the model could achieve with more iteration.
•
Single-variable expansion produces correlated test cases from one seed, so a single implementation defect can cascade into several failed cases and inflate apparent severity.
•
The builder harness is fixed to OpenCode specifically to avoid giving Claude or GPT families a home-field advantage from their vendor harnesses. Results are conditional on that choice.
•
The architecture finding (fewer modules, higher score) is a partial correlation across 57 builds, not an intervention. The paper does not show that forcing a model to consolidate modules would raise its score.
•
Model lineup includes versions dated 2026 that this reader cannot independently verify; take absolute rankings as the paper’s measurements rather than established facts about those models.