HarnessDev asks whether frontier LLMs can build and iteratively improve their own agent Agent harness from a weak seed, then measures the resulting runnable system on held-out tasks. Model-built harnesses match human references on writing and ML experimentation but trail on code and search, and evolution gains often fail to transfer.
You ship an agent product. The model weights are fixed by whoever trained them, but the wrapper around the model, the loop that calls tools, manages context, retries on failure, verifies outputs, is code you own and continuously edit. In practice that wrapper decides half the win. The paper cites a concrete example: identical GPT-5 weights solve 35.2% of Terminal-Bench tasks inside one harness and 49.6% inside another. Today humans write and maintain that wrapper. The paper asks whether an LLM can take over that job, both the initial build and the ongoing improvement.
Most prior agent benchmarks like SWE-bench hold the harness fixed and score the model. This paper flips the unit of evaluation: the submitted artifact is a runnable harness, not a task answer.
The benchmark separates three model roles that agent papers usually blur. A creator LLM builds the harness inside a development environment (Claude Code or Codex). An executor LLM later runs downstream tasks inside the frozen harness. An evaluator scores the output. Because these roles are decoupled, you can ask questions like “does the harness Opus built work when Gemini runs it.”
Two stages. Creation: the creator starts from a deliberately weak seed, a runnable skeleton that parses inputs and exposes passive file/shell primitives but has no agent loop, no verifier, no retry logic. Unmodified, it scores zero on every benchmark. The creator gets a task-family spec and 1-3 development cases, then must build the full execution system. Evolution: the creator starts from its own Creation harness and gets a budget of ten paired full evaluations (100 SWE-bench Pro tasks plus 89 Terminal-Bench tasks) plus a few small probes, and tries to improve the harness using the returned per-task feedback.
Evaluation runs in two modes. Self-Eval uses the same model as creator and executor, which measures the deployed system. Unified-Eval fixes one executor (Gemini 3.1 Pro) across every submitted harness, isolating harness quality from executor capability. After Evolution ends, every frozen version is additionally scored on 630 held-out SWE-Pro tasks the creator never saw during development.
# Creation stage, per creator model
H = seed_harness() # runnable but performs no task work
for case in dev_cases: # 1-3 cases only
trace = run(H, case, executor=creator)
H = creator.edit(H, trace) # freely rewrite files
freeze(H)
score = evaluate(H, held_out_tasks, executor=chosen_LE)
Six creator models are tested: Claude Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek-V4-Pro, Qwen 3.7 Max, and Seed 2.0 Pro. Downstream benchmarks span code (SWE-bench Pro, Terminal-Bench), ML experimentation (MLE-bench), writing (EQ-Bench3), and browsing search (BrowseComp).
The prevailing framing treats the harness as fixed scaffolding: pick one, evaluate the model inside it. This paper argues the opposite. The harness is a persistent artifact that the model itself should be able to build, diagnose, and evolve, and evaluating that capability is a different question from evaluating task solving. The most telling evidence isn’t the headline scores. It’s that the same creator produces harnesses whose quality swings sharply when you change the executor: Opus’s SWE-Pro score drops from 69.3 to 33.0 when Gemini runs its harness, revealing that its stopping rules and prompts were silently co-adapted to Opus itself.
The load-bearing finding is that harness quality does not decouple cleanly from the model that built it or the model that runs it. Under Self-Eval, Opus 4.8 leads at 67.8 average, still trailing the human reference at 86.2. But swapping executors reorders everything: Qwen gains +17.6 points on BrowseComp when Gemini runs its harness, while Opus loses 36.3 SWE-Pro points under the same swap. Portability requires that prompts, budgets, and stopping rules aren’t wired to the original model.
Secondary findings support the same picture:
•
Domain gap varies wildly. Model-built harnesses match the reference on writing and beat it on MLE-bench (Opus and Gemini hit medal rates of 32.9 and 32.4). Search and research show the largest gap; code trails but is closer.
•
Cost doesn’t buy quality. On MLE-bench, GPT-5.5 reaches medal rate 19.1 with 29.3M executor tokens, DeepSeek reaches 19.6 with 208.4M. About a 19x cost spread for similar scores.
•
Dead code is common. Of 108 code-harness components, 18 never trigger. Of 587 writing features, 124 are confirmed dead. 11 of 18 code harnesses define a State class but only one implements periodic checkpointing, and no checkpoint fires in 26,679 recorded trajectories.
•
Evolution is noisy and local. Across 64 official version switches, only 34 (53%) move visible feedback and held-out scores in the same direction. All five self-runtime creators improve on held-out tasks (Opus best at +4.44 points), but under a fixed Gemini executor, only Opus still improves. Adjacent versions on the same commit vary by roughly \u00b14.75 points, so small feedback-set gains can’t be attributed to the code change.
•
Version selection is unreliable. Only 2 of 9 declared versions were actually the held-out optimum. Creators tend to pick the version that looked best on the feedback set, which repeatedly rewards a lucky run.
Reach for this framing when you’re deciding how much to invest in agent scaffolding versus swapping in a stronger base model. The paper’s separation of creator, executor, and evaluator is a template you can copy internally: when your team lands a harness improvement, run it under a second executor before shipping. If the gain evaporates, you built a model-specific hack, not a portable capability. Same for evolution: keep a held-out task split the improvement loop never sees, because the paper shows visible-feedback gains routinely fail to transfer.
The benchmark, seed harness, audit tooling, and held-out splits are released at the project page. Downstream tasks reuse public suites (SWE-bench Pro, Terminal-Bench, MLE-bench, EQ-Bench3, BrowseComp), and the human-engineered reference systems (Claude Code, OpenHands, SWE-agent, OpenCode, and others) are pinned by commit for reproducibility. A harness that only works with the model that wrote it is a bug, not a feature. Every scaffolding improvement should survive an executor swap and a held-out task split before you call it real, because the paper shows that feedback-set gains and same-model wins routinely dissolve the moment either variable changes.
•
Evolution has one trajectory per creator-runtime cell and post-freeze held-out coverage only on SWE-Pro, so the numbers describe existence proofs, not distributions. Rerun variance is comparable to the observed gains.
•
Human references are external system-level scores pairing a human harness with its own executor, not paired controls under a common executor. “Below human” here means “below the selected mature system,” not below a fair matched comparison.
•
The development environment (Claude Code, Codex) is held fixed. Whether an evolved harness could itself serve as the development environment for the next round is left open, and that recursion is where the real “self-developing” claim would have to be tested.