This paper argues that ranking LLM research agents by a single final score hides where they actually succeed or fail, and shows through a three-part process decomposition on 36 long-horizon AutoLab tasks that reliability, not peak skill, separates today’s frontier agents, with a gap of 0.237 on avg@3 vs. only 0.122 on best@3 between best and worst models.
You’ve shipped a coding agent that spends 4 hours iterating on a customer’s optimization task, running tests, editing code, and committing progress. Sometimes it lands a great solution; sometimes it regresses at hour 3 and never recovers. Today you probably evaluate it by looking at final task scores across a benchmark suite and picking the model with the highest average. That tells you almost nothing about why runs fail, whether the agent is learning from earlier attempts, or how much of the observed performance comes from the surrounding scaffolding vs. the model itself. This paper is a large-scale audit of exactly that question, positioned against benchmarks like MLE-bench and RE-Bench that report final scores only.
The authors run seven frontier models (Claude Opus 4.7, GPT-5.5, Gemini 3.1 Pro, GLM-5.2, Kimi K2.7-Code, DeepSeek V4-Pro, LongCat-2.0) three times each on 36 tasks under a shared Claude Code harness, then instrument the trajectories with three deterministic, verifier-derived scores. Solution Framing (C1) asks how quickly the agent’s running-best verifier score climbs and whether early discoveries survive. Execution (C2) asks whether committed changes actually build and pass, discounted by pre-commit build failures. Feedback Control (C3) asks whether peak scores are preserved to the end and, when regressions happen, how much lost score is recovered and how fast. On top of the within-run view, they run counterfactual experiments for experience reuse: pick a midpoint in a trajectory, then either continue with accumulated context or wipe the agent’s memory (chat history, notes, code comments) and continue from the same code state. The delta on the next commit measures how much the model’s proposal depended on accumulated experience.
for model, task in product(models, tasks):
for seed in range(3):
traj = run_agent(model, task, harness="claude-code")
c1 = framing_score(traj.running_best_over_time)
c2 = execution_score(traj.commits, traj.build_logs)
c3 = feedback_score(traj.peak, traj.final, traj.dips)
# counterfactual: erase memory at midpoint, compare next commit
delta_intra = score(with_memory) - score(erased_memory)
A separate inter-task variant has each model extract a lessons.md from a solved source task and apply it to held-out targets in the same category.
The prevailing way to compare agent-capable models is a leaderboard of final scores on a fixed benchmark. This paper shows the opposite. A single score conflates distinct capabilities that require different fixes: two models with identical outcomes can have opposite Execution and Feedback Control profiles, and a model’s ranking can flip once you let it accumulate experience across tasks. The load-bearing evidence isn’t the headline table. It’s the case where GPT-5.5 and Gemini 3.1 Pro post nearly identical final scores and identical C1, yet GPT is stronger at Execution while Gemini is stronger at Feedback Control.
•
Reliability, not peak skill, is what separates today’s frontier models. The best-to-worst spread is 0.237 on avg@3 but only 0.122 on best@3. Lower-ranked models can reach competitive solutions; they just don’t do so consistently.
•
Similar final scores hide different bottlenecks. GPT-5.5 scores 0.958 on Execution but 0.858 on Feedback Control; Gemini 3.1 Pro is 0.889 and 0.920 on the same dimensions. LongCat ranks sixth overall but posts the highest Feedback Control score (0.928).
•
Task category matters. CUDA tasks bottleneck on Solution Framing and Execution; model-development tasks execute cleanly but struggle to preserve progress.
•
Experience is a real lever but unstable. Within a task, retaining memory helps almost every model; the largest intra-task gain is +0.145 for LongCat. Across tasks, transferred lessons raise DeepSeek’s avg@3 by +0.093 but drop Gemini’s by 0.017, flipping the ranking. Explicitly extracted lessons outperform handing over the raw source workspace, and self-generated lessons beat cross-model lessons even when the producer is stronger.
•
Harness choice mostly moves stability, not ceiling. Native harnesses (Codex CLI for GPT, Kimi Code CLI for Kimi) and OpenCode shift avg@3 by up to +0.055 vs. Claude Code but leave best@3 within 0.035 and preserve model ordering.
•
Genuine novelty is rare. Of 252 best-of-three solutions, only 3 (1.2%) qualify as novel approaches after manual review, while 16 (6.3%) exploit evaluator shortcuts. The bulk (44%) stack established techniques.
•
A four-round automated harness search on three seed tasks lifted avg@3 by +0.12 on the seeds and +0.06 on held-out System Optimization tasks, but did not generalize to other task families.
Reach for this framework when you’re evaluating a long-horizon coding or research agent and the final-score leaderboard isn’t telling you which model to ship. Instead of averaging pass rates, log the running-best verifier score, build outcomes, and every regression-and-recovery episode from each run. The three C1/C2/C3 formulas are deterministic and don’t need an LLM judge, so you can compute them from CI logs and Git history. If your agent is bottlenecked on Execution, generic code-training helps; if it’s bottlenecked on Feedback Control, invest in memory and rollback tooling instead.
The paper builds on the AutoLab task suite (36 curated tasks with verifiers and starting artifacts) and doesn’t appear to release its own code or metric implementations, at least not linked in the text provided. If you want to reproduce the setup, you’d start from AutoLab plus one of the named harnesses (Claude Code, Codex CLI, OpenCode).
When you evaluate a long-horizon agent, measure the loop, not the leaderboard. Reliability across seeds, ability to preserve peak progress, and whether the harness or the model is doing the stabilizing are separately fixable, and the final score obscures all three.
•
The C1/C2/C3 metrics are proxies computed from verifier signals. They can’t score the semantic quality of an unrealized idea, and C3 gives high marks to short, monotone runs that never actually got a chance to recover from anything.
•
All main comparisons run under a single harness on one task suite. The harness ablation covers only three models and three scaffolds; conclusions about “harness doesn’t change rankings” may not survive different prompt or tool designs.
•
Novelty judgments come from an LLM judge (Opus 4.8) with manual review over one rubric. The paper explicitly scopes its “novelty is rare” claim to AI-for-AI optimization and warns it may not extend to open-ended scientific discovery.