EarlyEval halts an agent evaluation run once a lightweight classifier trained on prior agents’ trajectories predicts its final pass/fail with calibrated confidence, cutting 13–26% of execution steps and up to 44% of input tokens while shifting resolve rates by only ~1–2 pp.
You’re iterating on a coding agent. Every prompt tweak or scaffold change means re-running your eval suite, and one pass over SWE-bench Verified with a frontier model costs several hundred dollars. Longer benchmarks push past $2,000 per pass. If you run the eval a dozen times per sprint, that’s your whole compute budget going into measurement, not improvement.
The existing playbook, called Benchmark Distillation, shrinks how many tasks you evaluate on: pick a representative subset (as in Anchor Points or tinyBenchmarks) whose scores track the full suite. That leaves the per-task cost untouched. Each retained task still runs to completion, burning tokens over dozens of steps. EarlyEval cuts along the orthogonal axis: stop each task early once the outcome is obvious.
The core observation: agents often reveal their fate midway through a run. A coding agent that applies the correct one-line patch at step 23 of 45 is going to pass. one that keeps retrying the same edit against the same error is going to fail. You don’t need to watch the rest.
EarlyEval turns this into a supervised learning problem over prefixes of agent trajectories. For a given benchmark, the authors collect thousands of completed runs from many agents (a scaffold plus a base LLM), each labeled with its true pass/fail. They chop every trajectory into all its prefixes and pair each prefix with the trajectory’s final label. From each prefix they extract features in three families:
•
Behavioral: counts and rhythms of the agent’s actions so far. Edits per test, error streaks, whether tests are trending toward passing, submissions without testing.
•
Textual: TF-IDF vectors of the task prompt, action history, and environment feedback, compressed with truncated Singular Value Decomposition (SVD) to keep per-step scoring cheap.
•
Reference-solution: on benchmarks that ship gold patches, overlap between files or symbols the agent has touched and those in the gold fix.
Two separate LightGBM classifiers are trained on these prefix–label pairs: a success head that fires when a prefix looks like a winner, a failure head that fires when it looks doomed. Training them separately (rather than one joint classifier) lets positive and negative evidence accumulate independently and leaves an explicit “not sure yet” region where both heads stay quiet. Raw scores are then Platt Scaling recalibrated so thresholds mean the same thing across both heads.
At inference, at every agent step:
for k in range(1, T+1):
phi = extract_features(trajectory[:k])
p_success = success_head.predict(phi)
p_failure = failure_head.predict(phi)
if p_success >= s_threshold:
return "pass" # halt run
if p_failure >= f_threshold:
return "fail" # halt run
# otherwise keep going
return actual_final_outcome
Tree ensembles score a several-hundred-dim vector in under a millisecond on CPU, so the monitoring overhead is negligible next to a single LLM call. Crucially, the classifiers never see the held-out agent during training: the authors use a Leave-One-Agent-Out Protocol protocol so results reflect judging an agent whose behavior the predictor has never observed.
The usual way to save money on agent evals is to run fewer tasks. This paper shows the opposite is also available and stacks on top. Run every task, but stop each one as soon as a cheap classifier looking at behavioral prefixes crosses a calibrated confidence threshold; a tree ensemble over hand-designed trajectory features beats an LLM judge on the accuracy–cost curve because the LLM judge’s own inference eats the savings. The load-bearing evidence is that removing the reference-patch features on SWE-bench Verified barely hurts, which is why the same setup transfers cleanly to benchmarks with no gold solutions.
The most load-bearing result is the feature ablation. On SWE-bench Verified, dropping the entire reference-solution family (gold patch overlap, etc.) drops coverage only from 34.8% to 32.1% and step savings from 26.0% to 24.7%. This is why the method works on TerminalBench and Toolathlon, which release no gold solutions, using only behavioral and textual signals. The behavioral family is the primary driver: removing it cuts step savings to 16.4%. Within behavioral, no single subgroup is critical, signals are redundantly encoded.
Headline efficiency numbers, at the paper’s recommended operating point (~2 pp fidelity budget):
•
SWE-bench Verified (dual threshold 0.95): −26.0% steps, −32.7% input tokens, −28.7% output tokens, 95% prediction accuracy, 1.1 pp mean absolute deviation in per-agent resolve rate.
•
Toolathlon (0.90): −23.0% steps, −44.1% input tokens, −29.4% output tokens, 0.9 pp deviation.
•
TerminalBench under the stricter “no same scaffold in training” split (0.85): −12.7% steps, 0.9 pp deviation.
Leaderboard rankings survive: Spearman ρ is 0.991 on SWE-bench (16 agents), 0.994 on Toolathlon, 0.959–0.994 on TerminalBench across the two leakage splits. At worst, a few adjacently ranked agents swap by one position.
Two asymmetries worth noting. First, the failure head is more reliable than the success head across all three benchmarks, so for TerminalBench and Toolathlon the authors run failure-only for ranking. Second, an unseen scaffold hurts more than an unseen base model, because the scaffold sets the structural rhythm of the trajectory the behavioral features key on.
An architectural sweep on SWE-bench keeps the dual-head setup fixed but swaps the backbone. LightGBM dominates. A LoRA-fine-tuned Qwen-0.5B judge that reads raw trajectory text gets close on accuracy (90.7% vs 95.0%) but only saves 17.9% of steps versus 26.0%, and its per-step LLM call directly offsets the savings the framework is trying to bank.
Reach for this when you’re iterating on an agent against a benchmark that already has a pool of completed, outcome-labeled runs. You build a per-benchmark predictor once from public leaderboard submissions and baseline runs, then wrap it around your dev loop as a step-level halting rule. Every time you tweak a prompt or swap the base model, evaluation costs ~25% less, at the price of a ~1 pp systematic bias in the measured resolve rate. For relative comparisons across your own configs, that bias is largely a wash. The authors are explicit that final citable leaderboard numbers should still come from full runs.
Code and experimental data are on GitHub. The trained predictors are benchmark-specific and refresh cheaply as more runs accumulate. Directly reusable if you’re evaluating on SWE-bench Verified, TerminalBench, or Toolathlon; for a new benchmark, you need to first collect enough labeled trajectories from diverse agents to train the two heads.
Agent eval cost has two dials, not one: how many tasks you run, and how far each task runs. Cheap behavioral classifiers over trajectory prefixes can turn the second dial without giving up leaderboard fidelity.
•
Cold-start problem: EarlyEval needs a pool of completed, outcome-labeled trajectories on the target benchmark. For a brand-new benchmark with no prior runs, there is no signal to train on and the method offers nothing until submissions accumulate.
•
~1–2 pp systematic deviation in measured resolve rate. Fine for internal iteration; not fine for the number you put in a paper’s headline table. The authors recommend full runs for canonical scores.
•
Unseen scaffolds hurt more than unseen base models. If your innovation is a new scaffold, the behavioral features it produces may look unfamiliar to the predictor, and both accuracy and step savings degrade. On the stricter TerminalBench split, success-head precision dropped from 82.7% to 69.0%.