Taste-Bench measures an agent’s taste, the ability to pick the better branch at a mid-run decision fork before seeing what happens next, by mining forks from existing trajectories and labeling them with the recorded downstream outcome. The best frontier model reaches only 59.7% accuracy.
Suppose you’re running a coding agent on a repository task. Halfway through, it picks a refactoring approach that looks reasonable. Twenty steps later, tests fail in ways that trace back to that choice. You’ve burned the budget. This is the pain point the paper targets: long-horizon agents make many mid-run judgments whose cost only shows up later.
Existing agent benchmarks like SWE-bench and RE-Bench score whether the agent finished the task, not whether it made good calls along the way. Two agents can both fail a task while making very different quality decisions, and end-to-end success hides that. The authors argue that judgment quality itself, which they call taste, is a distinct capability that deserves its own measurement.
The core trick is hindsight. If two agent attempts share the same task and the same trajectory prefix, then diverge into different actions, and one attempt succeeds while the other fails, the successful branch’s action was probably the better call at that fork. The paper calls this divergence point a decision fork, and the two continuations branches. Freeze the trajectory at the fork, hide everything after, and ask a model to pick between the two directions. The recorded outcome tells you which was right.
They mine forks two ways. Parallel forks come from two independent attempts at the same task that diverge midway. Detour forks come from a single trajectory where the agent commits to an approach, hits a failure, and recovers with a different approach. The abandoned direction and the recovery become the two candidates.
A generator model (GPT-5.6 Sol at high reasoning) proposes candidate forks and enforces a rubric: the deciding evidence must appear after the fork, both candidates must be neutrally worded, the failure must be a real observed error, and so on. Then a panel of four judge models filters out (a) trivial questions solvable from the candidate wording alone and (b) undecidable questions where judges disagree even after seeing the full outcome.
for traj_pair in mine_parallel_and_detour(trajectory_pool):
fork = generator.propose_fork(traj_pair) # locate divergence point
if not fork.passes_rubric(): continue
q = {task, prefix_before_fork, cand_A, cand_B, label_from_outcome}
if all(judge.answers_correctly(q, hide_traj=True) for judge in panel):
continue # trivial, drop
if not all(judge.picks_label(q, show_full_outcome=True) for judge in panel):
continue # undecidable, drop
release(q)
Each question is scored in both candidate orders. It counts as correct only when the model answers both orderings correctly, which controls for position bias. Random guessing therefore scores 25%, not 50%.
Frontier models struggle. Across 14 evaluated models including Claude Opus 5, GPT-5.6 Sol, Grok 4.5, and DeepSeek V4, the top score is 59.7% (GPT-5.6 Sol) on a 502-question benchmark split across parallel/detour construction and engineering/research domains. Most models sit well below that.
Detour forks are harder than parallel forks in both domains. The construction gap exceeds the engineering-vs-research gap. Detour engineering, where the model must catch a mistake before the acting agent did, is the hardest cell.
Time horizon crushes accuracy. The authors label each fork with how far past the fork you’d have to look to justify the right answer: in prefix, inferable, next step, or more work. Mean accuracy across the 14 models falls from 62.3% on in-prefix forks to 21.0% on more-work forks, near the 25% random-guess floor.
More reasoning does not help. Rerunning two models across three reasoning-effort settings changed accuracy by −0.2 and +2.2 points. The models spent the most reasoning tokens on the hardest (more-work) level and still failed there, which the authors read as: the deciding evidence genuinely isn’t in the prefix.
Taste is only partly correlated with end-to-end ability. Comparing model rankings against public SWE-bench Verified scores gives Pearson r = +0.63 overall, but only r = +0.37 on the engineering subset that was actually mined from SWE-bench Pro tasks. So this isn’t a restatement of coding benchmark rank.
Taste is trainable via distillation. Using Qwen3.6-27B with LoRA adapters, they distill a teacher that sees a hint pointing to the supported candidate into a student that doesn’t. The training objective is an SDPO-style token-level forward KL over the teacher’s reasoning and final choice. On held-out engineering questions the student jumps from 30.0% to 47.9% accuracy. When the student’s judgment is written as advice into the prompt of a fixed executor agent on 41 held-out SWE-bench Pro tasks, task success rises from 14.6% to 33.7%, most of the 39.0% ceiling that perfect advice provides.
•
If you build long-horizon agents, taste is a distinct axis worth tracking. A model can be near the top of SWE-bench Verified and mediocre at Taste-Bench, or vice versa. Two of eleven models swap ranks substantially between the two. If you rely on end-to-end scores to pick a model for agentic work, you’re likely missing this signal. The dataset and code are released.
•
Don’t expect more reasoning tokens to fix hard mid-run decisions. The evidence here says that at high time horizons, extra thinking doesn’t recover the missing future information. If your agent keeps making costly early mistakes, the fix is probably not raising reasoning effort. Consider instead: giving the agent access to prior trajectories on similar tasks, or training a small advisor.
•
The distillation-plus-advice pattern is worth testing if you have logs of prior agent runs on tasks your users repeat. The pipeline needs (a) multiple attempts per task with recorded outcomes, and (b) tasks similar enough that a fork identified once generalizes. The 33.7% number is on tasks the pipeline had already mined forks from, so “prior runs on this exact task” is a prerequisite, not a claim about cold-start generalization to brand-new tasks.
•
The mined labels are trustworthy enough to build on. Human review found 98.8% agreement with the mined labels when reviewers had access to the recorded outcomes, and κ = 0.973 between reviewers.
The training and end-to-end evaluation are confined to software engineering (SWE-bench Pro). The research-domain trajectories, drawn from METR’s public MALT release, are used only for the benchmark, not for the distillation or the executor test.
The end-to-end 33.7% number applies specifically when the pipeline has already mined forks from prior runs on that task and can write task-specific advice before the new run starts. It’s not a claim about general capability transfer.
One generator model (GPT-5.6 Sol) writes all questions, and one judge model (GPT-5.5) assigns time-horizon levels. Systematic biases from either would propagate into the benchmark. The judge panel for filtering does exclude the generator.
The SWE-bench Verified correlation is computed on only 11 models, and the paper reports a wide 95% interval of [+0.04, +0.89] for the overall correlation, so the “only partly correlated” claim is directionally supported rather than tightly established.