ActiveSaddler improves Automated Harness Optimization by treating training-scenario selection as a non-stationary bandit: it groups failures into failure-pattern arms and reallocates optimization budget as the harness evolves, lifting test Pass@1 by 4.4 and 7.5 pp over fixed-order baselines on two agent benchmarks.
Suppose you’re tuning an LLM agent that uses tools, retries, and prompts (its harness) to complete multi-step tasks. A tool like AutoSaddler runs the agent on training scenarios, inspects failures, patches the harness, and iterates. These optimizers obsess over how to patch, but the order of training scenarios is usually fixed upfront, either the whole training set or a pre-scheduled mini-batch stream.
That’s wasteful under a limited rollout budget. Some failures get patched after one visit and don’t need revisiting. Others stay broken but the schedule moves on. And whether you should re-try a known failure versus run a brand-new scenario (that might expose a new bug class) also shifts over time. The paper argues this missing dimension, deciding which scenarios feed the next update, deserves its own adaptive policy, borrowing from Automated Curriculum Learning.
The core idea: don’t tie the curriculum to individual scenarios or fixed difficulty categories. Tie it to recurring weaknesses in the harness. If three different scenarios all fail because the agent mishandles ambiguous contact names, those failures feed one shared optimization target called a failure-pattern arm. Patches aimed at that arm then get credit (or blame) across all three scenarios.
Three LLM-driven components wrap the existing optimizer:
•
Failure-Pattern Extractor: after each iteration, abstract each failed trace into a symptom, then normalize against the current arm list, either matching an existing arm or creating a new one.
•
Arm Prioritizer: for each arm, an LLM scores estimated learning progress along four axes (is the bug still active, is it fixable via harness changes, how broadly would a fix help, what’s the regression risk). Scores feed a softmax that samples the next arm, so high-value arms are favored but low-ranked ones can still be picked.
•
Exploration Controller: another LLM call decides each iteration between “pull” (revisit a known arm) and “draw” (sample unseen scenarios to discover new failure patterns).
Pseudo-code of the outer loop:
while budget_remains:
if controller.decide(H, arms, unseen, history) == "draw":
X = sample(unseen)
else:
scores = {a: llm_score(a, history) for a in arms}
a = sample_softmax(scores, tau=0.15)
X = sample(a.supporting_scenarios)
traces = execute(H, X)
H = optimizer.update(H, traces, D_dev) # AutoSaddler or GEPA
arms = extractor.update(arms, failures(traces))
The harness optimizer itself (they use AutoSaddler, also tested with GEPA) is unchanged. ActiveSaddler only chooses what it sees.
Two benchmarks: GAIA2 (general assistant tasks in a simulated phone environment, split by persona so test personas are unseen) and Terminal-Bench 2.0 (realistic terminal tasks across sysadmin, ML, security). Task agent and optimizer both use gpt-5.5. They report mean Pass@1 over three test runs.
•
Headline: ActiveSaddler on top of AutoSaddler hits 59.8% on GAIA2 and 80.0% on Terminal-Bench 2.0, versus AutoSaddler with a randomly shuffled fixed scenario order at 55.4% and 72.5%. That’s the +4.4 and +7.5 pp gap. Difficulty-ordered curricula (easy-to-hard by category or by scenario) do no better than random order.
•
Ablations isolate each component. Replacing failure-pattern arms with category-level arms drops test Pass@1 by 3.0 pp (GAIA2) / 10.8 pp (TB2); using scenario-level arms drops it by 3.6 / 6.7 pp. Removing the LLM arm prioritizer (uniform scores) costs 4.5 / 7.5 pp. Replacing adaptive exploration with a fixed every-5-iterations schedule costs 4.5 / 8.3 pp. Each piece matters.
•
Mechanism evidence: with failure-pattern arms, 52.9% (GAIA2) / 60.4% (TB2) of sampled iterations hit a scenario that still has an unresolved failure, far above the category and scenario-arm alternatives. On TB2, ActiveSaddler has 6.9× as many unresolved arms when choosing Pull as when choosing Draw, versus 1.3× for the fixed schedule, suggesting the LLM controller really does respond to remaining repair opportunities.
•
Transfers to another optimizer: swapping AutoSaddler for GEPA, ActiveSaddler still adds +3.0 pp on GAIA2.
•
Non-LLM baselines: an Exponential moving average failure-persistence scorer and a count-based UCB-AIR controller lag ActiveSaddler by 3.1 and 4.2 pp respectively. Simple numerical signals help, but LLM judgment of the four factors helps more.
•
Cost: curriculum calls add real optimizer overhead (roughly $11 vs $7 per generated patch), but because rollouts are allocated better, ActiveSaddler reaches 58.5% dev accuracy on GAIA2 at $298 while AutoSaddler needs $1,360.
The paper is careful to call these correlational ablations on whole-system swaps, not causal isolation of a single mechanism in isolation.
•
If you’re running an offline agent-harness optimizer (AutoSaddler, GEPA, Meta-Harness, or similar) with a bounded rollout budget, the paper’s evidence suggests the curriculum layer is a cheap wrapper that pays for itself. The optimizer-side cost rises, but task-agent rollouts (usually the expensive part) get spent on scenarios more likely to produce useful patches.
•
The failure-pattern-arm abstraction is the load-bearing idea. If you’re building your own harness optimizer, grouping failed traces by inferred weakness, rather than by scenario ID or hand-defined category, is worth testing. The ablation shows category arms hide live bugs behind sibling successes, and scenario arms fragment the same bug across too many targets to revisit under budget.
•
The LLM-scoring recipe is specific and copyable: four axes (severity, fixability, breadth, side-effect risk), rescored every iteration from current history, feeding a temperature-0.15 softmax. The prompts are in the appendix. Worth testing against a simpler EMA baseline on your own workload before committing.
•
Caveat on scope: evidence is from two benchmarks with LLM-graded or test-based pass signals and out-of-distribution splits by persona (GAIA2) or random split (TB2, 89 tasks). The paper does not claim this generalizes to online or test-time adaptation, model-weight tuning, or production deployment with safety constraints.
•
The project page and code are promised at aka.ms/ActiveSaddler-website; the paper also references an anonymous repo for the extraction skill files. •
Every curriculum decision (extract, score, explore) is an LLM call with reasoning effort set high. That’s extra latency and dollars; the net win depends on task-agent rollouts being expensive relative to optimizer calls, which held here but may not in cheaper domains.
•
Only two benchmarks, one base optimizer in the main table (AutoSaddler), with a single additional transfer check to GEPA. Robustness across more optimizers and task families is untested.
•
Failure-pattern arms are induced by LLMs from traces. Quality of the arm pool depends on the extractor’s ability to abstract and normalize symptoms consistently. The paper gives case studies and lists discovered patterns but does not quantify extraction error rates.
•
The reported numbers are mean Pass@1 over three test executions with modest standard deviations (e.g., ±1.0 to ±2.9 pp). Differences near the small end of the ablation deltas should be read as suggestive, not decisive.
•
The authors explicitly note that improved benchmark Pass@1 is not evidence of safe deployment. Harness patches can change tool access and control flow, so production use needs its own safety and security review beyond task-level objectives.