Aspire asks whether an LLM can self-improve when given only a vague capability goal (e.g. “get better at math”) with no benchmark, training set, or reward exposed. Across weight and harness edits, agents close the training loop but almost never retain gains: 1 of 12 two-run averages beats the base model, and 1 of 30 adaptive-search cells produces a kept improvement.
Suppose a customer tells your team “make the assistant better at handling insurance claims.” There’s no eval set, no reward function, no labeled data. Today a human forward-deployed engineer spends weeks turning that sentence into datasets, judges, a training recipe, and a rubric, and only then does an autopilot like PostTrainBench or SEAL take over the how-to-optimize search.
Prior self-evolution benchmarks all start after that translation work is done: the task, metric, and evaluator are already specified, so the agent only searches over training recipes. Aspire removes that scaffolding and asks whether the model can do the translation itself, then measures the result on a sealed evaluation the agent never sees.
The agent receives one sentence like “improve mathematical reasoning” and nothing else. It must decide what sub-capability to target, find or synthesize training data, pick an update method, build its own validation checks, and choose when to stop. A controller-side evaluator with 520 expert-authored items across six goals scores submitted checkpoints, but the items, answers, and rubrics stay hidden. Depending on the protocol, the agent gets either no intermediate scores (final-only) or bounded aggregate scores it can use to steer search (adaptive-feedback).
A single agent-facing tool exposes typed actions: register data, launch Supervised Fine-Tuning or Group Relative Policy Optimization (GRPO) with LoRA, run agent-authored validation, branch checkpoints, stop. The controller handles job scheduling, checks every registered dataset for overlap against the hidden eval, and enforces a safe-retention rule: a trained checkpoint replaces the base only if its hidden-eval score strictly improves, otherwise the system rolls back. Aspire studies two evolution surfaces separately: updating model weights (RQ1, RQ2) or editing the agent harness around a frozen model (RQ3).
for round in campaign:
while budget_left and not agent.stop():
action = agent.decide(goal, public_history)
controller.validate_and_execute(action) # register/train/validate/branch
candidate = agent.submit_final()
score = hidden_eval(candidate) # aggregate only, or withheld
state = candidate if score > base_score else base # safe rollback
The prevailing assumption in self-evolving-agent work is that once an LLM can autonomously run training jobs and read feedback, sustained self-improvement follows. Aspire shows the opposite. Closing the training loop is not the same as closing the capability loop: agents readily produce checkpoints and harness edits, but without a human-supplied task specification they train on mismatched proxies, trust narrow self-validations, and let later search erase earlier gains. The load-bearing evidence is not a headline score but the retention ledger: of 30 adaptive-search cells, only one keeps a checkpoint above base.
The load-bearing finding is retention, not peak score. In the adaptive-feedback protocol, 28 of 30 configuration-goal cells produce an evaluated checkpoint, 21 meet eligibility rules, but only 1 (Terra on Qwen3.5-4B math, 17.86 → 20.10) keeps a checkpoint that beats the base after rollback. In the final-only protocol across 24 runs, only 1 of 12 model-goal pairs has a two-run mean above base (Qwen3.5-9B science, 45.33 → 48.00).
•
Wrong proxy, format collapse. Qwen3.5 Self configuration on 4B imports GSM8K or Hendrycks math data in 30/32 dataset events, including for science, logic, and writing goals. Five submitted checkpoints trained on numeric-label MMLU-Pro-style SFT emit only single-digit outputs on every eval item; their scores are 0, 0, 0, 6.14, 0.
•
Recovery mistaken for progress. Of 62 within-cell transitions, 28 rise, but a monotonic rising trajectory can still sit far below base, e.g. 4B math climbing 0.79 → 1.75 → 2.38 while base is 17.86.
•
More search is not better search. Sol produces 33 evaluated checkpoints and burns 76.56 training GPU-hours; none beats its base score on any of the six goals.
•
Harness evolution underperforms an engineered baseline. With weights fixed, three GPT-5.6 creators each produce one successor harness; all three score below the reference Qwen-Agent (28.64 task-macro), with the best at 27.22.
•
Goal vagueness shifts effort. On PostTrainBench, matched Claude Opus 4.8 runs under vague goals spend 2,109 more seconds on decision-model thinking and 1.27 fewer hours on active training per pair, and finish below the explicit-task reference (27.07 vs 32.90 weighted average).
Reach for Aspire when you’re building an autonomous fine-tuning or agent-improvement system and want to stress-test whether it can operate without a hand-crafted eval. The benchmark supplies six sealed goals, a controlled interactive environment with safe rollback, and audit trails that separate “produced a checkpoint” from “produced a checkpoint that beat base and survived continued search.” The failure modes it surfaces (training on a familiar proxy dataset regardless of goal, over-trusting an 8-item validation checklist, letting a later training round overwrite a good earlier checkpoint) are directly applicable diagnostics for anyone shipping an auto-training pipeline.
The project page links the artifact. The paper does not specify a license or release date for the 520 evaluation items, and the hidden set is designed to stay controller-side, so external teams likely get the environment and protocol rather than the eval items themselves.
A model that can run training jobs is not yet a model that can decide what to train on. Autonomous post-training pipelines should be judged against the base model under sealed evaluation, not against their own previous checkpoint. Local improvement curves are compatible with sitting far below where you started.
•
Every adaptive-feedback cell is a single run; the retention statistics are cell-level counts, not multi-seed estimates, so the exact 1-of-30 figure has wide uncertainty.
•
The base models tested are Qwen3.5-4B and 9B; a stronger base (or a decision model promoted across rounds, which no reported run does) could change whether “closing the capability loop” is achievable.
•
The six goals lean on knowledge and reasoning; goals where a proxy dataset is genuinely close to the hidden eval (unlike GSM8K standing in for writing) might show a very different retention rate.