A controlled 132-run benchmark compares ten self-distillation methods against plain Group Relative Policy Optimization (GRPO) for training language-model agents, finding that no method consistently beats the baseline across tasks and model sizes, that strong final scores often hide mid-training collapses, and that extra compute rarely buys better agents.
Suppose you’re training an LLM agent to shop on a website or run a text adventure. You use RL with a sparse reward (did it finish the task?), but that single bit of feedback per episode is a thin learning signal. A popular fix is on-policy self-distillation: during training only, give a “teacher” copy of the model extra context the student doesn’t see (a skill hint, a future observation, a verified success trajectory), and use the teacher’s token probabilities to densify supervision. Over roughly 90 days, the authors counted about one new paper every two days proposing variants of this idea.
The practical problem: these papers rarely compare against each other under matched conditions. Each one typically reports gains over a reused baseline score from an earlier paper, under different training setups, with no repeated-seed evaluation and no compute accounting. If you’re deciding which method to actually adopt, the published record doesn’t tell you.
The baseline here is Group Relative Policy Optimization (GRPO), the outcome-only RL recipe from DeepSeekMath that most of these self-distillation methods extend. The authors deliberately chose to run an independent evaluation instead of proposing yet another method.
This is a benchmarking paper, so the “mechanism” is the evaluation protocol itself. The authors fix everything they can and vary only the training method.
They train Qwen3.5 at 2B, 4B, and 9B on four agent benchmarks: ALFWorld, ScienceWorld, Search-QA (a retrieval-QA mix spanning NQ, TriviaQA, HotpotQA and others), and WebShop. That’s 12 model-task settings. For each, they run GRPO plus ten self-distillation methods (132 training runs total), each for 100 global steps, with validation every 5 steps and final evaluation averaged over 8 rollout seeds. Total cost: 12,124.4 H800 GPU-hours.
The ten methods fall into four families of how teacher supervision enters the loss:
•
Auxiliary losses on student tokens weighted by teacher scores (GRPO+OPSD, SDAR, Skill-SD, ICSD, PCSD).
•
Advantage reweighting using teacher-student log-prob gaps (RLSD, OCSD, AgentOPSD).
•
Reward shaping from teacher alignment (ADRS).
•
Verified teacher continuations appended to the batch (OVCSD).
Shared infrastructure, same prompts, same reward definitions, same evaluation sets. Method-specific hyperparameters follow each source paper’s released configs. Teachers get privileged context (skills, future observations, or verified success suffixes) only during training; at eval everyone runs the same.
The evaluation looks at three things most papers skip:
for method in [GRPO, GRPO+OPSD, SDAR, ..., PCSD]:
for setting in 12_model_task_combos:
trajectory = train(method, setting, steps=100, validate_every=5)
final_scores = [eval(seed) for seed in range(8)]
record(final_mean, final_sd,
auc(trajectory), # sustained progress
peak_minus_final(trajectory), # late regression
wall_clock_gpu_hours) # cost
No method wins across the board. Averaging each benchmark’s own metric with equal weight across the 12 settings, only PCSD (+4.58 points) and AgentOPSD (+1.66 points) beat GRPO’s mean of 49.68. The other eight fall below GRPO. Even PCSD leads no single setting outright and beats GRPO in only 6 of 12.
Win counts hide catastrophes. ICSD wins 7 of 12 settings but averages below GRPO, because it scores 0.00% success on 4B WebShop (a complete collapse). GRPO+OPSD similarly wins 6 settings but hits 1.52% on 4B ALFWorld. A simple win-rate tally would call these methods competitive.
Final score and training stability are different things. OCSD leads by validation-curve AUC (sustained performance throughout training) but ranks 4th at the endpoint. AgentOPSD is 2nd at the endpoint but 9th by AUC, below GRPO. On 4B ALFWorld, AgentOPSD hit 60% at step 35, then scored under 4.3% for six consecutive validations (steps 45-70), before recovering to 79.3% at step 100. The endpoint looks fine; the training was a near-disaster. Separately, 9 of 132 runs ended at least 20 points below their observed peak, meaning they learned something and then lost it.
A Search-QA decoding wrinkle. 2B OVCSD reaches 38% greedy accuracy at step 70 and collapses to 1.34% greedy accuracy by step 100, yet the same final checkpoint scores 33.21% under sampled evaluation (T=0.4). Greedy decoding collapsed; sampling masked it. The failure mode you see depends on how you evaluate.
Compute rarely pays off. Of 120 method-vs-GRPO comparisons, 112 required more training time; 58 of those also finished lower than GRPO. PCSD and AgentOPSD, the two aggregate winners, cost ~1.17× and ~1.19× GRPO’s training time. OVCSD runs 1.37× longer with no aggregate gain. When the authors re-score each method at GRPO’s wall-clock budget rather than at step 100, AgentOPSD’s ALFWorld advantage (68.1% → 37.6%) and PCSD’s WebShop advantage both evaporate.
•
If you’re picking a self-distillation method to build on, the paper suggests GRPO is still a reasonable default and that PCSD and AgentOPSD are the only candidates with a positive aggregate signal. Even then, their advantages are setting-dependent and partially disappear under matched compute, so test on your actual task before committing.
•
If you’re evaluating any agent-training method, treat validation-curve AUC, peak-to-final drop, and compute-matched comparisons as first-class metrics alongside the endpoint score. The paper documents concrete cases where endpoint-only reporting would have hidden collapses lasting dozens of training steps. Code and configs are released at GitHub.
•
If you’re reviewing or reading this literature, the paper’s Table 1 is a reality check: of the ten methods surveyed, only one reported repeated evaluation, half investigated training failures, and three measured cost. Reused baseline scores from earlier papers are the norm, not the exception.
•
Worth testing but not established: whether PCSD’s and AgentOPSD’s aggregate edge holds on backbones other than Qwen3.5, on longer training horizons, or with task batches outside the four benchmarks studied.
The authors trained for only 100 global steps per run, which is short; some methods might separate more clearly with longer training, and some collapses might also appear later. Only Qwen3.5 backbones were tested, so model-family-specific effects can’t be ruled out. The ten methods were implemented following their source papers and released code, but subtle implementation choices (e.g., Skill-SD and RLSD using SDAR’s skill-bank adaptation) could affect individual rankings. Finally, rollout-seed standard deviations describe evaluation variance, not variance across independent training runs, so the authors are careful not to make statistical-significance claims about method differences.