Skill-α trains an LLM to build reusable agent skills by making one local edit at a time. Each edit’s reward comes from replaying the SAME query under the old and new skill, so credit goes to the edit itself, not the worker’s luck.
You’re shipping an agent that solves customer-support tickets or spreadsheet tasks. To make it reliable, you hand it a SKILL.md: a hand-written procedural playbook that tells it how to decompose the task, which tools to call, and how to verify results. Writing and maintaining that playbook by hand doesn’t scale, and the obvious automation, having an LLM summarize past documents or trajectories into a skill, produces text that reads fine but often doesn’t actually help the downstream agent. Prior automated skill writers like Ctx2Skill for documents or ExpeL for experience are pipeline-specific and rely on heuristics; you need a separate design per evidence type, and none of them optimizes directly for whether the skill changes agent behavior for the better.
Instead of generating a whole skill in one shot, Skill-α treats skill construction as a sequence of small edits to a living Markdown document. At each step, an editor model sees the current skill plus one new chunk of evidence (a document segment or a trajectory batch) and picks one of five actions: Create, Update, Merge, Prune, or Noop. That’s the action space.
The hard part is teaching the editor which edits are actually good. The paper’s key move is the rollback reward. For each candidate edit, the system picks an anchored query tied to the same evidence source, runs the fixed worker agent (GPT-4o) on that query TWICE, once with the old skill and once with the edited skill, and lets a benchmark-specific verifier score both. The edit gets reward 1 only if it beats the control on that same query. Noop gets reward 1 only if no sibling edit in the sampled group beat the control. This design cancels out the worker’s baseline capability and query difficulty, because both branches face the identical query. The editor is then trained with Group Relative Policy Optimization (GRPO) on groups of candidate edits, initialized from qwen3-8b after a supervised warm-up on synthetic edit traces from DeepSeek-V4-Pro.
for (skill, evidence, anchor_q, verifier) in dataset:
r_ctrl = verifier(worker(anchor_q, skill))
candidates = policy.sample_group(skill, evidence, G=8)
rewards = []
for edit in candidates:
new_skill = apply(skill, edit)
r_edit = verifier(worker(anchor_q, new_skill))
rewards.append(1 if r_edit > r_ctrl else 0)
grpo_update(policy, candidates, rewards)
The usual way to auto-generate agent skills is to prompt an LLM to summarize documents or trajectories and hope the resulting text helps downstream. This paper argues the opposite. The value of a skill edit can only be judged by whether the same worker, on the same query, does better with the edit than without it. So train on that comparison directly, one small edit at a time. The load-bearing evidence is the ablation showing that swapping rollback reward for a plain verifier score on the edited skill alone collapses gains to roughly the supervised-warmup level.
The ablation is the most telling result. Removing rollback reward (using the edited skill’s raw verifier score instead of the paired comparison) drops CL-Bench average from 10.38 to 3.68, SpreadsheetBench from 27.50 to 17.00, and tau2-bench average from 55.83 to 46.67, essentially back to the supervised-only baseline of 3.46 / 15.50 / 44.17. That gap is what makes the mechanism claim credible: without the paired comparison, RL adds almost nothing over just teaching the edit format.
Secondary findings, treat as evidence the mechanism generalizes:
•
On the headline benchmarks with GPT-4o as worker, Skill-α beats the strongest baseline by +3.3 points on CL-Bench and +6.7 points on tau2-bench average. On experience-to-skill it wins or ties every column.
•
Skills generated using GPT-4o traces transfer to Claude-Sonnet-4.5 as the downstream worker (tau2-bench average 70.33 vs 65.83 for the next best), which suggests the editor is producing genuinely task-useful text rather than a GPT-4o-shaped prompt hack.
•
Removing Merge/Prune from the action space tanks tau2-bench to 39.17: pure accumulation of new rules without consolidation is worse than editing with the full toolkit.
•
Evidence batch size matters more than evidence order. Four evidence units per step is the sweet spot; one unit makes the editor myopic, eight overloads it.
Reach for this when you’re building an agent that carries a procedural playbook (SKILL.md, system-prompt library, workflow memory) and you can define a programmatic or rubric-based verifier for the target tasks. The pattern to steal: when you’re evaluating any change to the agent’s context (a new rule, a merged section, a pruned instruction), evaluate it by A/B-ing the SAME task under the old and new context with the same model, not by asking the model to grade the new context in isolation. That’s the piece that makes credit assignment work, and you can use it even without the full RL loop.
Code is at github.com/ejhshen/skill-alpha. The benchmarks used are CL-Bench (documents to skills), SpreadsheetBench (spreadsheet tasks), and tau2-bench (customer-service workflows across Airline, Retail, Telecom), all with their native verifiers. Training uses a Qwen3-8B editor, GPT-4o as the fixed worker for rollback evaluation, and GPT-5.5 as the rubric judge for CL-Bench.
Score the edit, not the artifact: replay the same query with and without the change, and let the delta be the reward. Prompted summarization of documents or traces looks like skill generation but doesn’t optimize for the thing that matters, which is whether the downstream agent actually behaves better. A paired rollback comparison collapses that gap into a signal you can train on.
•
The reward loop requires a per-benchmark verifier (rubric judge or environment feedback) and an anchored query pool sitting in the same task family as the evidence. In domains without a clean automated verifier, the whole mechanism doesn’t apply.
•
One rollback comparison is a noisy Bernoulli sample. The appendix proves it ranks edits correctly in expectation, but ties and small degradations both get zero reward, so the signal is coarse. Absolute gains on CL-Bench are also small in raw points (single digits on most categories), so headroom above no-skill is limited on some task families.
•
The worker is held fixed at GPT-4o during training. Skills transfer to Claude-Sonnet-4.5, but the paper doesn’t test weaker or open-source workers, where the strong-model prefix assumption may not hold.