Code2Skill mines GitHub repositories for reusable procedures, distilling each function or workflow into a skill record whose accuracy is checked by having an LLM reconstruct the code from the skill alone. The resulting bank lifts agent scores across SWE-bench Verified and math/terminal tasks by roughly 11.7% relative.
Suppose you’re building a coding agent that has to fix a bug in an unfamiliar repo. The agent can reason, but it doesn’t know the specific validation-then-retry patterns your codebase uses. Two existing fixes exist. One is Trajectory-derived skills: let the agent run tasks, then distill successful runs into reusable hints. That works but is bounded by whatever the agent has already been good enough to solve, and it goes stale when tools or prompts change. The other is document-based skill synthesis from human-written docs, which avoids that dependency but has nothing executable to check the extracted skill against.
The Ant International team argues source code is a better substrate: it is executable, version-controlled, and already contains billions of debugged procedures encoding when they apply, what invariants must hold, and how they fail. The engineering question is how to turn a function body into a skill card that actually captures those things without hallucinating constraints the code doesn’t support.
The pipeline, called Code2Skill, runs four stages over 19,769 GitHub repos with >500 stars.
1.
Select candidate units. Parse functions, methods, CLI entry points, and file-level components. An LLM tagger scores each for reusable intent, ordered steps, visible invariants, and non-triviality (skip getters, wrappers, config).
2.
Extract a typed skill record. For each kept unit, an extractor writes a JSON card at one of three granularities: atomic (single operation), composite (ordered workflow), or recurring-pattern (higher-level structure). The card contains when-to-use, workflow steps, invariants, error cases, anti-goals, and provenance pointers back to the source spans.
3.
Verify by source-body-blind reconstruction. This is the load-bearing idea. A second LLM sees only the skill card (not the original code) and tries to regenerate the implementation. A source-aware judge then compares the reconstruction to the true source. If they match behaviorally, the skill is accepted directly. If not, an adjudicator decides whether the skill itself was faithful but the regeneration flubbed it, versus the skill being genuinely underspecified.
4.
Index for retrieval. Accepted records get feature tags from a fixed vocabulary (task family, capability, constraint, success signal, interaction pattern) plus a purpose-indexed view that clusters near-duplicates and picks one representative per cluster.
The control-flow, in pseudocode:
for unit in parse_repo(repo):
if not tagger.keep(unit): # stage 1
continue
skill = extractor.extract(unit) # stage 2
if skill.value_score < 0.45:
continue
regen = reconstructor(skill) # source hidden
verdict = judge(unit.code, regen, skill.summary)
if verdict.equivalent:
accept(skill, stage="direct")
elif adjudicator(skill, unit.code, regen).keep:
accept(skill, stage="adjudicated")
Applied at scale, this produces CodeSkillBank: 1,006,822 accepted records. At inference, an agent retrieves top-k records for a task and injects them at some point in its loop (before generation, during planning, or during post-hoc review).
Retrieval helps across the board. Under a fixed Reflexion-style draft/review/revise loop, adding CodeSkillBank improves 57 of 72 protocol-matched runs across five model families (DS4-Flash, Qwen3.5, Qwen3.6, Gemini-2.5-Pro, GPT-5.2) on eight benchmarks. Macro average rises from 42.90 to 47.90. All nine SWE-bench Verified pairs improve.
Code-derived beats trajectory-derived under a shared interface. Compared against three trajectory-based baselines (Trace2Skill, ExpeL, and a SkillRL-Bank-adapted SkillBank) built on held-out tasks and evaluated through the same loop, Code2Skill wins on all seven shared benchmarks. Average score is 49.5 vs 31.0 / 27.9 / 32.8 for the baselines. Even an oracle picking the best baseline per benchmark averages only 40.1.
Where you inject the skill matters. In the default review-stage setup, gains are broadest. Planning-time injection improves all eight tested pairs. Generation-time injection (skills in the first-pass prompt) helps DS4-Flash but is inconsistent for Qwen3.5. In a coding-RL setting on a Qwen3-32B SWE-World checkpoint, all injection points beat the no-skill 24% resolve rate at step 150, but post-generation review reaches 38%, roughly double the gain of policy-side or reward-side injection.
Compact rendering is cheap. At k=3 on BigCodeBench Instruct-Hard, replacing full skill records with short summaries cuts rendered context by 88.9% (6,352 to 707 chars) without losing utility. Increasing k from 1 to 10 mostly adds tokens without adding accuracy.
Human audit. Annotators judged 92% of accepted skill descriptions accurate and 80% worth retaining; 84% of directly-accepted records support correct blind reconstruction. Rejected records score 32% / 28% / 0% on the same axes, so the filter is doing real work.
AI-written code works as a source too. On a 25-pair sample where Codex (with GPT 5.1) reimplemented human functions to pass identical tests, skill banks derived from either source scored ~93.5% on a 400-task LiveCodeBench subset. They disagreed on 16 tasks, suggesting the AI-code bank is not redundant with the human-code bank.
•
If you’re building a coding or terminal-control agent, the dataset is released. Retrieval as a post-generation review context is the most consistent win in this paper. Worth testing before you commit to trajectory collection. •
If you’re deciding between trajectory-mining vs code-mining for a domain-specific agent, the RQ2 comparison is the strongest evidence in the paper that code-derived skills give better cold-start procedural knowledge than distilled agent experience. This is under a shared retrieval interface, though; a fully tuned trajectory pipeline might close some of the gap.
•
If you’re context-constrained, summary rendering at k=3 preserves nearly all utility while cutting ~9/10 of the retrieved text. Don’t retrieve deep by default.
•
If you’re doing RL on coding tasks, exposing skills to the reviewer rather than the policy or reward model was the strongest single-checkpoint result. Treat this as suggestive rather than settled: the paper reports one step (150) with no seeds or learning curves.
•
The reconstruction-based check is a general trick for validating any code-derived structured artifact: if an LLM can rebuild the artifact’s referent from the artifact alone, the artifact captured enough. Worth borrowing for API-spec extraction, test generation, or documentation mining.
•
Reported macro-averages combine benchmarks with different evaluators and task counts; they’re descriptive, not pooled success rates. Item-level significance tests were not run across all pairs.
•
The RL result (RQ5) is a single training step (150), single seed, single checkpoint. It does not establish anything about convergence or final policy quality.
•
The AI-code-source experiment (RQ6) uses only 25 function pairs on 400 LiveCodeBench tasks. The 0.5-point aggregate gap does not establish equivalence with human-code sources; it establishes that AI-generated code contains extractable signal.
•
The reconstruction judge and the extractor are both LLMs, so systematic blind spots (e.g., subtle concurrency bugs an LLM won’t flag) can pass both stages. Human accuracy on descriptions is 92%, not 100%.
•
Baselines in RQ2 were re-implemented by the authors on a held-out split using Qwen3.5-397B-A17B; original-paper numbers for Trace2Skill, ExpeL, and SkillRL-Bank may differ.
•
Gains are smallest on short problems that capable models already solve directly (e.g., mixed results on BigCodeBench under reasoning mode). The retrieval overhead may not pay off outside multi-step, repository-scale, or system-interaction tasks.