Resource2Skills turns tutorial videos, code repos, and articles into a hierarchical wiki of executable, multimodal skills that agents browse before acting, lifting authoring-task quality by +11.9 points over the same agent with no library.
You’ve shipped an agent that drives a real application: it generates PowerPoint decks, edits Blender scenes, or wires up Reaper audio projects through a scripting API. In practice, the model knows the API surface but keeps re-deriving conventions a human learned in ten minutes of YouTube. Existing skill libraries like Voyager or Anthropic Agent Skills either grow from the agent’s own trace history in one narrow domain, or are hand-authored bundles of text and code. Neither taps the largest pool of human procedural knowledge: screen-recorded tutorials, where the ordering of clicks and the visual before/after are the actual lesson.
The pipeline has four stages that share one interface. First, a construction operator walks four resource families (tutorial videos, source repos, articles, reference artifacts) and asks a vision-capable LLM to distill each into a structured skill entry. An entry bundles a taxonomy path, prose describing when to use it, a visual thumbnail, optional executable code, and provenance metadata. A deterministic acceptance predicate then runs five gates (schema completeness, provenance traceability, dedup by content hash, modality consistency, and a sandboxed executability check on the code field) before the skill enters the wiki.
At inference time, the agent uses MetaBrowse: a BM25 shortlist over the taxonomy narrows ~thousands of skills to K=20 candidates, then a second LLM call reads their frontmatter and picks n=5 to fully expose. The taxonomy path itself is part of the retrieval text, so the tree structure biases matches toward the right subtree instead of treating the library as flat. Selected skills execute directly through an MCP (Model Context Protocol) tool surface against the domain backend, so there’s no LLM translation between “which skill” and “run it.” When retrieval finds nothing adequate, the same construction operator runs online to distill fresh skills into a separate pool.
def solve(brief, wiki):
plan = agent.plan(brief)
candidates = bm25_topk(wiki, brief, k=20) # taxonomy-aware
if not adequate(candidates):
candidates += online_distill(brief) # same operator
picks = agent.select(candidates, brief, n=5) # can pick 0
for skill in picks:
mcp.apply(skill.code or agent.adapt(skill.text, skill.visual))
return render(artifact)
The common assumption in agent memory is that procedural knowledge should grow from the agent’s own execution traces, text documentation, or hand-written skill bundles, because that’s what fits cleanly into a prompt. This paper shows the opposite. The most valuable procedural signal lives in tutorial videos and other multimodal human resources, and the winning move is to distill it offline into typed, executable, visually-grounded wiki entries rather than stream raw content into context. The load-bearing evidence is the source-mix ablation isolating video’s contribution, not the headline benchmark lift.
The most telling result is the source ablation: holding video out of the four resource families drops average score from 68.9% to 59.4%, and a video-only library still outscores a three-source no-video library by 7.4 points. The video-removal hit concentrates on Excel (-14.2 pp) and Web (-11.5 pp), where temporal ordering carries signal text can’t easily encode. Video is non-substitutable; the other sources add coverage on top.
Secondary evidence lines up:
•
Main comparison: with-skills beats without-skills in all 28 of 28 model-domain cells (four backends × seven domains), averaging 56.8% vs 45.0%, and beats off-the-shelf Claude Code and Codex CLI harnesses in 26 of 28 cells. Paired Wilcoxon p<10⁻³ everywhere reported.
•
Scaling: performance rises monotonically with library size and saturates near ~200 skills per domain; the 400→full step adds at most +0.8 pp.
•
Wiki structure: even a flat text-only skill list beats no skills, but the full hierarchical multimodal wiki adds another 2.5–8.2 pp on top.
•
Online acquisition: adds only +0.7 pp on standard briefs but +21.6 pp on a stress-test suite of capabilities the offline pool was known to miss (41.2% → 62.8%).
•
Human A/B: five raters per pair on 200 pairs prefer with-skills 85.5% of non-tied ratings.
Reach for this when you’re building an agent that drives creative or authoring software through a Python or scripting API, and you notice the model keeps re-inventing conventions that a domain expert would apply automatically. Instead of stuffing docs into the system prompt or waiting for the agent to accumulate its own trace history, harvest the tutorial-video corpus that already exists for your target application, distill each into a typed wiki entry with executable code where possible, and route retrieval through a taxonomy-aware first stage before the LLM picks what to compose.
The paper describes the on-disk schema in enough detail to replicate (skill_id folder with source/, text/, visual/, code/, meta.json), and specifies the MCP (Model Context Protocol)-mediated tool surface. Per-domain library sizes range from ~300 (CAD) to ~1000 (PPT). The paper does not clearly state a public code release URL, so treat the artifacts as reproducible-in-principle rather than downloadable.
Tutorial videos are the highest-density source of agent procedural knowledge, but only after you distill them into typed, executable, taxonomy-indexed entries. Raw video in context is waste; text summaries throw out the part that matters; the payoff is in the distillation and the browsing structure that sits between the agent and the library.
•
Scoring routes through LLM judges (GPT-5.4 vision, GPT-4o for audio) with judge-human Spearman of only 0.71 on overall score; the human A/B confirms the direction but not the magnitude of every per-domain delta.
•
The retrieval baselines (BM25, dense embedding) compete over the already-distilled skill library, not over the raw resource corpus at matched token budget. A well-tuned raw-resource Retrieval-Augmented Generation setup might close some of the gap.
•
Every evaluated domain has both a programmatic tool interface and a rich public stream of tutorial content. Domains missing either (proprietary software with no video corpus, or applications with no scripting surface) are explicitly out of scope.