Work2Skill turns recorded human desktop sessions into reusable text skills for computer-use agents by letting an induction agent inspect video, screenshots, and accessibility data on demand, lifting mean task scores by roughly 3 to 10 points over giving the agent no skill.
You’ve built a computer-use agent that drives a real desktop: clicking buttons, typing into forms, running an internal reimbursement portal. It works on generic tasks but fails on your company’s workflows because the user’s request (“submit my reimbursement”) never spells out which portal, which receipts, or which pre-submit checks matter. That domain knowledge lives in people’s heads, and today your options are either fine-tuning the model or writing procedure docs by hand.
One natural fix: record a human doing the task once, then convert that recording into reusable guidance. Prior approaches like ShowUI-Aloha convert demonstrations into semantic action traces, and Demo2Tutorial produces before/after screenshot tutorials. Both lock in a fixed pipeline: parse actions, then abstract the procedure. If a later step needs evidence the parser already discarded, you’re stuck. Commercial tools like OpenAI’s Record & Replay and Claude’s Record a skill sit in the same space.
The pipeline has three parts. First, WorkRecorder captures a demonstration: screen video, mouse and keyboard events, keyframe images, and whatever the OS’s Accessibility tree exposes about the clicked controls (names, roles, bounds). On macOS it records locally; for Linux desktops it tunnels in an AT-SPI collector to grab UI metadata. Everything gets aligned on one video-relative timeline so a click can be looked up alongside its target element and the frames right before and after.
Second, WorkDemo-350 packages 350 screened human demonstrations: 64 from MyPCBench Bounded Action (personal office workflows) and 286 from a curated subset of OSWorld-Verified (general desktop and web tasks). Each episode ships as task metadata, an ordered action trace, and linked media.
Third, the induction agent. Instead of a fixed parse-then-abstract pipeline, a tool-using agent built on NexAU alternates evidence inspection with skill writing. It has tools to list files, read traces, pull specific images, fetch video metadata, sample a video interval (watch_video), or grab individual frames (extract_video_frames). The system prompt is deliberately short: inspect initial, critical, and final states; otherwise decide your own exploration. A separate authoring policy tells it to generalize demonstration-specific inputs into task-dependent descriptions, describe UI targets by semantics not coordinates, and keep selection rules, dependencies, stop conditions, and completion checks. Output is a SKILL.md file following Anthropic’s Agent Skills convention. A finalize_extraction call enforces structural checks (frontmatter, read-after-write) but does not verify correctness.
agent.load_policy("skill-creator")
task = agent.read_file("task.json")
agent.inspect(initial_frame, critical_frame, final_frame)
while agent.has_open_questions():
evidence = agent.pick_next(images_or_video_interval)
draft = agent.write_or_revise_skill(task, evidence)
agent.read_file("SKILL.md")
agent.finalize_extraction()
At execution time, the generated skill text is appended to the executor’s system prompt inside a <task_skills> block. The executor never sees the original recording.
Main result (RQ1). Across four executors (GPT-5.5, Opus-4.8, Qwen3.5-35B-A3B, Qwen3.5-9B), Work2Skill beats a No-Skill baseline by 4.67 to 10.54 score points on MyPCBench Bounded Action and 3.11 to 7.50 points on the 286-task OSWorld-Verified subset. Against the stronger of ShowUI-Aloha or Demo2Tutorial on each executor, the Work2Skill advantage is 3.95 to 7.79 points on MyPCBench and 1.65 to 6.96 points on OSWorld. For GPT-5.5 on MyPCBench, full-credit success rate reaches 48.44% versus 37.50% for both baselines. Execution-step savings are model-dependent: lower for Opus and both Qwens, higher for GPT-5.5. Induction token cost is not consistently lower than the baselines.
Generalization to task variants (RQ2). On MyPCBench Jitter (paired variants that change entities, parameters, or constraints within the same app), skills induced for the original tasks are reused without re-induction. All four executors still gain 3.93 to 8.64 score points over No Skill, and average execution steps drop by 2.49 to 5.44.
Cross-model transfer (RQ3). Skills induced by GPT-5.5 outperform those induced by Qwen3.5-9B regardless of which model executes them. But execution capability dominates: Qwen3.5-9B stays substantially weaker as an executor even when handed GPT-5.5’s skills.
Evidence ablation (RQ4). With GPT-5.5 on MyPCBench, full evidence scores 85.27 with 31 full-credit tasks. Removing indexed images, UI annotations, or key-log overlays costs 5.02 to 10.63 score points and 14.06 to 21.88 success-rate points. A video-only setup underperforms a text-only setup that still carries UI target semantics, suggesting the structured target descriptions matter more than raw frames.
Prompt ablation. The concise “you decide the sequence” prompt beats a Pipeline Prompt that prescribes a stage order by 3.97 score points and 6.25 success-rate points, at 12.7% higher induction token cost. The authors suggest a prescribed route may narrow which evidence the model considers.
Case study. On a multi-step restaurant booking with fallback and favorites requirements, baseline guidance passes 4 of 8 rubric items (55/100); the Work2Skill skill passes all 8 (100/100), largely because it makes dependent subgoals (fallback coverage, favoriting, post-refresh persistence) explicit.
•
If you own a computer-use agent that stumbles on your organization’s bespoke workflows, this paper is evidence that one well-recorded human demo plus an induction pass can be more useful than nothing, and more useful than earlier demo-to-text pipelines, across both strong and weak executors. The guidance plugs in as prompt text, so you don’t need to retrain the executor.
•
The evidence ablation is the practically useful part: the biggest signal came from indexed, event-aligned images and from UI target semantics (control names and roles from the accessibility tree), not from raw video. If you’re building your own recorder, prioritize accessibility capture and event-aligned screenshots before investing in video sampling.
•
Worth testing: does the “let the agent decide exploration order” pattern generalize to your pipeline? The authors found a prescribed pipeline prompt hurt quality. If you’ve hand-engineered a strict parse-then-abstract flow for an LLM authoring task, the concise-prompt comparison is cheap to replicate.
•
Code and data are released: GitHub. •
OSWorld numbers apply only to the 286-task curated subset after excluding infeasible tasks, not the full benchmark. Treat them as a controlled comparison, not a leaderboard entry.
•
Accessibility coverage depends on the app and OS. Workflows in apps with poor accessibility support will give the induction agent less to work with than the paper’s setup assumes.
•
Transfer results come from exactly one cross-model pair (GPT-5.5 and Qwen3.5-9B). Broader claims about induction-model choice aren’t supported.
•
finalize_extraction validates structure, not semantic correctness. A skill can pass finalization and still be wrong; the paper’s downstream evaluation is the only correctness check reported.
•
Induction token cost is not always lower than baselines, and execution-step savings vary by model. Budget accordingly.