KnowAct-GUIClaw runs a mobile GUI agent as a host that decomposes tasks and a lightweight executor that operates the screen, then distills each run into reusable memory and state-validated skills. Skill reuse cuts ~3 GUI steps and ~6% tokens on skill-invoking tasks while raising success.
You’re shipping an assistant that clicks through mobile apps on behalf of users: check email, pull the address, open Maps, message a colleague. Today, most GUI agents treat this as one monolithic screenshot-in, action-out loop, and every new task starts from zero even if last week’s run solved something nearly identical. Prior GUI agent lines like AppAgent and Mobile-Agent focus on per-screen visual grounding but throw the trajectory away when the task ends. That means redundant navigation, lost intermediate values across apps, and no way to accumulate what worked. The practical cost is token bloat (screenshots dominate context) and brittle cross-app workflows.
The system splits responsibilities. A host agent owns the conversation, user profile, memory, and non-visual tools like web search. A GUI executor (called GUIClaw) owns screenshots, taps, swipes, and device control. The host decides what to do; the executor decides how to do it on screen.
A task flows through four stages: Know (retrieve relevant past memory and candidate skills), Route (decompose into per-app subtasks with typed inputs/outputs), Act (run the observe-reason-act loop), Reflect (distill the trajectory into memory and skills).
Two mechanisms carry the weight. First, a typed blackboard for cross-app data: each subtask declares what values it needs and what it produces, and later subtasks read those explicit fields instead of parsing a fuzzy free-text summary. If a required output is missing, the workflow fails closed rather than fabricating. Second, a skill library: repeated interaction patterns get compressed into parameterized, state-validated procedures. Before each step, the executor checks the current screen matches the skill’s expected state; on mismatch it falls back to normal GUI actions. Android deeplinks and intents are treated as one-step skills once validated on-device.
Reflection separates repair from creation: if a reused skill failed, the system edits that skill in place rather than mining a new one. Memory items are inspired by ReasoningBank but kept distinct from executable skills.
for subtask in router.decompose(user_request):
memory = store.retrieve(subtask) # advisory lessons
skills = store.retrieve_skills(subtask.app)
if host.can_resolve(subtask): # non-GUI tools suffice?
result = host.run_tools(subtask)
else:
result = guiclaw.observe_reason_act(subtask, memory, skills)
blackboard.write(subtask.outputs, result)
reflect.distill(trajectory) # updates memory + skills
The prevailing GUI-agent design treats each task as a fresh monolithic screenshot-to-action loop, with the same model handling planning, cross-app bookkeeping, and pixel-level control. This paper shows the opposite. Separate a capable host from a lightweight screen-driver, force cross-app data through a typed blackboard, and let successful trajectories crystallize into state-validated skills the next run can commit to as single decisions. The load-bearing evidence is that memory and skills distilled from one model family lift a different, weaker executor by 16.2 points, showing the knowledge is portable, not tied to the model that produced it.
The most load-bearing result is the transfer test: memory and skills distilled from Kimi K2.6 trajectories, dropped into the smaller Qwen3.5-35B-A3B executor, lift it from 24.8% to 41.0% on MobileWorld. That’s larger than the same executor’s own self-distilled configuration (37.9%) and shows the reusable artifacts encode task knowledge, not model-specific quirks.
•
Headline: with Kimi-K2.6 as both host and executor, KnowAct-GUIClaw hits 64.1% on MobileWorld GUI-Only, top of the reported table, above Seed-2.0-Pro (63.2) and GPT-5.5 (62.4).
•
Ablation on the 397B host: adding host+memory lifts SR from 40.7 to 43.3; adding skills lifts to 46.2 while cutting GUI steps from 26.8 to 23.7.
•
On tasks that actually invoke a skill: -3.3 GUI steps, ~6% fewer tokens, and +4.9 SR on the 35B executor.
•
Letting the strong host answer information-lookup subtasks directly (instead of always delegating to the GUI executor) is the best-accuracy, lowest-cost Qwen configuration: host token overhead drops from ~65k to ~11k per task.
•
Cross-platform: 76.2% on HarmonyOS mobile checks (48/63), 70.0% on Windows desktop checks (21/30).
Reach for this pattern when you’re shipping a multi-app mobile or desktop assistant where users say things like “pull the address from that email and text it to Sam.” Instead of one giant vision-language loop, run a planner that names the subtasks and declares their inputs/outputs, keep a shared typed store for values that must cross app boundaries, and log successful action prefixes as parameterized skills gated by a state check. The state check is what makes replay safe: if the screen doesn’t match, drop back to normal control.
Code is at GitHub with a companion website and released experimental logs. Evaluation uses the public MobileWorld benchmark and AndroidDaily; the paper doesn’t release its own benchmark. The skill store format (parameterized steps + state contracts + reliability counters) is reproducible from the appendix prompts.
Long-horizon GUI agents get better when you stop rerunning the whole task and start reusing verified fragments of the last one. The wins compound only when a typed contract sits between subtasks: free-text handoffs let intermediate values decay, typed handoffs preserve them and expose failures early.
•
The 64.1% headline uses a strong closed-behavior base model as both host and executor. The gains attributable purely to the framework (host+memory+skills over plain executor) are more modest: roughly 9-14 points depending on scale.
•
Transfer was tested in one direction only (Kimi-distilled artifacts into Qwen). The paper explicitly declines to claim universal cross-model transfer.
•
Failures on HarmonyOS and Windows expose real limits: inertial wheel pickers, short-lived toast notifications, and mapping affect words to unlabeled emoji icons all break the executor. The abstraction helps planning; it doesn’t fix pixel-level grounding gaps.