RecreationWorld trains and tests hybrid computer-use agents by asking them to rebuild a running reference application across five operating platforms, using the reference itself as an oracle for automatic behavioral tests, and shows that trajectories from this loop transfer to unrelated coding and GUI benchmarks with gains up to 17.9 percentage points.
Today’s agent research splits into two camps. GUI agents like OSWorld click through real applications but can’t build the software behind them. Coding agents like SWE-bench write and test code but never see the interface their code produces. Real software work needs both, interleaved: you observe how something behaves, write code to match it, run the code, look at the result, fix what’s wrong, look again.
Concretely: imagine you want to train (or evaluate) an agent that can look at a competitor’s app, figure out what it does, and reimplement it. Today you’d need to hand-author both the task and the grading rubric per app, which doesn’t scale. The paper’s move is to make the reference app itself do the grading, so you can throw thousands of open-source apps at the agent without writing tests by hand.
The core idea is recreation as a task: give the agent a running reference application (source hidden where possible), and a workspace with GUI control plus normal dev tools. The agent must deliver runnable source that reproduces the observable behavior. The paper calls the resulting inner loop explore-implement-verify: click around the reference, write code, build and launch your candidate, inspect it, repeat.
Evaluation works by turning the reference into a test oracle. An orchestrator LLM drives the reference, records what happens (widget states, text, screenshots after specific interaction sequences), and emits two kinds of hidden checks:
•
Programmatic assertions read structured state through each platform’s accessibility layer (AT-SPI on Ubuntu, AXUIElement on macOS, UI Automation on Windows, UiAutomator on Android, DOM/ARIA on Web).
•
Visual assertions pair a screenshot checkpoint with a natural-language description that a vision-language judge scores.
Every generated assertion is run against the reference and reviewed by humans before the suite is frozen. Then it’s replayed unchanged against candidates.
For training, they run this same loop at scale using Qwen3.8-Max to generate trajectories, keep only high-scoring ones via rejection sampling, and get 35,000 trajectories (7,000 per platform) for Supervised Fine-Tuning.
# Per-task loop, run in isolated VM with 20-hour budget
agent = Agent(tools=[gui_control, shell, file_edit])
ref = launch_reference(task) # source hidden on desktop/mobile
while not agent.done() and within_budget():
obs = agent.explore(ref) # click, screenshot, read a11y tree
agent.write_or_edit_code(workspace) # build candidate
candidate = build_and_launch(workspace)
agent.inspect(candidate) # visual + behavioral self-check
trajectory = record(agent.history)
score = replay_hidden_tests(candidate) # prog + VLM, frozen suite
if score > threshold: keep_for_training(trajectory)
RecreationBench holds out 250 apps (50 per platform). Ten frontier models were evaluated with a Claude Code or Codex scaffold plus MCP tools.
•
GPT-6 Astra leads at 58.1% average of programmatic and visual scores, followed by Claude Opus 5 at 44.2%. But it passes the full programmatic suite on only 2.8% of tasks, and hits 90%+ on 17.6%. Every other model is at or below 0.8% for full-suite passes. Reference-level fidelity is far off.
•
Static structure is easier than interaction. Across platforms, assertions about what an interface contains (buttons, labels, layout) pass much more often than assertions about what happens when you use it (state changes, computed outputs). On Web, static-content scores beat interaction scores by 28-37 points.
•
Recreations are smaller and more monolithic. 89.4% of delivered apps have less production source than the reference; median ratio is 16.9%. 92.3% use fewer files, and agents frequently swap the reference’s framework for a platform-native one (Electron references often become GTK or AppKit).
•
Training transfer. Fine-tuning two model starting points (a post-trained Qwen3.7-Plus and a continual-pretrained Qwen-Flash checkpoint) on the 35k recreation trajectories improved both across five out-of-distribution benchmarks (ProgramBench, GameCraft-Bench, Vision2Web, OSWorld 2.0, WeaveBench), with gains up to 17.9 pp from first to last checkpoint. Trained agents also more often re-inspect their own rendered output.
•
Programmable runtime side-study. Exposing GUI tools through a persistent JavaScript REPL (instead of one MCP call per primitive action) cut input tokens by 40.7%, wall-clock by 26%, and estimated cost per task from $90.50 to $41.58 on Windows with Claude Opus 4.8, with similar task quality. This is one rollout per condition, so it’s an observed configuration difference, not an isolated causal effect.
One repeated caveat from the authors: broader reference exploration co-occurs with higher scores, but they don’t claim exploration causes the gains.
•
If you’re building agent-evaluation infra, the recreation trick is worth stealing: an executable reference lets you skip most of the per-task rubric-authoring work, and the two-channel (programmatic + VLM) grading catches things pure accessibility-tree checks miss. The ConnectYou case in the appendix shows two recreations with nearly identical programmatic scores (~41%) but very different visual fidelity (61 vs 45), because one lost the empty-state illustration entirely.
•
If you’re evaluating whether your agent “finishes” a coding task, note the paper’s finding that 23-48% of trajectories end without relaunching the patched build. The Plus Plus Battery case shows an agent editing Kotlin and XML to fix a broken layout, then submitting without re-running: the final artifact scored 100% visually but 55% programmatically. Worth checking whether your harness’s final-turn behavior actually validates the submitted artifact.
•
The transfer result suggests recreation-style trajectories are usable Supervised Fine-Tuning data for agent training even if you don’t care about recreation as an end goal. This is a claim about improving from a first checkpoint, not a claim of state-of-the-art; the trajectories aren’t uniformly monotonic across checkpoints.
•
Artifacts released: benchmark and code on GitHub, dataset on Hugging Face, and a live comparison site.
•
The 58.1% headline number is a mean of programmatic and visual channel scores, macro-averaged across platforms. It’s not a pass rate. Only 2.8% of tasks get full programmatic credit even for the best model.
•
Contamination risk is real: many reference repos are public and may be in pretraining data. The authors audit exact-line source overlap (median 0.01-0.09% across platforms, max under 3.2%) but note this can’t detect memorization via renamed identifiers or refactored control flow.
•
Web can’t hide client source (it’s what the server sends), so the source-blind protocol only holds cleanly on Ubuntu, macOS, Windows, and Android. Android additionally leaves the installed reference APK technically recoverable and doesn’t enforce packet-level network isolation.
•
Two documented failure modes on Web: one Gemini rollout scraped the reference’s HTML and rendered it through a generic renderer; another restructured the reference content into JSON and replayed it, which their scrape detector missed. Programmatic and VLM scores can reward replay unless you audit provenance separately.
•
The training-transfer experiments use one rollout per task per checkpoint, differ in budgets and harness revisions across benchmarks, and the authors label them as “initial evidence,” not controlled comparisons.