SoftwareGym synthesizes 182 runnable desktop apps through a design-implement-test loop with GUI feedback repair, giving computer-use agents transferable training environments that lift OSWorld Pass@3 by ~15 points without touching OSWorld apps.
If you’re training a Computer-Using Agent, you need a lot of real software the agent can click through, where you can reset state, verify outcomes, and collect clean trajectories. Two paths exist today, and both hurt. Wrapping real desktop apps (the approach of benchmarks like OSWorld) is expensive to integrate per app, so coverage stays narrow and models overfit to the specific apps you wrapped. Browser-hosted synthetic environments scale better, but they share one HTML/DOM substrate, so interaction depth and persistent workflows stay shallow.
SoftwareGym’s bet is that you can generate standalone desktop applications from scratch, with controllable interfaces, functions, and interaction flow, as long as you verify them end-to-end by actually driving their GUIs.
The pipeline is a three-stage loop run by multiple LLM-driven agents (built on Codex CLI with GPT-5.5).
•
Design. A FunctionDesigner expands a short seed (“image editor”) into a detailed functional spec: target users, core objects, workflows, failure handling, and per-function acceptance checkpoints. A StyleDesigner then generates multiple UI blueprints for the same spec, forcing each pair of variants to differ along at least two layout dimensions (navigation, spatial organization, browsing, action placement). A tree-edit-distance check on the normalized layout tree rejects variants that are structurally too similar, so you don’t get ten cosmetically different clones.
•
Implementation. A Coder agent builds the app in Avalonia, and a DevCoder audits it against the spec. Each specified function must have working command bindings, state updates, persistence, and visible feedback.
•
Evaluation and repair. A separate GUIAgent (driven by Seed 2.1 Pro) runs mouse-and-keyboard test cases inside an Ubuntu VM, up to 50 steps per case. Failed traces, with suggested fixes, go back to the Coder for the next round. Up to three repair rounds; a visual-fidelity score guards against repairs that drift from the intended look.
Only apps that pass this loop enter the library. A separate task-synthesis stage then builds training/eval tasks grounded only in capabilities that have a passed GUI checkpoint, which prevents the model from learning to invoke features that don’t actually work.
for app in designed_apps:
code = coder.implement(app.spec, app.blueprint)
for round in range(3):
report = gui_agent.run_tests(code, app.test_cases)
if report.all_pass(): break
code = coder.repair(code, report.failures)
if report.pass_rate >= threshold:
library.add(app, verified_capabilities=report.passed)
•
Library quality. 182 apps across twelve categories, mean GUI task pass rate climbs 54.9% → 81.1% → 91.5% across the three repair rounds. Across 121 apps in categories that overlap OSWorld, GUI evidence covers all 49 major functional modules and 78.8% of concrete native functions.
•
Out-of-domain transfer. A Qwen3.5-9B model fine-tuned only on SoftwareGym trajectories, with zero OSWorld apps, tasks, or trajectories in training, improves OSWorld Pass@3 from 41.39% to 56.97%. The paper frames this as evidence that synthetic environments teach transferable GUI skills, not just app-specific procedures.
•
In-domain. On the authors’ own SoftwareGym-Bench (120 tasks, 10 per category, with distractors and protected objects), the same model reaches 40.0% Pass@3, beating several larger specialized CUAs reported in the comparison.
•
Ablations. Structured style design lifts macro-average structural diversity from 4.9 to 83.6 on a 0-100 scale. GUI-guided repair reaches 83.1% pass rate from the same 50.4% start, while source-only repair ends at 48.8%, meaning code review without runtime testing can actively regress functionality. Combining SoftwareGym and OSWorld trajectories pushes SoftwareGym-Bench to 47.5% while keeping OSWorld at 58.61%, suggesting the two data sources are complementary rather than redundant.
•
Cost. Sampling 20 apps, each takes on average 237 minutes, 14.8M tokens, 1,718 model calls, ~$70.76. Cheap compared to wrapping a real app, expensive compared to a single prompt.
•
If you’re training or fine-tuning a computer-use agent and your data is bottlenecked on wrapped real apps, this paper is direct evidence that synthetic desktop environments can carry a meaningful share of the training signal. The transfer to OSWorld without any OSWorld data is the headline result worth replicating.
•
If you’re building any pipeline that generates then self-evaluates artifacts (code, UIs, agents), the repair ablation is the sharper lesson: source-level self-critique alone drifted backward on functional pass rate. Runtime feedback from actually exercising the artifact was decisive. Worth testing in your own loops before trusting LLM-as-judge over code review.
•
If you’re constructing benchmarks, note the construction trick in SoftwareGym-Bench: tasks include distractors and protected objects in the initial state, so a correct action depends on inspecting state rather than replaying a canned click sequence. Harder to game, closer to real desktop work.
•
The paper reports releasing the 182 apps and the 120-task benchmark; the supplied text doesn’t give a repository URL.
•
The 56.97% OSWorld number is a 9B model trained from one specific base (Qwen3.5-9B), compared against its own 41.39% baseline. The OSWorld-only control reached 61.48% on OSWorld, so SoftwareGym does not beat training on in-distribution data; it substitutes for it with reasonable loss and complements it when combined.
•
“Out-of-domain” means OSWorld’s apps were unseen, but the eight OSWorld domains overlap heavily with the twelve synthesized categories (browser, writer, spreadsheet, etc.). This is near-domain transfer across interface styles, not transfer to a wholly different task family.
•
The synthesized apps are Avalonia desktop apps generated by LLMs; they resemble real productivity software but are not the real software. Behaviors a real app has that no spec captured (OS integration quirks, third-party plugins, network dependencies) are absent from training.
•
Design, implementation, and evaluation all lean on frontier models (GPT-5.5, Seed 2.1 Pro). The pipeline’s reliability is partly a reflection of those judges and generators; weaker models in the loop would likely yield weaker apps and noisier verification.