Player-1 trains screen-reading game agents by first imitating a few thousand gameplay recordings, then doing RL inside sixteen purpose-built synthetic mini-games whose rewards can be checked programmatically. Control skills learned there transfer to unseen commercial games.
Suppose you want an agent that watches a game screen and plays with keyboard and mouse, the way Computer-Using Agent already handle web pages. Games break those agents: the world moves on its own, inputs must be held or combined, and the camera has no cursor to click. The usual fix is to collect huge demonstration datasets, but demonstrations only cover states a human happened to visit, and standing up a reward signal inside each commercial game is hard. Prior work like SIMA trains across many real virtual worlds; Player-1 takes a different route and asks whether a small, controlled synthetic suite is enough.
The pipeline has three pieces that chain together.
First, the action vocabulary gets extended. Standard computer-use agents click absolute pixel coordinates and tap keys. For games the authors add relative pointer motion (for mouselook without a visible cursor), sustained holds (a key or button held across a fixed ~200 ms control step), and composite actions (several key and mouse inputs issued concurrently in one step, like holding W while tapping Space).
Second, a supervised warm-start. They curate 6,044 gameplay trajectories (~295k decision turns) from public datasets and in-house recordings, retime them onto the shared 200 ms grid, and use Gemini 3.8 Flash to label each turn with a task instruction and a short visual reasoning trace. They then fine-tune two Qwen3.5 backbones (9B dense and 35B-A3B Mixture of Experts) with action-balanced sampling so rare but important inputs (like left-clicks) are not drowned out.
Third, RL inside Player-1-Gym. The authors built sixteen 2D and 3D browser environments organized around five recurring control demands: aiming, combat, driving, navigation, UI interaction. Each environment exposes a privileged game state the policy cannot see, from which a scoring rule emits dense per-step rewards. They randomize appearance, camera, scene layout, and control response (for example, mouse sensitivity) per episode. Training uses PPO across all sixteen environments jointly. Because reward scales differ wildly between a driving game and a UI game, they rescale each advantage by the mean and standard deviation computed within its own environment, so no single game dominates the gradient. Plain English: before updating, every environment’s rewards get re-centered to the same scale.
for step in range(200):
rollouts = []
for env in sixteen_envs: # 16 per step, 256 episodes total
obs = env.reset(randomize=['appearance','camera','scene','control'])
traj = policy.rollout(obs, max_turns=50)
traj.reward = env.score_delta() - invalid_penalty - length_penalty
rollouts.append(traj)
adv = gae(rollouts); adv = normalize_per_env(adv)
ppo_update(policy, critic, rollouts, adv)
The headline evaluation is MineDojo-300, 300 Minecraft combat and harvest tasks with 8 trials each.
•
Player-1-9B rises from 10.54% success after SFT to 38.33% after RL, a jump of 27.8 pp. The 35B-A3B version goes from 23.38% to 41.04%.
•
The strongest external baseline tested, GameWAM, scores 12.63% under the same protocol. The paper is careful to call this “the strongest evaluated baseline” rather than a general SOTA claim.
•
Task coverage (tasks solved in at least 4 of 8 trials) rises from 20 to 136 for 9B, so gains are not concentrated in a few easy tasks.
Two generalization checks matter because neither game is in the RL suite:
•
On ViZDoom-10, a first-person shooter benchmark, Player-1-9B’s scaled return rises from 0.4361 to 0.6922.
•
On GameWorld (170 browser-game tasks across 34 games), progress rises from 31.43% to 34.45%. A smaller lift, but still positive on a very different interface.
•
Supervised init is load-bearing. Running RL directly from the base Qwen3.5-9B lifts Minecraft success from 12.67% to 16.67% in 100 steps; running it from the SFT checkpoint lifts 17.67% to 51.67%, a 35 pp gap at step 100. RL alone, from a base model, barely moves.
•
Breadth of the environment suite helps. At a fixed budget of 25,600 episodes, scaling from 2 to 16 training environments improves transfer to held-out games.
•
Composite-action use emerges on its own. Share of turns issuing combined inputs rises from 64.7% to 93.2% over 100 RL steps, with no explicit reward for coordination. The authors read this as task pressure selecting for things like “aim while fire” or “move while turn”.
A qualitative demo shows Player-1-9B playing Watch Dogs 2, unseen in both SFT and RL, handling on-foot navigation, entering a car, and driving. This is a demo, not a measured success rate.
•
If you are building a screen-reading agent for anything that is not a static web page (games, simulators, CAD, live dashboards), the three action-space extensions (relative motion, sustained holds, concurrent inputs) are a cheap change worth replicating. The emergence result suggests composite inputs will be learned if the reward actually depends on timing, so the main lift is making them expressible in the first place.
•
The core methodological bet (build a small suite of synthetic environments with programmatic rewards instead of chasing demonstrations in each target game) is worth testing when you can author rewards but cannot author trajectories. The scaling result argues for breadth of environments over depth in any one.
•
The SFT-vs-RL ablation is the sharpest practical warning: skipping imitation and going straight to RL from a general base model essentially did not work here. Budget for a demonstration phase even if demonstrations do not cover your target states.
•
The authors do not say whether code, weights, or Player-1-Gym will be released; the paper has no artifact links in the text provided, so plan for a from-scratch reimplementation.
•
Comparisons to external baselines use the authors’ own harness, action adapters, and screenshot resizes. Each baseline was built for a different setup, so the 12.63% figure for GameWAM reflects performance under Player-1’s protocol, not a native benchmark number.
•
Transfer is demonstrated on three external suites and one commercial-game walkthrough. “Generalizes beyond the RL training suite” is supported for these cases, not as a universal claim about arbitrary games.
•
Rewards in Player-1-Gym come from privileged game state the authors wrote. If you cannot inspect state in your target environment, you can still use the trained policy, but you cannot extend the RL stage to it without building equivalent reward hooks.
•
The Watch Dogs 2 sequence is a storyboard example, not a measured success rate on commercial titles.