GameHorizon Suite is a data and evaluation package for measuring whether models can play AAA video games across seconds-to-minutes time horizons, built from 5,000 hours of human recordings across 21 titles with automatically generated instructions at three temporal levels.
Suppose you want to know if a general VLM can actually play Cyberpunk 2077, not just describe a screenshot of it. Today you either build a bespoke agent harness (like the ones people used for Gemini or Claude playing Pokémon) and run a handful of trials, or you test on Minecraft with MineDojo and hope it generalizes. Both options give you noisy, unreproducible scores that collapse very different failure modes (misreading the screen, planning the wrong subgoal, executing the wrong keypress) into one success rate.
Existing gameplay datasets have parallel problems: they cover one game (STEVE-1 on Minecraft, WildWorld on Monster Hunter), skip language instructions entirely, or use action labels inferred by an IDM rather than actual human keypresses. The authors argue you need a single yardstick that spans many AAA titles, records real human keyboard/mouse traces, and carries language instructions at multiple time scales so you can separately test perception, planning, and execution.
The suite has three pieces that build on each other.
Data collection. 100 trained human players recorded gameplay at 2K resolution with synchronized keyboard and mouse events, across 21 AAA games. This gives real action labels, not pseudo-labels from an inverse dynamics model.
GameHorizon-Annotator turns raw recordings into a three-level instruction pyramid: L1 short-horizon operations (1–5 seconds, e.g., “sprint toward the door”), L2 medium-horizon goals (1–2 minutes, e.g., “clear this building”), L3 long-horizon strategies (5–8 minutes). It works bottom-up. First it cuts video into short clips using the actual keyboard/mouse trace to find action boundaries (rather than PySceneDetect, which fragments continuous motion under camera shake). Then a VLM (Gemini 3.5 Flash) labels each short clip. Then adjacent clips are merged into medium clips when the VLM judges they share a goal, and again into long clips sharing a strategy. Higher levels see the lower-level labels as context, so the abstraction stays consistent.
clips_L1 = action_aware_segment(video, keyboard_mouse_trace)
for c in clips_L1:
c.label = vlm_annotate(c.frames, c.actions, keybinds)
clips_L2 = merge_while(clips_L1, same_goal_judged_by_vlm, dur=(60, 120))
for c in clips_L2:
c.label = vlm_annotate(c.frames, c.actions, child_L1_labels)
clips_L3 = merge_while(clips_L2, same_strategy_judged_by_vlm, dur=(300, 480))
for c in clips_L3:
c.label = vlm_annotate(c.frames, child_L2_labels) # no actions at L3
GameHorizon-Bench has two tracks. The offline track is 5,000 multiple-choice questions across three primary tasks: T1 pick the right action given frames and an L1 instruction; T2 decompose an L2 goal into the correct ordered L1 sequence; T3 pick the consistent (L1, L2, L3, actions) tuple. Ten diagnostic variants isolate perception vs. planning, top-down vs. bottom-up, and instruction-only vs. instruction+action. The online track runs actual gameplay in Minecraft (the only environment where they can read game state) on 20 long tasks made of 62 verifiable subtasks; when a subtask fails, the environment resets to the success state so the next subtask can still be scored. That gives per-step failure localization instead of one collapsed success rate.
They ran 47 models across general VLMs, unified multimodal models (Unified Multimodal Model), GUI agents like UI-TARS, coding agents (Claude Sonnet/Opus/Fable), and dedicated game agents like JARVIS-VLA. Over one million model calls total. Highlights:
•
Task difficulty ordering as designed. Mean accuracy was 57.3% on single-horizon action (T1), 65.1% on decomposition (T2), 71.6% on cross-horizon consistency (T3). Overall model range 44.6% to 80.2%, so the benchmark discriminates.
•
Proprietary VLMs dominate. GPT-6-Astra leads at 80.2%; Gemini 3.8 Flash and 3.7 Flash follow. Kimi-K3 is the only open-weight model in the top tier.
•
Dedicated game agents fail on unseen AAA titles. All three (NitroGen, Open-P2P, JARVIS-VLA) score near the 25% random baseline on GameHorizon’s AAA questions. JARVIS-VLA hits 54.2% on in-domain Minecraft (VPT (Visual Prompt Tuning) data) but drops to 28.1% on AAA games. Domain-specific action training doesn’t transfer.
•
Instructions help, and higher horizons help planning specifically. Adding L1 instructions to visual input lifts current-action perception by +14.1 pp on average. Adding full multi-horizon instructions lifts future-action planning by +7.0 pp over L1-only, but barely helps current perception. So the L2/L3 layers earn their keep on the planning side.
•
Planning lags perception. Current-action perception beats future-action planning by ~10–24 pp depending on input condition.
•
Bottom-up abstraction is nearly saturated (98.4%) while top-down decomposition sits at 69.5%. The authors read this as justifying their design choice: use bottom-up for annotation (it’s reliable) and top-down for evaluation (it’s discriminating).
•
Thinking mode is a mixed bag. It helped stronger models (Doubao-Seed-2.1-Pro +9.3 pp, big gains on T2 decomposition) and hurt lighter ones (UI-TARS-1.5-7B −8.2 pp), which the authors attribute to hallucinated reasoning chains in weaker models.
•
Offline scores predict online play. The 12 models tested online rank consistently with their offline tier. Even so, only GPT-6-Astra clears 45% on long-horizon Minecraft tasks; ten of twelve are at 10% or below. One failed subtask kills a whole task.
Caveat on the causal reading: the +7.0 pp future-planning gain from multi-horizon instructions is measured on a 200-question variant across four models, and the authors don’t isolate L2 vs. L3 contributions.
•
If you’re evaluating a general-purpose VLM or agent for game-like control, the offline MCQ track is the cheap first move. It runs without a game environment or harness and correlates with actual online play in this study, so it can filter candidates before you invest in a rollout rig.
•
If you’re training a game agent and considering whether to add language supervision, the perception-vs-planning split is the relevant signal: short-horizon instructions mostly help identify what’s happening now, longer-horizon instructions mostly help decide what to do next. Pick the level that matches your bottleneck. This is a correlational finding on frozen models, so “worth testing” during your fine-tuning rather than a guaranteed recipe.
•
If you’re choosing between reasoning-mode and fast-mode for real-time control, the thinking-mode results suggest testing it per-model rather than defaulting on. Weaker models hallucinated plans that hurt action selection.
•
Don’t read the online numbers as a Minecraft leaderboard for deployment. The online track is 20 tasks in one game with a 400-call budget per subtask and environment resets on failure. That protocol is designed for diagnosis, not for benchmarking production agents.
•
Artifacts: dataset, annotator, and benchmark are promised for release at the project page and GitHub. The paper doesn’t specify licensing for the AAA game recordings, which matters if you want to redistribute derived work.
•
The online track is Minecraft-only. The authors say this is because most AAA titles don’t expose game state for automated verification. So the offline-to-online correlation is established in one environment, not across the 21 AAA titles the offline track covers.
•
The instruction pyramid is VLM-generated, mostly by Gemini 3.5 Flash. That introduces the annotator model’s biases into both training data and evaluation questions. The paper doesn’t report human agreement rates on the auto-labels.
•
Game agents are only tested on T1 (single-horizon action) because they don’t answer MCQs. Their poor scores partly reflect that action-sequence-to-option matching via Needleman-Wunsch alignment can penalize any output that doesn’t align cleanly, including empty sequences. The comparison to VLMs on other tasks isn’t apples-to-apples.
•
"5,000 hours" is raw recording; only 4,341 hours (86.8%) survived filtering for instruction annotation. Actions are available for the full 5,000.
•
The offline variants use 200 questions across four models, so several of the more interesting effect sizes (multi-horizon planning gain, thinking-mode swings) rest on small samples and shouldn’t be over-read as precise.