Kaggle Game Arena evaluates frontier LLMs through repeated head-to-head play in Chess, Poker, and Werewolf, using ground-truth outcomes and variance-reduction tricks like Duplicate poker (hand mirroring) and Game Theoretic Evaluation to produce statistically sharp rankings that don’t saturate as models improve.
Static LLM benchmarks like MMLU and GSM8K have two well-known problems: frontier models are approaching ceiling scores, and test items leak into training data. The response has been dynamic evaluation, notably Chatbot Arena and the LLM-as-a-Judge approach popularized by MT-Bench. Those fix saturation but introduce their own noise: human and LLM judges are subjective, verbose-answer biased, and sycophantic.
Games give you the best of both. Each match generates fresh trajectories (hard to memorize) and produces objective outcomes (wins, losses, chip counts). Prior LLM-game benchmarks exist (GTBench, PokerBench, SmartPlay) but tend to be one-off datasets with modest sample sizes. If you want to compare, say, GPT-5.2 vs Claude Opus 4.5 with confidence, a few thousand game instances is not enough for high-variance domains like poker. Game Arena is the infrastructure play: standardized harness, huge sample counts, and room to add new games without invalidating past results.
Every game uses the same shape: at each decision point, the model gets a text description of state and history, and must return a single well-formatted action. Illegal outputs trigger a small number of retries with minimal feedback. No move lists are given as hints, so the model has to know the rules. Then domain-appropriate metrics turn raw outcomes into rankings.
The three pilot environments cover distinct information structures:
•
Chess (perfect information, deterministic). Model sees the board in Forsyth-Edwards Notation and move history in Portable Game Notation, must return a legal move in Standard Algebraic Notation. 40 games per model pair with color balancing. Ratings come from a Bradley-Terry loss Elo fit, calibrated externally against Stockfish skill levels.
•
Poker (imperfect information, stochastic). Heads-up no-limit Texas Hold’em, 20,000 hands per pair, 900,000 hands total across the ten-model round-robin. The key variance-reduction trick is Duplicate poker (hand mirroring): each 100-hand deck is played twice with players swapping seats and hole cards. Metric is BB/100. Both players’ hole cards are revealed after every hand to accelerate opponent-modeling within an episode.
•
Werewolf (imperfect information, multiplayer, natural-language). Eight players, four roles (Werewolf, Seer, Doctor, Villager). Uses a ReAct-style harness with a private-reasoning field and a public-action field. Because raw win rates confound skill with lucky role assignment, ranking uses Game Theoretic Evaluation, which decomposes each model’s contribution role-by-role.
The evaluation loop, roughly:
for pair in round_robin(models):
for episode in range(n_episodes[game]):
state = env.reset(seed=episode)
while not state.terminal:
for player in state.active_players:
prompt = render(state, history, role=player.role)
action = player.model(prompt, retries=k)
state = env.step(action)
log(trajectory, outcome, reasoning_traces)
rank = domain_metric(all_outcomes) # Elo, BB/100, or GTE
Ten models from five labs were evaluated (OpenAI, Anthropic, Google, xAI, DeepSeek) via public APIs with default sampling, no fine-tuning or tools.
Cross-game skill is not consistent. The headline observation is that being good at one game does not predict being good at another. Gemini 3 Pro Preview tops Chess and Werewolf but posts a negative -15.2 BB/100 in poker. GPT-5.2 wins poker convincingly (+46.6 BB/100, the only model beating every opponent head-to-head) but sits mid-pack in Chess Elo. Claude 4.5 variants are last in Chess but middle-tier profitable in poker.
Chess results stratify sharply. In Chess Text, Gemini 3 Pro Preview and Gemini 3 Flash Preview lead by roughly 300 Elo over the next tier (o3, GPT-5.2). Claude variants finish at the bottom. Using Stockfish annotations, the authors localize where the gap opens: models are roughly balanced in the opening, then diverge in the middlegame. Weaker models also produce more “rethinks” (illegal or malformed moves) as games progress, correlating strength with reliability. A Chess Opening variant that forces play from 20 randomized two-ply openings produces nearly identical rankings, suggesting results are not driven by memorized opening books.
Poker reveals style, not just strength. Preflop statistics like VPIP range from 55% to 96% across models, but VPIP alone does not predict profit: GPT-5.2 and GPT-5 mini both open extremely wide, yet their win rates differ by over 140 BB/100. The authors read this as postflop decision-making being the main differentiator.
Werewolf shows role-specific weaknesses. Under GTE, Gemini 3 Pro and Flash top the board with positive contributions in both informed-minority (Werewolf) and uninformed-majority roles. Claude Sonnet 4.5 and Grok 4.1 Fast Reasoning post negative contributions specifically as the Seer, which the authors interpret as difficulty conveying private information without arousing suspicion. Traditional Elo and OpenSkill produce the same ordering but with wider confidence intervals; GTE separates the top models more cleanly. Common heuristic metrics (survival rate, identification precision, voting success) turn out to have too little variance to be useful discriminators.
•
If you’re picking a model for a strategic task, do not extrapolate from one game. The paper’s cleanest single finding is that Chess, Poker, and Werewolf reorder the leaderboard differently, including flipping which models sit on the cost-performance Pareto frontier. Test on a task class that resembles yours, not on a generic “reasoning” benchmark.
•
If you’re building an LLM evaluation, the sample-size lesson is concrete. PokerBench and PokerBattle reported roughly 2,000 and 3,800 hands; Game Arena uses 180,000 per model and still shows overlapping confidence intervals within tiers. High-variance domains need dramatically more interactions than static benchmarks to produce statistically defensible rankings, and duplicate/mirrored formats are worth borrowing.
•
If you want to reproduce or extend this, the harness and full trajectories are released. The game_arena repo and the Kaggle Game Arena site host the code and data. New games can be added without invalidating prior results, which is what makes this a platform rather than a one-shot benchmark. •
Worth testing but not established here: whether the observed cross-game skill divergence would hold with tool use, chain-of-thought scaffolding, or fine-tuning. All models here were run with default API settings and no augmentation. The paper does not claim these rankings persist under agentic wrappers.
•
Only ten models from five labs are evaluated, chosen as of February 2026. The frontier moves fast and the authors flag “model churn” as an open problem for longitudinal comparability.
•
Rankings within a tier often have overlapping bootstrap intervals, especially in poker’s middle tier and among the bottom-tier losers. “Model A beats Model B” claims should be checked against the confidence intervals, not point estimates.
•
No consolidated cross-game rating exists yet. Each game reports its own primary metric, and the authors explicitly note that a Bradley-Terry-style meta-rating across heterogeneous games is future work.
•
Werewolf’s balanced ruleset was calibrated via self-play with a single strong model (Gemini 3 Pro Preview); the authors acknowledge this is an empirical proxy for true game balance, not a Nash equilibrium computation.
•
Compute is allocated uniformly across matchups rather than adaptively toward close matches, so some head-to-head confidence intervals stay wider than necessary.