SpyRL turns open-ended tasks (writing, summarization) into a Who Is the Spy? game where one agent gets degraded input and peers vote to spot it, so RL with verifiable rewards-style rule-based rewards emerge from vote outcomes, without any external judge or reward model.
You’ve shipped an assistant that writes marketing copy or executive summaries. To improve it with RL, you need a reward. Today you either pay for human preferences, run an expensive LLM-as-judge on every rollout, or write a rubric grader. All three cap your model at the judge’s taste and add per-step inference cost.
The dominant workaround for non-verifiable tasks has been Reinforcement Learning from Human Feedback and its cousins (DPO, LLM-as-judge, rubric rewards). This paper takes a different route inspired by self-supervised learning: don’t approximate the missing quality score, transform the task into one where a rule can score it exactly.
The core move is a task transformation. Instead of asking “is this summary good?” (unverifiable), the environment sets up a small game with a hidden fact it knows, then asks “which player is the spy?” (verifiable by lookup). If the game is designed right, winning the game requires doing the underlying task well.
Concretely in SpyRL: sample 5 players. Pick one at random to be the spy. Civilians see the full input document; the spy sees a version with a chunk masked out (20% for summarization/writing, 40% for math). All 5 do the same task on their private view, producing public outputs. Then all 5 vote on who they think the spy is. The spy’s identity is known to the environment, so votes are graded exactly.
Two reward streams flow from this:
•
Detection reward: 1 if you voted for the actual spy, 0 otherwise, then normalized Group Relative Policy Optimization (GRPO)-style within the group.
•
Performing reward: zero-sum. If the spy attracts most suspicion votes, civilians win; if a civilian gets scapegoated, they lose. This pressures civilians to produce outputs clearly better than the spy’s, and pressures the spy to hide by matching civilian quality despite missing information.
Because the spy has strictly less information, its outputs are structurally weaker on the task, so “who looks weakest” becomes a proxy for “whose output is worst”. A Role-Advantage Estimation (RAE) baseline corrects for the fact that being spy is inherently harder.
for epoch in range(T):
x = sample(D); u = randint(1, n) # spy index
obs = [x if i != u else degrade(x) for i in range(n)]
outputs = [performer(o) for o in obs]
votes = [detector(obs[i], outputs) for i in range(n)]
r_detect = [1 if v == u else 0 for v in votes]
r_perform = zero_sum_from_votes(votes, spy=u)
if detector_saturated(): update(performer)
else: update(detector)
Performer and detector are updated alternately, not jointly, to avoid the coupled-noise problem where both policies chase each other’s mistakes.
The prevailing view is that open-ended tasks need a learned or model-based judge because there’s no ground truth to check against. This paper shows the opposite. Verifiability isn’t a property of the task, it’s a property of the environment you wrap around it. Inject a latent variable (which player is the spy) that the environment itself assigns, and any rule computed against that latent becomes a verifiable reward, even for creative writing. The load-bearing evidence is the vote-quality correlation: players who receive more suspicion votes are independently ranked as lower quality by GPT-4o, confirming the vote signal actually tracks output quality.
The finding that makes the thesis credible: in 100 games each on WritingPrompts and GovReport, the number of suspicion votes a player receives correlates monotonically with GPT-4o’s independent quality rank (1=best, 5=worst). More votes, worse output. This is what licenses treating votes as a reward.
Given that, the headline numbers on Qwen3-4B and Qwen3-8B:
•
Creative writing A/B vs base: SpyRL wins ~75–81% of pairwise comparisons across novelty, emotion, coherence, consistency. R-Zero and Absolute Zero barely move the needle (~46–58%).
•
Summarization ROUGE-L: +4.56 and +4.04 average points across 5 benchmarks for the two model sizes.
•
Math reasoning (verifiable, so a sanity check): SpyRL still helps, e.g. +8.4% average across 5 math benchmarks on Qwen3-4B, and AIME25 jumps 6.7 → 20.0.
•
Vs rubric-as-reward baselines: SpyRL beats a Qwen3.5-27B rubric grader (~56–59% win rate) and ties a GPT-4o rubric grader, while using $0 in external verifier cost vs ~$200 and ~$900 respectively.
•
Ablation: freezing detection, freezing performing, or removing the spy asymmetry all plateau around Math500 = 70–72%; the full two-stage loop reaches 79.5%.
•
Group size: gains grow from n=3 to n=5, flatten at n=6–8, so 5 players is the sweet spot.
•
Human eval: 10 PhD annotators prefer SpyRL over baselines 74–85% of the time, matching the GPT-4o judgments.
Reach for this when you’re training a model on a task where you don’t have a cheap automatic grader: summarization, drafting, style-matching, structured writing. Instead of paying an LLM judge on every rollout, run a small group of copies of your policy, hide part of the input from one of them, have them all produce outputs and vote. The vote is your free reward. Your inference cost is n rollouts per prompt (n=5 works) plus one vote call each, and you never touch an external verifier.
The authors release models and code at GitHub. The information-degradation operator is task-specific (span masking for text, context removal for math) and the paper gives templates for all three domains. Training uses Group Relative Policy Optimization (GRPO) via the verl framework on Qwen3-4B/8B; a single 8-GPU node for 100 iterations was enough for the reported results.
Verifiability is a design choice, not a task property. If you can hide a fact the environment knows and make recovering it depend on task quality, you have a rule-based reward for anything.
•
The information-degradation operator is hand-designed per task. If your degradation removes stylistic cues instead of task-relevant content, detectors will exploit the shortcut and the vote signal decouples from quality. The paper is explicit that g(·) must preserve style, length, and theme.
•
The reward is relative: models learn to be better than their peers in the same group, not better in some absolute sense. On tasks where all n copies of a weak policy make the same mistake, no learning pressure exists to fix it.
•
Cross-task transfer is limited to tasks with overlapping capabilities. A model trained on math via SpyRL does not transfer to creative writing (win rate ~40%), so you still need a separate training run per capability cluster.
•
Results are shown on Qwen3-4B/8B only. Whether the vote signal stays well-aligned with quality on much smaller or much larger models, or on domains without a clean “missing information” degradation (e.g. tone, persuasion), is not tested.