J-Zero trains three copies of a model (task-maker, task-solver, task-judge) against each other with no external data. The judge learns from preference pairs whose winner is fixed by construction, letting all three improve for 10+ iterations where prior zero-data methods plateau after 2.
Suppose you’re post-training a model on open-ended work: writing product copy, drafting internal docs, giving planning advice. There’s no unit test that says the output is correct, so you reach for a reward model or an LLM-as-judge. The judge’s own taste becomes the ceiling: once your policy learns everything the judge can distinguish, Group Relative Policy Optimization (GRPO) runs out of signal and quality drifts. Prior zero-data self-play like R-Zero sidesteps this by using majority-vote rewards, but that only works on verifiable tasks like math. This paper asks: can the judge itself keep learning, inside the same loop, without any human labels?
Each iteration runs three roles built from the same base model. The Challenger writes tasks. The Solver answers them. The Judge scores answers. Challenger and Solver play a minimax game trained with Group Relative Policy Optimization (GRPO): the Challenger is rewarded when the Solver scores badly (plus penalties for duplicate or malformed tasks), the Solver is rewarded when it scores well. Standard adversarial setup so far. The real move is how the Judge trains without becoming a hall of mirrors. The authors build preference pairs where the winner is known by construction, not by asking the Judge:
•
Role-asymmetry pairs: for the same task, the Solver’s answer beats the Challenger’s answer, because only the Solver was trained to answer well. The Challenger was trained to pose hard problems, so its own attempt at answering is systematically weaker.
•
Subtask-amplification pairs: the Challenger decomposes a task into subtasks, the Solver answers each, the Challenger stitches them together. This composed answer beats the Solver’s one-shot answer, borrowing the Iterated Amplification argument that subtasks are easier than wholes.
The Judge is then updated with a Bradley-Terry loss loss on the union of these pairs.
for iteration in range(N):
tasks = challenger.sample()
responses = solver.sample(tasks)
scores = judge.score(tasks, responses)
challenger.grpo_update(reward = 1 - mean_score - repetition_penalty)
hard_tasks = top_k_by_score_variance(tasks, scores)
solver.grpo_update(hard_tasks, reward = judge.score)
D_role = [(x, solver(x), challenger_answering(x)) for x in held_out]
D_amp = [(x, compose(subtask_answers), solver(x)) for x in held_out]
judge.bt_update(D_role + D_amp)
One design detail worth flagging: tasks fed to the Solver are filtered by response-score variance, not by intermediate accuracy. Tasks where the Solver’s rollouts disagree are the ones with the most room to learn.
The usual instinct in judge-based self-training is to use the Judge’s own scores to label chosen/rejected pairs, then train on those. That’s what self-rewarding methods do, and it silently reinforces whatever biases the Judge already has. This paper shows the opposite: get your preference labels from how the response was produced, not from what the current Judge thinks of it. Roles and decomposition give you ordering guarantees the Judge can’t have invented. The evidence that matters is the sustained-improvement curve, not the headline averages: frozen-Judge ablations track J-Zero for three iterations then flatline, exactly at the point the Solver hits the Judge’s ceiling.
•
The load-bearing result is the iteration curve. Baselines R-Zero and G-Zero peak at iteration 2 and decline. J-Zero climbs monotonically through iteration 10 in both verifiable and unverifiable domains. Freeze the Judge and it plateaus after iteration 3, ending 1.66 and 4.44 points below the full run. That gap is the mechanism.
•
Aggregate lifts over the base model: +9.47 points on verifiable, +11.23 points on unverifiable (Qwen3-4B). Versus the strongest prior baseline, +4.2 points verifiable, +8.0 points unverifiable on average.
•
Preference-label reliability, checked by Claude Opus 4.8 as an external judge: role-asymmetry pairs are correct 60–88% of the time throughout training. Subtask-amplification pairs start unreliable (21% correct at iteration 1, decomposition doesn’t help a weak Solver) but cross 50% by iteration 4 and reach 70–80%. The two sources hand off cleanly.
•
Ablations: dropping subtask-amplification pairs costs 1.64 overall points, dropping role-asymmetry costs 0.97. Both matter, amplification more.
•
As a side effect, the co-evolved Judge also improves on RM-Bench (+1.34 average), with the gain concentrated on Hard pairs (+4.77) where style and quality disagree.
Reach for this pattern when you’re building a domain-specific reward model for open-ended output and you can’t collect fresh human preferences every week. Instead of asking annotators, generate pairs where you already know the winner: a strong pipeline’s output beats a weak pipeline’s output on the same prompt, a decomposed-and-recomposed answer beats a one-shot answer from the same model. Train the reward model on those. You get a reward model whose ceiling rises with your policy instead of capping it.
The paper doesn’t mention a code release or model weights. Implementation runs on the verl framework with Qwen3-4B-Base and Qwen3-8B-Base as the shared Challenger/Solver init, and Skywork Reward V2 as the Judge init. The training recipe (5 Challenger steps, 15 Solver steps, 8 Judge steps per iteration, equal mix of role and amplification pairs) is fully specified in the appendix.
Label your preference pairs by construction, not by asking the judge you’re trying to train. A judge trained on its own scores can only sharpen its existing biases; a judge trained on pairs where the winner is fixed by how the response was produced can actually learn something new, which is what keeps a self-play loop from plateauing.
•
Everything is at 4B–8B scale with base models. The authors explicitly flag that long-chain-of-thought reasoning models and larger scales are untested, and the ordering guarantees might weaken if the Challenger becomes competent enough to answer its own hard tasks well.
•
Subtask-amplification pairs are wrong for the first three iterations (21% correct at iteration 1). The scheme works because role-asymmetry pairs cover that early window. Take away either source and you likely lose either early stability or late-stage headroom.
•
The Judge is a discriminative reward model, not a generative critic. Whether the same construction-based labeling works when the Judge produces natural-language critiques (and how you’d even define “chosen by construction” for free-form critiques) is open.