SP³O trains the PPO critic on only a few well-spaced token positions per response instead of every token, avoiding an implicit variance penalty that otherwise flattens value predictions across a reasoning trajectory.
When you train an LLM with reinforcement learning on long reasoning traces, the standard recipe uses PPO with a critic network. The critic’s job is to predict, at each token, how likely the model is to eventually get the right answer from that point onward. Those per-token predictions get turned into advantages, which tell the policy which tokens deserve credit or blame. If the critic is any good, credit assignment improves and updates become less noisy than critic-free alternatives like Group Relative Policy Optimization (GRPO).
The authors find the critic is quietly broken. They call the failure Value Flattening: as the model reasons through a response, the true probability of success (measured by running many rollouts from each intermediate state) swings up and down sharply, but the critic’s predictions stay nearly flat, sometimes even moving the wrong direction. So the critic can still tell good responses from bad ones on average, but within a single response it gives almost the same score to every token. That defeats the whole point of having per-token advantages.
They confirm this isn’t an LLM-only artifact by reproducing it in a stochastic FrozenLake gridworld: as the maze grows, the critic’s value map becomes visibly smoother and less accurate.
The diagnosis has two parts. First, an implicit variance penalty. In the common setup where the reward is only given at the end of the response (terminal reward, no discounting), every token in a response gets supervised with the same number: the final reward R. The mean-squared-error loss over all T tokens algebraically splits into two pieces: how far the average prediction is from R, plus the variance of predictions within the response. Minimizing the second piece literally pushes all per-token predictions toward each other. Dense per-token supervision therefore actively flattens the value profile.
Second, redundant updates from temporal correlation. Adjacent LLM states differ by exactly one appended token, so their hidden representations sit very close together. The authors measure gradient cosine similarity between nearby positions and show it stays high, decaying with token distance. Supervising all T positions is thus much less informative than T independent updates, and it further reinforces the flattening.
The fix, SP³O, is one line: keep everything about PPO the same (actor loss, rollouts, GAE targets, the critic even still produces values at every token), but only apply the critic MSE loss at a small set I(τ) of well-separated anchor positions per response.
for tau in batch:
values = critic(tau.states) # still all tokens
G_hat = compute_gae_targets(tau)
anchors = [0.3, 0.6, 0.9] * tau.length
if tau.length >= 6144:
anchors.append(0.95 * tau.length)
critic_loss += sum((values[t] - G_hat[t])**2 for t in anchors)
critic_loss /= total_anchor_count
Default is three anchors at 30%, 60%, and 90% of the response, plus a 95% anchor for responses over 6144 tokens.
On Qwen3-Base (4B and 8B) trained on DAPO-Math-17k, SP³O beat both PPO and Group Relative Policy Optimization (GRPO) on math and out-of-distribution reasoning suites.
•
On Qwen3-4B-Base, in-domain math average (7 tasks including AIME, MATH500, OlympiadBench): base 17.95, PPO 37.60, GRPO 39.26, SP³O 45.57. That’s +7.97 points over PPO.
•
On the OOD suite (six tasks including MMLU-Pro, GPQA, BIG-Bench Hard (BBH), ZebraLogic-Grid): PPO 51.95 vs SP³O 59.28 on the 4B. On the 8B, PPO 64.38 vs SP³O 66.37.
•
Critic-quality check: measuring MSE between critic predictions and Monte Carlo value estimates from repeated continuations, SP³O reduces MSE by 36%, 11%, and 21% at 30%, 60%, and 90% response progress.
•
Effective rank of the critic’s hidden-state matrix (a proxy for how much dimensional diversity the value head sees) rises from a median of 4.33 under PPO to 5.63 under SP³O, evidence the representations aren’t collapsing.
•
Ablation on anchor count K: performance peaks around K=3, stays reasonable through K=8, and collapses back to the PPO baseline at K=16 and K=64. More supervision is worse.
•
Placement ablation at K=3: random placement (36.59%) underperforms even PPO, while fixed schedules at 0.2/0.5/0.8 (44.65%) and 0.3/0.6/0.9 (45.57%) both win. Spacing matters, not just count.
•
Tail anchor: removing the 0.95 anchor drops accuracy from 45.57 to 44.10 and, more strikingly, sends repetition rate from 1.12% to 18.33%. Explicit coverage of the response end appears to prevent degenerate looping.
The authors interpret these together as evidence that sparse, spaced supervision preserves within-response value resolution without giving up response-level discrimination, and that the improved critic then produces smoother, more stable actor updates (they show smaller within-iteration policy changes during training).
•
If you run PPO on long reasoning traces with terminal-only rewards, the value-flattening diagnosis probably applies to your setup too. It’s worth logging a Monte Carlo value estimate at a handful of intermediate states (repeat continuations from the same prefix, average the terminal reward) and comparing it to your critic’s prediction. If the critic’s within-response variance is tiny compared to the MC swings, you have the same failure mode.
•
If you do, sparse critic supervision is cheap to try: it changes only which token positions contribute to the value loss. The actor loss, rollout code, and GAE computation stay identical. Start with three anchors at 0.3/0.6/0.9 of the response and add a 0.95 anchor for long responses, since the paper shows the tail anchor specifically suppresses repetition.
•
If you’re choosing between PPO and critic-free methods like Group Relative Policy Optimization (GRPO) for reasoning tasks, this paper is a reason to reconsider PPO rather than abandon it. The evidence here is that a poorly-resolved critic (not the critic architecture itself) is what makes vanilla PPO underwhelming on long reasoning.
•
Scope limits: results are on Qwen3-Base 4B and 8B, on math training data, with binary terminal rewards and no KL penalty. The mechanism argument (variance penalty from shared terminal targets) should extend to other terminal-reward setups, but shaped or per-step rewards break the derivation and the paper doesn’t test them. Code is at GitHub.
•
All training uses one dataset (DAPO-Math-17k) and one model family (Qwen3-Base). Whether the same anchor schedule transfers to code, agentic, or dialogue RL is untested.
•
The default anchor schedule (0.3/0.6/0.9 plus conditional 0.95) is hand-picked. The ablation shows placement matters a lot, and random placement is actively worse than dense PPO, so this isn’t a knob you can set carelessly.
•
The analysis assumes terminal-only reward with γ=λ=1. The clean loss decomposition into mean-fit plus variance penalty depends on every token sharing the same target R. Under dense process rewards or nonzero KL shaping, the argument for why sparsity helps weakens.
•
Gains on the 8B are smaller than on the 4B (roughly +2 in-domain, +2 OOD versus +8 and +7). The paper doesn’t scale beyond 8B, so it’s unclear whether the effect shrinks further at larger sizes.
•
The paper shows SP³O improves several critic-quality metrics and downstream accuracy simultaneously, but does not isolate which of the two proposed causes (implicit variance penalty vs. redundant temporally-correlated updates) contributes more. Both mechanisms are addressed by the same intervention.