Cliff turns a rollout into a correct prefix plus an incorrect suffix by asking a teacher LLM to locate the first mistake, then hands positive Group Relative Policy Optimization (GRPO) advantage to the prefix and negative advantage to the suffix, beating vanilla GRPO by ~7% and On-Policy Distillation by ~15% across 12 settings.
You’re training a reasoning model with RL and a grader that only checks the final answer. A rollout that was 90% right and slipped on the last step gets the same zero as pure gibberish. That’s the standard RL with verifiable rewards setup, and it wastes most of the signal inside each trajectory.
The two usual fixes both come with strings attached. Process reward models need their own training data and tend to get gamed. On-policy distillation only works cleanly when teacher and student share a tokenizer and reasoning style. Cliff sidesteps both: it uses an off-the-shelf LLM as a per-rollout judge, not a trained reward head or a distillation target.
The core intuition: once reasoning goes off the rails, everything after that point is conditioned on a broken premise, so grading it token by token is mostly noise. You only need to find where it first broke. The authors call that boundary the Pitfall Step.
The procedure runs in two stages per training query. First, the teacher solves the problem itself; the automatic verifier checks that solution, and if the teacher got it wrong, that group falls back to plain GRPO. Second, for each student rollout in the surviving groups, the teacher reads the rollout against its own verified solution and (a) says whether the student’s final answer is right, (b) if wrong, names the sentence index where reasoning first became invalid.
That boundary gets converted to token-level advantages. In standard GRPO, every token in a rollout shares one advantage: a positive value A_cor for correct rollouts, a negative A_inc for incorrect ones. Cliff keeps this for correct rollouts, but splits incorrect ones: tokens before the Pitfall Step get λ · A_cor (with the paper using λ=0), tokens from the Pitfall Step onward get the negative A_inc. A constant b is subtracted everywhere to keep the group’s mean advantage at zero.
for group in batch:
ref = teacher.solve(group.query)
if not verifier.check(ref):
apply_grpo(group); continue
for rollout in group.rollouts:
correct, pitfall = teacher.judge(rollout, ref)
if correct:
adv[rollout] = A_cor - b
else:
adv[rollout, :pitfall] = lam * A_cor - b # valid prefix
adv[rollout, pitfall:] = A_inc - b # erroneous suffix
Why λ=0 and not something positive? A large λ rewards any long correct-looking prefix, which lets the model pad reasoning to inflate the prefix without ever reaching the answer. The appendix works out the length-hacking condition and shows λ=0 removes it; the paper also hard-caps p(a)=0 for truncated rollouts so overlong rambles don’t get rewarded as “valid so far.”
The prevailing move for adding process supervision is to score every step, either through a trained Process Reward Model or through token-level teacher distributions. This paper argues the opposite. You don’t need dense per-step scores; you need one cut at the first mistake, because everything after a broken premise carries no usable signal anyway. The load-bearing evidence is the “GRPO with teacher” ablation, which uses the same teacher but keeps a single rollout-level advantage. It barely moves over plain GRPO, so the win is coming from the prefix/suffix split, not from having a smarter judge in the loop.
•
The ablation that isolates the mechanism. Adding the teacher without the Pitfall Step split (“GRPO with teacher”) gives only marginal gains over plain GRPO, while full Cliff produces substantially larger gains under the same teacher. Credit assignment is doing the work, not the extra LLM call.
•
Headline numbers. Averaged over 12 student × teacher × domain configurations, Cliff beats GRPO by ~7% and beats on-policy distillation by ~15%. On Qwen3-4B with a SOTA teacher, math average goes from 61.68 → 65.66; on Phi-4-mini with the same teacher, 49.78 → 51.73. Coding shows the same direction on CodeContests, LiveCodeBench, and DeepCoder.
•
Judges don’t need to be frontier models. In the calibration study, Qwen3-32B and Gemma3-27B solve the math problems less accurately than a SOTA teacher, but their agreement with human annotators on where the first mistake occurs is still usable (~88% judge accuracy, average pitfall-position distance around 3 sentences). Judging is easier than solving.
•
Ground-truth filter matters more for weak teachers. Dropping the step that discards queries the teacher gets wrong costs strong teachers almost nothing but costs weaker teachers ~2%. Even without the filter, Cliff still beats GRPO.
•
λ sweep confirms the theory. λ=0 gives the best accuracy at reasonable length; λ=1.0 blows up response length (1279 → 1959 tokens) and hurts accuracy, matching the length-hacking analysis.
Reach for this when you’re doing RLVR post-training on a reasoning model and your reward is a pass/fail verifier (unit tests, math answer checker). Instead of building a process reward model, wire in any competent off-the-shelf LLM as an offline judge that emits one integer per failed rollout: the sentence index of the first mistake. Feed that into your existing GRPO loss with the prefix/suffix advantage split. The teacher doesn’t need to match your student’s tokenizer or family, which is the constraint that usually kills on-policy distillation.
The paper reports experiments on DAPO-Math-17k for math and Deepcoder for code, students are Qwen3-4B-Base and Phi-4-mini-Instruct, teachers are Qwen3-32B, Gemma3-27B, and an unnamed SOTA model. Training is built on verl. The paper does not link a code release in the text provided.
When outcome rewards are too sparse, don’t score every step, just find the first broken one. Everything after that point is downstream of an invalid premise and isn’t worth grading; everything before it is genuine progress and deserves credit even though the rollout ultimately failed. That single cut is most of what “process supervision” needs to be.
•
The judge has to be good enough that its Pitfall Step actually tracks the real first mistake. The case studies show the teacher sometimes flags a step that a human considers correct-but-insufficient, which would push false negative signal into training on harder problems.
•
The setup assumes an automatic verifier exists to filter teacher solutions and to define correctness. Domains without one (open-ended writing, multi-turn agent tasks) are outside what’s tested; the authors mention agentic settings only as future work.
•
λ=0 is the safe choice against length hacking, but it also means the valid prefix of a failed rollout gets no positive reinforcement, only a smaller penalty than the suffix. On very weak students where almost every rollout fails, this may leave little positive gradient to learn from.