RISE trains an LLM by having it distill from a synthetic “future self” built by extrapolating recent RL with verifiable rewards parameter (or logit) updates past the current checkpoint, then teaching the current model to match that projected teacher, turning a sparse outcome reward into dense per-token supervision at 1.3–1.6× wall-time and no extra sampling.
You’re post-training a coding or reasoning model with reinforcement learning from a grader (unit tests pass or fail, answer matches or doesn’t). The reward is one scalar per rollout, so gradient updates can’t tell you which specific tokens in a 2000-token chain-of-thought were the good ones. Practitioners have tried patching this with On-Policy Distillation: a stronger teacher gives a full next-token distribution at every position, so the student learns how to fix each token, not just whether the whole answer worked.
The catch is finding a teacher. A bigger external model drifts off-distribution once your student explores its own reasoning styles. The workaround, feeding the same model a known-correct solution as a hint and calling that the “teacher” (the On-Policy Self-Distillation recipe used by baselines like GRPO+SDPO), is bounded by how well the model can actually use that hint in-context. This paper attacks the teacher problem itself instead of tolerating a bad one.
The idea is to build the teacher out of the training run itself. Take two checkpoints: an earlier anchor θ and the current checkpoint θ’ right after an RLVR gradient step. The vector θ’−θ points in the direction training is currently improving. Extend that vector past θ’ by a factor β>1 and you get a hypothetical “future” checkpoint that has taken a bigger step in the same direction. Use that as the teacher and distill it back into your current model.
The paper does this in two flavors. Weight-space: literally add β·(θ’−θ) to the parameters and forward-pass through the resulting model to get teacher logits (this is Task arithmetic with β>1 instead of the usual interpolation). Logit-space: cache the log-probabilities from both checkpoints on your rollouts and extrapolate the logits directly, which works out to a geometric mixture proportional to π_θ’^β / π_θ^(β−1). Logit-space is cheaper because there’s no third model to hold in memory; weight-space is more faithful because logit-space is only the first-order Taylor approximation of it.
Crucially, RLVR does the actual capability discovery and OPD only refines. If you extrapolate without RLVR grounding the direction, you amplify whatever noise the model produced. If you skip OPD and just adopt the extrapolated weights as your next policy, you overshoot. OPD acts as a trust-region projection back toward the safer post-RLVR checkpoint. The paper also justifies the extrapolation empirically: measured on their own runs, three principal directions capture ~87% of the RLVR trajectory’s variance, so the path really is nearly linear over short spans.
anchor = theta_0
for n in range(N):
beta = 1 + (beta_0 - 1) * (1 - n/N) # decay toward 1
rollouts = sample(pi[theta])
theta_prime = grpo_update(theta, rollouts) # RLVR phase
# teacher = extrapolated future self
log_pi_future = log_pi[anchor] + beta * (log_pi[theta_prime] - log_pi[anchor])
for batch in rollouts: # OPD phase
theta = optimizer_step(theta, JSD(pi[theta], stop_grad(pi_future)))
anchor = (1 - eta) * anchor + eta * theta # EMA anchor
One detail worth flagging: β decays toward 1 over training, because the safe extrapolation range shrinks as the policy approaches its optimum. Late in training an aggressive β would fly past the true optimum and poison the teacher.
The usual move when your RL signal is too sparse is to import a teacher from outside, a bigger model, or the same model conditioned on a hint. Both accept that the teacher is a fixed, imperfect object. This paper shows the opposite. The training trajectory itself already contains a better teacher than any external one: extrapolate the model’s own recent update to project a “future self,” and distill that back in. Because the teacher is rebuilt every iteration from the student’s latest step, distillation stops being a one-shot compression and becomes a recursive improvement loop with no fixed capability ceiling. The load-bearing evidence is not the headline lift but the ablation that removes RLVR grounding and watches training collapse to 2.4% accuracy within 60 steps.
The finding that makes the thesis true: extrapolation only helps because RLVR anchors the direction. Strip out RLVR and let self-distillation alone define the update vector, and within 60 steps MATH-500 accuracy crashes from baseline to 2.4%, generation length blows up to the 8K context cap, and reward hits zero. Strip out OPD instead and just adopt the extrapolated weights as the next policy, and you get essentially GRPO performance (60.0 → 60.3 on Qwen3-8B math). Both phases are load-bearing; neither alone suffices.
On the headline benchmarks, that mechanism translates into:
•
On Qwen3-8B trained on DAPOMath, RISE lifts math average from 60.0 to 62.7 over GRPO, with AIME’24 going 54.4 → 58.1.
•
On OLMo3-7B on OpenR1-Math, the lift is larger: math average 47.6 → 56.4, with AIME’24 jumping 30.2 → 46.9 (+16.7).
•
On multi-turn agent tasks (ALFWorld and WebShop) with Qwen2.5-3B, weight-space RISE beats GRPO by +9.4 on ALFWorld and +10.9 on WebShop accuracy.
•
The three privileged-teacher OPSD baselines (GRPO+SDPO, SDAR, RLSD (Self-Distilled RLVR)) mostly stay within ~2 points of plain GRPO, sometimes worse, consistent with the paper’s claim that in-context privileged teachers are capped by the model’s own ICL ability.
•
Out-of-distribution benchmarks (GPQA, IFEval, MMLU-Pro) hold or slightly improve, so this isn’t over-specialization to the training distribution.
•
pass@16 improves alongside avg@16, especially on the 1.7B model (+9.6 vs +6.3 on AIME’24), meaning the teacher expands the set of solvable problems, not just sharpens existing ones.
Sensitivity is mild: β₀ in [1.2, 1.5] all work, though β₀=2.0 diverges because linear decay can’t shrink β fast enough once the policy nears optimum. The safe range for β contracts visibly over training, matching the theory.
Reach for this if you’re already running Group Relative Policy Optimization (GRPO) or a similar RLVR loop on a reasoning or agent model and feel the credit-assignment ceiling: the reward tells you the trajectory worked, but you can see the model burning tokens on dead-end reasoning it should learn to skip. RISE plugs into an existing GRPO pipeline as a second gradient phase on the same rollouts (no extra sampling, ~1.3–1.6× wall time), and you don’t need to procure a stronger teacher model or curate hint-conditioned prompts. The two knobs that matter are β₀ (start at 1.2, decay linearly to 1) and the anchor EMA rate η (0.1 for Qwen-style models where per-step updates are noisy, 1.0 for models with large stable steps like the OLMo run here).
The paper implements on top of the verl framework, but doesn’t advertise a code release in the text provided. Datasets used (DAPOMath, OpenR1-Math, Skywork-OR1-Code, ALFWorld, WebShop) are all pre-existing public benchmarks, so the recipe is straightforward to reproduce on your own RLVR setup.
Your model’s own recent training step, amplified, is a better teacher than anything you can import from outside. This only works because a verifier grounds the direction; without outcome rewards, extrapolating the model’s self-generated update just amplifies noise into collapse. The teacher is free, refreshes every iteration, and has no fixed ceiling.
•
Linearity of the training trajectory is load-bearing. The paper measures ~87% of variance in three directions on their runs, but that’s for GRPO on math-heavy corpora at 1.7B–8B scale. If your post-training trajectory is more curved (different objective, much longer training, larger model), the safe β range shrinks and the extrapolated teacher can hurt.
•
The method inherits and amplifies any bias in the reward. If your verifier is hackable, extrapolation pushes harder along the hackable direction, not just the correct one. RISE has no built-in defense against reward exploitation.
•
Gains on code generation were noticeably more modest than on math, and the paper attributes this to benchmark saturation, but it also means the mechanism’s payoff depends on there being real headroom for the RLVR direction to point into. Near-saturated tasks won’t benefit much.