SA-MRPO trains reasoning LLMs against multiple rewards by downweighting each objective in proportion to how close its batch-average reward is to its ceiling, so gradient budget flows to the still-unsolved objective. Up to +9.2 pp accuracy on AMC23 when a length reward has saturated.
Say you’re post-training a coding assistant with RL with verifiable rewards. You want three things: the code compiles, it passes tests, and it stays under a length budget. You hand the trainer a weighted sum of those three rewards. Two weeks in, compile-rate is basically 100%, length is fine, but pass-rate is stuck. The trainer is still spending a third of every gradient step reinforcing “the code compiled,” which the model already nails.
The dominant recipe here is Group Relative Policy Optimization (GRPO) and its multi-reward extension Group Reward-Decoupled Policy Optimization (GDPO). GRPO sums rewards then normalizes within a rollout group. GDPO fixes one problem (it normalizes each reward dimension separately so a good-on-A-bad-on-B rollout isn’t confused with a mediocre-on-both rollout), but it still uses fixed relative weights across training. SA-MRPO is the piece on top of GDPO that makes those weights move.
The core observation: for verifiable rewards like “compiles” or “under 4000 tokens,” you know the maximum score. So you can measure, per batch, how much of that ceiling the current policy has already claimed. Call that the saturation ratio s_k for objective k: batch-mean reward, rescaled into [0, 1] against the objective’s known min and max.
Then the trick: multiply each objective’s contribution to the advantage by (1 - s_k)^γ, where γ is a single knob controlling how aggressively saturated objectives get muted. γ=0 recovers plain GDPO. Larger γ means “almost-solved objectives count for almost nothing this batch.”
Critically, per-objective advantages are computed the standard way (subtract group mean, divide by group std, per reward dimension), then combined with these adaptive weights, then re-normalized across the batch. The PPO-style clipped update is untouched. All the intelligence is in how the scalar advantage got built.
One subtle consequence the authors flag: because the reweighting happens before the final batch normalization, it can flip the sign of a rollout’s aggregate advantage, not just shrink it. A rollout that looked good overall under fixed weights can become a negative example once the “easy win” objective it exploited is discounted.
for k in objectives:
r_bar_k = mean(rewards[k]) # batch mean
s_k = (r_bar_k - r_min[k]) / (r_max[k] - r_min[k])
w_tilde[k] = w[k] * (1 - s_k) ** gamma
for i, j in rollouts: # per query, per sample
A_ij = sum(w_tilde[k] * zscore(r[k][i,j], group=i) for k in objectives)
A_hat = (A_ij - mean(A_ij)) / std(A_ij) # batch renormalize
theta = grpo_clipped_update(theta, A_hat) # standard PPO-style step
The usual instinct in multi-reward RL is to pick weights up front, maybe tune them, and trust the group-relative normalization to sort out the rest. This paper argues the opposite. Objective weights should move during training, and the signal that moves them is how much of each reward’s ceiling you’ve already hit, not reward variance or gradient agreement between objectives. The load-bearing evidence isn’t the math benchmark table. It’s the adaptive-reasoning setup where the length reward is deliberately built to saturate, and you can watch correctness accuracy climb as γ grows.
•
The clean mechanism test is the adaptive-reasoning setup on DeepSeek-R1-Distill-Qwen-7B: a length reward that maxes out at 1024 tokens (so it will saturate) plus binary correctness. SA-MRPO beats GDPO on all five math benchmarks, +3.8 pp average, up to +9.2 pp on AMC23 (28.3% → 37.5%). Average response length grows from 333 to 459 tokens but stays under the 1024 saturation threshold, so the length reward is preserved even though its optimization pressure was cut.
•
On standard math training (Qwen2.5-3B and 7B on DeepScaleR-Preview, with correctness + length ± format rewards), SA-MRPO beats GDPO in 12 of 15 benchmark comparisons, with up to +5.0 pp on AIME 2024.
•
On code generation (Qwen2.5-7B on Eurus-2-RL, executability + test-case pass rate), pass rate improves on three of four benchmarks by up to +2.3 pp while bug rates stay comparable. Executability was the fast-saturating objective; pass rate was where headroom lived.
•
The γ sweep behaves as the theory predicts: larger γ trades length reward down and correctness reward up during training. γ=0.5 is the sweet spot in their setup; very large γ starts blowing the length budget.
•
The paper is explicit that this is not a Pareto guarantee. Section 4.2 shows that when two objectives have conflicting gradients, reducing pressure on the saturated one can regress it. They frame SA-MRPO as an allocation rule, not a preservation constraint.
Reach for this when you’re doing RL with verifiable rewards with two or three verifiable rewards that saturate at different rates, and you’re watching the easy one hit ceiling in the first epoch. The change from a GDPO codebase is mechanical: track a running batch mean per reward, know each reward’s min and max (trivial for binary or bounded rewards), multiply per-objective advantages by (1 - s_k)^γ before the final normalization. Start at γ ≈ 0.25–0.5.
The paper releases no code or model weights that are called out in the text. Training is done with verl and vLLM on public datasets (DeepScaleR-Preview, Eurus-2-RL) and public base models (Qwen2.5-3B/7B-Instruct, DeepSeek-R1-Distill-Qwen-7B), so reproduction is possible but you’re writing the SA-MRPO advantage yourself.
When rewards are bounded and verifiable, spend gradient on the reward with headroom left, not the one you’ve already won. The knob is a single exponent on (1 - saturation), and it belongs in the advantage construction, not in the loss or the sampler.
•
Saturation is measured against the nominal reward max, not what the current model class can actually reach. If your base model is already at its capacity ceiling on the “hard” objective, SA-MRPO will keep pouring gradient into a wall.
•
No monotonic-preservation guarantee. On TACO (code), pass rate actually dropped 0.4 pp vs GDPO, and the theory section explicitly permits this when objectives have conflicting gradients.
•
The saturation ratio needs known reward bounds. This is natural for rule-based rewards (compiles/doesn’t, under-budget/over) but awkward for learned reward models or rubric-scored rewards where the effective max isn’t well-defined.