Supervised fine-tuning and reinforcement learning behave differently under multi-task training because Group Relative Policy Optimization (GRPO)-style advantage normalization strips out the shared gradient direction, leaving per-task updates that are ~10⁻⁵ cosine-similar and roughly two orders of magnitude smaller than SFT’s.
You’ve shipped an LLM that does math, code review, and log triage. Your team wants to add SQL debugging next month without regressing the other three. The default playbook is to mix all four datasets and run Supervised Fine-Tuning once, because everyone knows sequential SFT causes catastrophic forgetting. This paper says: if you use RL instead of SFT for the post-training stage, you can train each capability separately, in parallel, and just add the weight deltas together. The updates barely interfere. That reframes multi-task post-training from a data-mixing problem into a modular composition problem.
The authors compare SFT and RL gradients directly. SFT’s gradient is the log-likelihood gradient toward a fixed expert answer. RL’s gradient (using Group Relative Policy Optimization (GRPO)) weights the same log-likelihood gradient by a group-normalized advantage — for each prompt, the model samples G rollouts, and each rollout’s reward is standardized by the group mean and standard deviation.
Here’s the load-bearing algebra in plain terms. Because the standardized advantages within a group sum to zero, when you take the weighted average of gradients across the group, the common direction shared by all rollouts (the “mean score”) cancels out. What remains is only the residual: how each individual rollout’s gradient differs from the group average.
# Per-prompt RL gradient (GRPO)
rollouts = [sample(policy, x) for _ in range(G)]
rewards = [reward(y) for y in rollouts]
adv = (rewards - mean(rewards)) / std(rewards) # sums to 0
scores = [grad_log_prob(policy, y, x) for y in rollouts]
S_bar = mean(scores)
# Zero-sum advantage kills the S_bar term:
g = sum(adv[k] * (scores[k] - S_bar) for k in range(G)) / G
That residual is small (fixed prompt, fixed policy, only sampling noise separates rollouts) and, for two independent tasks, statistically independent. In the high-dimensional weight space of an LLM, two independent small zero-mean vectors are almost orthogonal. The paper formalizes this as an upper bound: SFT interference is bounded by the norm of the gradient (large), RL interference is bounded by the intra-group variance of rollouts (tiny).
They then propose Parallel-RL: train N separate LoRA or full-parameter RL runs, one per task, and merge the resulting weight deltas by summing, averaging, TIES Merging, or a brief 5% adaptation pass.
The prevailing assumption is that multi-task post-training is fundamentally a data problem: mix the right ratios, schedule the right curriculum, reweight the conflicting gradients. This paper shows the opposite. The training objective determines whether task gradients conflict at all. RL’s advantage normalization mechanically subtracts the shared gradient component that causes SFT’s task conflicts, so RL updates from different tasks live in nearly disjoint subspaces and can just be added together. The cleanest evidence isn’t the headline benchmark lift — it’s the parameter-level measurement that cross-task cosine similarity drops from ~10⁻¹ under SFT to ~10⁻⁵ under RL.
•
The load-bearing measurement: on the same base model, single-task RL updates have pairwise cosine similarity across tasks of ~10⁻⁵, versus ~10⁻¹ to 1.0 for SFT. RL updates are also ~200× smaller in L2 norm (~3×10⁻² vs ~7.4) and only ~20% of parameters change meaningfully, versus 93% for SFT.
•
Single-task generalization: training SFT on one task drops the other tasks by an average of 5.1 pp; training RL on one task improves the untrained tasks by 2.3 pp on average.
•
Multi-stage collapse: sequential SFT across Math → Science → Code → Logic drops the base model by 23.1% on average; sequential RL gains 24.9%.
•
Parallel-RL headline (1.5B, full-param Group Relative Policy Optimization (GRPO)): Naive sum of four independent RL deltas retains 94% of single-task RL performance. Adding TIES Merging retains 97%. A 5%-data adaptation pass after summing reaches 103% of single-task performance, beating the specialists. Parallel-SFT under the same setup collapses to 65–67% retention.
•
Ablation isolating decoupling: removing one task’s delta from the merged Parallel-RL model drops that task by an average of 7.1 pp while the other three tasks stay flat or improve by 0.6 pp. Capabilities are modular.
•
Generalizes to PPO: batch-level advantage normalization gives the same cancellation, though with slightly higher interference than GRPO’s group-level normalization.
Reach for this when you’re shipping an LLM that needs several verifiable skills — math, code, SQL, structured extraction — and you want each capability team to iterate independently. Instead of coordinating a single mixed-data run every time anyone changes their dataset, each team runs Group Relative Policy Optimization (GRPO) on their own task from the same base checkpoint. When you ship, you sum the deltas (optionally with TIES Merging) and optionally run a 5% adaptation pass on a mixed subset. The paper also gives you a diagnostic: run t-SNE on the score-function vectors from two candidate tasks; if the clusters overlap heavily (as their Sudoku-style Game task does with Math and Code), those tasks will interfere and shouldn’t be merged naively.
The authors release Code (the abstract links to a code repo; no license specified in the paper text). Experiments use DeepSeek-R1-Distill-Qwen-1.5B and its 7B sibling on MATH500, AIME2025, MMLU, GPQA, Knights & Knaves, and LiveCodeBench. Training data sources for each task are enumerated in Appendix A.
Advantage normalization is a gradient interference filter, not just a variance-reduction trick. Because zero-sum advantages algebraically erase the shared gradient direction across rollouts, RL post-training leaves per-task updates that are small, sparse, and nearly orthogonal — which is why you can train tasks in parallel and merge weights, and why you cannot do the same with SFT.
•
The tasks tested (math, science MCQ, logic puzzles, code) all have clean verifiable rewards and share a reasoning-heavy base model (DeepSeek-R1-Distill-Qwen-1.5B/7B). The orthogonality claim may weaken for tasks with dense reward shaping, or for tasks whose training data distributions genuinely overlap — the paper’s own Game counterexample shows interference when t-SNE clusters overlap.
•
The theoretical bound depends on intra-group rollout variance being small, which requires the base model to already be capable enough that G rollouts on the same prompt look similar. On a weak base model with high sampling entropy, the variance bound loosens and interference grows; the appendix shows a non-monotonic sweet spot around temperature ~0.65.
•
“Parallel-RL matches single-task performance” is measured after either TIES Merging or a 5% adaptation pass. Naive summation alone still loses ~5–6 pp relative to sequential multi-stage RL on the 7B model, so the merge step is not free.