MMGRPO extends Group Relative Policy Optimization (GRPO) to multi-step LM pipelines by grouping rollouts at the module level (same module, same invocation index across trajectories) and propagating the final program reward back uniformly, lifting accuracy +11% on average when stacked with prompt optimization.
You’ve shipped a DSPy pipeline where one module rewrites a user query to strip PII, a second module calls a strong external LLM with the redacted version, and a third module composes the final answer. You have an end-to-end score (did it answer well, did it leak PII?), but no per-step labels. Today you can run a prompt optimizer like MIPROv2 over the module instructions, or you can flatten everything into one giant prompt and run GRPO on it. The first leaves the weights untouched; the second throws away the modular structure that lets you, say, keep private data away from the proprietary LLM. This paper is the missing third option: do online policy-gradient RL on each module’s weights while keeping the program’s control flow intact.
The core problem GRPO assumes away: it expects a batch of rollouts that share one input prompt, so it can rank them and push the policy toward the better ones. In a multi-module program, twelve rollouts on the same input produce twelve different trajectories. Module A might fire twice in one rollout and four times in another. Some rollouts crash with a parse error halfway through.
MMGRPO’s trick is to regroup. Instead of one group per input, you make one group per (module, invocation-index) pair. The first call to the query-generation module across all twelve rollouts forms one group. The second call to it forms another. Each entry in a group carries that module’s local prompt, its local output, and the final program-level reward for the whole trajectory it came from. Credit assignment is uniform: every module call in a successful trajectory gets the success reward, every call in a failed one gets the failure reward. The standard GRPO loss then runs independently on each group, updating only the LM weights tied to that module (in practice a shared LoRA adapter since all modules use the same base model).
Two housekeeping steps handle ragged trajectories: PadGroups aligns group sizes when modules fire different numbers of times, and SelectKDiverseElements picks a fixed-size subset that maximizes reward variance within the group. The recommended recipe stacks it with prompt optimization via BetterTogether:
# Stage 1: optimize prompt templates, freeze them
program_po = MIPROv2(metric).compile(program, trainset)
# Stage 2: with prompts frozen, RL the weights
program_rl = GRPO(metric).compile(program_po, trainset)
The instinct when bringing RL to a multi-step LM pipeline is either to collapse it into one long auto-regressive trajectory (losing modularity) or to invent per-step rewards (requiring intermediate supervision you don’t have). This paper shows a third path. Align module calls structurally across rollouts, hand each module group the same final reward, and let GRPO’s relative-advantage math sort out which local prompt/output pairs were on the winning side. The evidence that matters is not the headline accuracy lift but the composition result: MMGRPO and prompt optimization don’t substitute for each other, they stack.
The load-bearing finding is the composition gap. MMGRPO alone beats vanilla chain-of-thought by +7% averaged across tasks and models; MIPROv2 alone beats it by +5%. Stacking them via BetterTogether beats vanilla CoT by +11%, MIPROv2 by +5%, and MMGRPO by +3%. Neither method dominates the other on its own, which is the point: they’re operating on different parameters (prompts vs. weights) and the gains add.
Secondary observations across Banking77, PAPILLON, and HoVer using llama3.1-8b-instruct and qwen3-8b:
•
The biggest single-task jumps come on multi-stage programs. On PAPILLON with qwen3, MMGRPO lifts score to 83.3 vs. 78.1 for MIPROv2 and 78.3 vs. vanilla CoT.
•
MIPROv2 is dramatically cheaper: ~1.4 GPU-hours on 1 H100 vs. ~18.7 GPU-hours on 2 H100s for MMGRPO. If budget is tight, prompt optimization is the better first move.
•
Warm-starting MMGRPO from prompt-optimized prompts (the BetterTogether order) consistently beats running MMGRPO from the vanilla CoT starting point. Better initial rollouts give a cleaner training signal.
•
On Banking77, which is a single-module program, MMGRPO degenerates exactly to standard GRPO. Even there, working purely from reward signals (no gold intent labels) it can’t match a supervised encoder, and the authors flag this honestly.
Reach for this when you’re shipping a multi-module agent built in DSPy, you have an end-to-end metric (a unit-test pass rate, a judge score, a retrieval recall@k), and you’ve already prompt-optimized but want more. The concrete move: run MIPROv2 first to get good prompts, freeze them, then run dspy.GRPO to fine-tune a single LoRA adapter shared across modules. You don’t need per-step labels and you don’t need to rewrite the program as a single flat prompt.
The optimizer ships as dspy.GRPO in the DSPy library, backed by the Arbor RL training library. Training settings reported: LoRA rank 16, 750 steps, 4 examples/step, 12 rollouts each, on 2 H100s. The teacher-program hook (sampling rollouts from a stronger model or a prompt-optimized variant) is built in, so partially off-policy training and distillation-style warm-starts are first-class.
Prompt optimization and weight-level RL are not competing knobs on the same surface. They tune different parameters of a modular LM program, and the cheapest gain is to do prompts first, then RL with prompts frozen. Aligning rollouts by (module, invocation index) rather than by shared input is what makes GRPO work on programs whose trajectories diverge in length and structure.
•
Uniform credit assignment is the simplest possible choice. Every module call in a failed trajectory takes the blame equally, even if one specific module was the actual culprit. On programs with one weak module surrounded by competent ones, the signal will be noisier than it should be.
•
All experiments use 8B-parameter models with LoRA. Whether the +3% MMGRPO-on-top-of-PO gain survives at frontier scale, or with full-parameter fine-tuning, is open. The authors flag both limitations.
•
The compute cost is real: ~13× more GPU-hours than MIPROv2 alone for ~3 percentage points of additional lift. If your metric is cheap and your prompts are already strong, the marginal RL pass may not pay for itself.