ReMaRk trains an LLM recommender with Reinforcement Learning by rewarding each generated list once against the union of all human responders’ answers, instead of picking one responder per training row. This cuts the dominant source of reward variance and reaches the baseline’s score in 25-53% of the Group Relative Policy Optimization (GRPO) steps.
Suppose you train an LLM to answer Reddit-style requests like “mind-bending sci-fi, not action-heavy.” Five people reply, each listing a few films. You want the model to generate a ranked list of 20 titles that overlaps with what those humans suggested.
The current recipe, Rank-GRPO, handles the multiple replies by turning each (request, responder) pair into a separate training row. The model samples 8 candidate lists, each list gets a reward counting hits against one responder’s items, and GRPO shifts probability toward above-average lists. Problem: responders rarely agree, so a list’s score mostly reflects which responder happened to be drawn. The authors measure this directly: the responder draw accounts for 85% of the variance in the advantage signal, and in more than half of all 8-list comparisons every list ties, so the group contributes zero gradient.
Intuition: the responder is noise from how the data was collected, not information about list quality. Average that noise away, and each gradient step carries a cleaner signal.
Formally, this is a Rao-Blackwellization: replacing a noisy estimator with its conditional expectation given the request keeps the expected update the same while shrinking variance. The clean version would reward each item by the fraction of responders who named it. But GRPO divides advantages by each group’s standard deviation, and fractional rewards on a fine grid can blow up: a single vote among 40 responders can get the same advantage as a hit everyone named (the paper calls this the “n-fold amplification” risk).
ReMaRk’s practical fix keeps 0/1 rewards by using the bounded sum: an item scores 1 if any responder named it, else 0. It also samples each request uniformly, not once per responder. Nothing else in Rank-GRPO changes.
for step in range(2000):
requests = sample_uniform(train_set, n=48) # was: sample by responder count
for x in requests:
lists = policy.sample(x, G=8)
union = set().union(*[r.items for r in x.responders])
for L in lists:
for k, item in enumerate(L):
R[L][k] = 1 if item in union else 0 # was: item in one responder's set
advantages = group_normalize(R) # per-position, per-group std
policy.update(advantages) # standard GRPO surrogate + KL
•
Compute savings. Across 7 model backbones (Qwen3.5 2B/4B/9B/27B/35B-A3B Mixture of Experts, Llama-3.2-3B, Llama-3.1-8B) and 3 datasets, ReMaRk reaches the baseline’s 2,000-step validation Recall@20 in 25-53% of the steps. At the full 2,000-step budget it leads by 0.8 to 1.6 points.
•
Trained to plateau. In 4 of 7 pairs ReMaRk matches Rank-GRPO’s best with 18-63% of the GPU-hours; it is never significantly behind on test. At 9B and on Reddit-Books, Rank-GRPO’s validation peak ends 0.08-0.36 points higher (not significant).
•
The gain tracks the variance removed. Exact Rao-Blackwellization of Rank-GRPO’s own update, holding everything else fixed, gains +0.93 points over three seeds. A dose-response experiment averaging over 1, 2, 4, or all responders shows recall rising with the share of noise removed.
•
Pooling, not the specific aggregate. The averaged reward (fractional) and the bounded-sum union end up level at 2B once list lengths are controlled. What matters is combining the responders, not which combiner you pick.
•
Transfers beyond recommendation. On ProtoQA, where ~100 people answer each commonsense question, the union reward gains +3.67 recall@10 over per-respondent rows at 250 steps.
•
Doubling the samples per request, filtering empty-signal groups (DAPO-style), or offline Direct Preference Optimization on pooled positives do not close the gap. Blind pairwise judges from two LLM families also prefer ReMaRk’s lists or rate them level, so the recall gain is not just chasing the recall metric.
•
If you run GRPO with one training row per annotator and the annotators disagree, that disagreement is probably eating most of your gradient signal. Pool the references into one 0/1 reward per prompt and sample each prompt uniformly. The change touches only how rows are built and scored.
•
The payoff scales with how many references each prompt has. On datasets where every prompt has exactly one reference (ReDial, INSPIRED), ReMaRk reduces to Rank-GRPO and buys nothing. Worth testing when average references per prompt is 2+.
•
Keep rewards binary under per-group advantage scaling. The paper’s bound shows fractional rewards on a 1/n grid can arbitrarily inflate a single annotator’s vote through the group std; a 0/1 union sidesteps this. If you want to try the exact averaged reward, batch-level scaling is safer.
•
Validation-based checkpoint selection matters. ReMaRk can peak early and decline; the compute savings depend on actually picking the right checkpoint, not just training longer.
•
The authors release Reddit-Books and Reddit-Music as IDs plus derived labels (not raw text, per Reddit’s terms). Code pointer is at the paper’s project page.
•
All three recommender datasets are Reddit threads with LLM-extracted and catalog-grounded items; grounding accuracy is high but not perfect.
•
The recall metric and the training reward both measure overlap with the same responders, so some of the gain is “the reward targets what we measure.” The LLM-judge results partly address this but are not independent of model families seen during pretraining.
•
Matched-budget comparisons stop at 2,000 steps where both arms are still improving; longer runs of either method show instabilities (KL spikes, list-length drift). Several backbones have only a single seed.
•
Rao-Blackwell variance guarantees do not formally carry through GRPO’s per-group scaling that ReMaRk actually uses; the propositions describe one update under a fixed policy, not full training dynamics. The practical gains are established empirically.