RewardVerse fixes unstable video-quality reward models by inserting a learned, query-specific rubric between the prompt and the scorer, then reading scores from token logits rather than generated text. Trained with only 30 preference pairs per evaluation dimension, it produces reward signals stable enough to drive downstream Group Relative Policy Optimization (GRPO) fine-tuning without reward hacking.
If you use reinforcement learning to fine-tune a video generation model, you need a reward model that assigns a scalar score to each generated clip. The paper’s problem: existing video reward models (e.g., VideoScore, VideoReward, UnifiedReward) map a video directly to a number, with no explicit criteria. This produces what the authors call scalar drift: scores collapse into a narrow high band (so the gap between good and bad videos is tiny, giving RL near-flat gradients), and the same video gets different scores under paraphrased prompts or reordered inputs.
An illustrative scenario, not something the paper evaluated: you’re doing RL on a text-to-video model. Your reward model rates almost every sample between 4.6 and 4.9 out of 5. The policy has nothing to climb, and small prompt rephrasings shuffle the ranking. That’s the failure mode this paper targets.
The baseline framing: professional human annotators don’t score directly. They write down criteria first, then score against them. Prior work brought this “rubric-as-reward” idea to LLMs and image generation (e.g., Rubric-as-Reward (RaR)), but video reward models still score in one unconstrained step.
The core move: put a dynamic rubric between the query and the scorer, and make both learnable. A rubric here is a small JSON structure listing 3-5 evaluation themes, each with a weight (summing to 1) and a few observable “tips” describing what to look for.
The pipeline uses one shared Qwen2.5-VL-7B backbone playing two roles:
1.
Rubric generator reads only the query (target dimension plus text prompt), never the candidate video. This prevents the rubric from bending to flatter or punish specific clips.
2.
Scorer reads the video plus one rubric theme, then instead of generating a digit as text, the system reads the logits of the tokens “1” through “5” and takes their probability-weighted average. This Soft-logits readout readout gives a continuous score even though the model was asked for an integer. Themes are then combined by their weights into a single scalar.
Why soft-logits matters: an earlier ablation shows that a rubric alone paired with natural-language integer output still ties two-thirds of pairs, because the model’s linguistic prior parks it in the high band. Reading logits directly bypasses that bias.
Training uses Rubric-Guided Policy Optimization in two stages, both built on Group Relative Policy Optimization (GRPO):
•
Stage 1 (scorer warm-up): freeze a hand-built “seed” rubric per dimension (generated offline by Gemini 3.1 Pro proposing, verifying, and revising rubrics against 30 human-labeled pairs). Train the shared model in its scorer role using a preference reward (sigmoid of the score gap between preferred and non-preferred video) plus a calibration loss that pushes the predicted score difference toward the human-annotated margin.
•
Stage 2 (joint): now let the generator sample rubrics per query. Reward each sampled rubric by how well its induced scores separate the preferred from non-preferred video, plus a cosine-similarity term against seed rubric themes (using BGE-M3 embeddings) and a JSON-format check. Critically, the scorer receives no policy gradient from this reward. It only gets the calibration loss under the sampled rubric. Otherwise the scorer could game the reward by just enlarging score gaps.
for (query, y_win, y_lose) in pairs:
rubrics = [gen(query) for _ in range(G)] # Stage 2 samples rubrics
for R in rubrics:
s_win = sum(w_k * soft_logit_score(y_win, theme_k) for k in R)
s_lose = sum(w_k * soft_logit_score(y_lose, theme_k) for k in R)
r_pref = sigmoid(s_win - s_lose)
reward = r_pref + beta*format_ok(R) + lam*cos_sim(R, seed_R)
update_generator_by_grpo(rubrics, rewards)
update_scorer_by_calibration_loss(s_win - s_lose, human_margin)
The paper evaluates on EvalVerse (16 fine-grained dimensions like Action, Logic, Lighting, Consistency) and the external pairwise benchmark VideoGen-RewardBench (VGRB).
•
Pointwise correlation: RewardVerse gets the highest PLCC on 14 of 16 EvalVerse dimensions. On Logic, PLCC is 0.750 vs. next-best 0.593; on Action, 0.566 vs. 0.390. Untrained baselines can be badly miscalibrated (VideoScore-v1.1 hits -0.245 PLCC on Logic).
•
Pairwise transfer: On VGRB Visual Quality (seen dimension), RewardVerse beats the strongest non-oracle baseline (VisionReward) by +7.1 points accuracy-with-ties, despite training on only 480 pairs total. On Text Alignment (unseen dimension), it still leads non-oracle baselines. VideoReward is shown as an oracle upper bound because VGRB was built from its training distribution.
•
Ablations under a fixed 480-pair budget: removing Stage 1 drops PLCC by 0.099; removing Stage 2 drops it by 0.074; direct rubric-free pointwise training on the same budget lands well below full RGPO. So both stages contribute, and the rubric bottleneck is what makes the low-data regime work.
•
Scalar drift, before vs. after RGPO (same dynamic-rubric protocol, same 240-video holdout): score standard deviation expands from 0.457 to 1.332 (range decompression), mean bias shrinks from +0.610 to +0.164, and prompt-paraphrase sensitivity drops 58.7% after normalizing for the wider range.
•
Downstream RL: using RewardVerse as the reward for Group Relative Policy Optimization (GRPO) fine-tuning of Wan-2.2-A14B on Visual Quality lifts VBench Imaging Quality from 0.640 to 0.653 while VBench-Text also rises (0.428 to 0.446). Using VideoReward instead lifts Imaging Quality less and drops VBench-Text to 0.392, which the authors read as reward hacking.
One caution on causal reading: the ablations isolate stages of RGPO, but the downstream RL comparison is one reward model vs. one other on one base generator. That’s suggestive of less reward hacking, not proof.
•
If you’re building a reward model for a subjective generation task (video, image, long-form text) and your scores are collapsing into a narrow band, the diagnostic recipe here is directly reusable: score the same items under paraphrased prompts, measure standard deviation, check the flip rate under A/B order swaps. The paper shows a naive A/B protocol flips 56% of pairs even with a rubric, which is why they abandoned pairwise scoring for RL rewards.
•
Soft-logits readout is a cheap standalone trick worth trying whenever you’re asking an MLLM for a bounded integer score. It needs logit access, so it applies to open-weights models (Qwen, Llama-VL) but not to most hosted APIs. The paper explicitly notes closed APIs like Gemini 3.1 Pro can’t be used this way.
•
The rubric-as-intermediate structure is worth testing when you have very few preference pairs (dozens per dimension, not thousands) and need dimension-specific evaluation. The paper’s evidence is that under a matched 480-pair budget, direct pointwise training barely improves over zero-shot, but rubric-mediated training does. This is a low-data claim, not a general one.
•
If you use RewardVerse or a similar reward in RL, monitor non-target metrics. The paper’s downstream table shows optimizing one dimension can move others, sometimes down. Their reward preserved VBench better than the comparison, but “better” is not “neutral.”
•
Backbone is fixed at 7B Qwen2.5-VL. The authors note RGPO cannot recover visual details the backbone can’t see, so gains at larger scale are conjecture.
•
Only the aggregated score margin is calibrated against human labels. Individual theme scores can drift or be inconsistent even when the overall reward looks good, which limits interpretability of the per-theme breakdown.
•
Seed rubrics are bootstrapped by a proprietary frontier model (Gemini 3.1 Pro). Reproducing the exact seed pool requires access to a comparably strong teacher.
•
The 30-pairs-per-dimension claim is per-dimension in a 16-dimension taxonomy, so the total human-labeled budget is 480 pairs, not 30.
•
Downstream RL is demonstrated on one generator (Wan-2.2-A14B) optimizing one dimension (Visual Quality). Generalization to other generators, other dimensions, or joint multi-dimension optimization is unverified.