SpectraReward turns any frozen Multimodal Large Language Model into a text-to-image reward model by scoring how well the prompt can be read back from the generated image, using image-conditioned prompt log-likelihood as the reward. No preference labels, no reward fine-tuning.
Say you’re shipping a text-to-image feature and users complain the model ignores “two cats on the left, one dog on the right.” The standard fix is Reinforcement Learning from Human Feedback with a reward model trained on human preferences (PickScore, ImageReward), which means collecting annotations, training a reward model, and re-doing it whenever your generator changes. The training-free alternative, prompting an Multimodal Large Language Model to score images 1-5 or answer decomposed yes/no questions (VIEScore, AlphaGRPO), is noisy or engineering-heavy. This paper argues you don’t need any of that. The Multimodal Large Language Model you already have can produce a clean scalar reward through one forward pass.
Given a generated image and its prompt, feed the image into a frozen Multimodal Large Language Model as visual context, then run one teacher-forced pass over the prompt tokens. Average the per-token log-probabilities. That number is the reward. Intuition: if the image really depicts “two cats on a red sofa,” then conditioned on the image, the model should assign high probability to those exact words. If the image shows five cats, the token “two” gets a low probability and the average drops. The authors call the per-token profile a semantic spectrum. A worry about language priors (common words are easy regardless of the image) is dismissed because they use Group-relative RL: the prior is shared across rollouts of the same prompt and cancels in the advantage computation.
The specialization for unified models is Self-SpectraReward. If your generator is a Unified Multimodal Model like BAGEL, its own understanding branch scores images produced by its generation branch. Same tokenizer, same vision encoder, same pretraining distribution. The reward is calibrated to the exact image distribution the policy samples from.
def spectra_reward(image, prompt, mllm):
# teacher-forced pass, image as visual condition
logits = mllm.forward(image=image, text=prompt)
logps = log_softmax(logits)
token_logps = [logps[t, prompt[t+1]] for t in range(len(prompt)-1)]
token_logps = mask_eos(token_logps) # EOS dominates short prompts
return mean(token_logps) # scalar reward for GRPO advantage
The prevailing move for MLLM-as-judge is to ask the model to emit a judgment: a 1-5 score, a yes/no answer, a chain-of-thought verdict. This paper shows the opposite. Don’t ask the MLLM to judge; ask it to predict the prompt from the image and read off the likelihood it already assigns. The judging capability is a downstream skill; image-conditioned captioning is what MLLMs are pretrained on, so tapping the likelihood directly is closer to the model’s native competence. The load-bearing evidence is the reward-function ablation where scalar scoring and VQA-yes-probability underperform the baseline, while prompt-likelihood beats it cleanly.
The reward-function ablation is the finding that carries the thesis. Swapping only the reward while holding the RL algorithm and policy fixed: scalar 1-5 scoring drops GenEval by 6.3 points below the un-trained baseline, VQA-yes-probability is roughly flat, and prompt-likelihood jumps GenEval from 84.0 to 89.5. Same MLLM, same policy, different way of squeezing a number out. That gap says the mechanism, not the model, is doing the work.
•
On BAGEL at 512 resolution, SpectraReward beats the BAGEL baseline by +10.0 on TIIF-Bench short prompts and +4.3 on GenEval; Self-SpectraReward matches it and adds +5.5 on GenEval.
•
Against AlphaGRPO, the strongest prior MLLM-derived reward baseline, gains are +6.3 TIIF / +2.1 GenEval.
•
Scaling the reward MLLM is non-monotonic: Qwen3-VL improves from 8B to 30B, then drops at 235B. A 30B external reward is the sweet spot.
•
Self-SpectraReward (BAGEL’s own ~7B-scale understanding branch) matches or beats the 30B external and beats the 235B one. Reward-policy alignment substitutes for scale.
•
Pretraining-checkpoint MLLMs beat instruction-tuned ones as reward backbones, consistent with the thesis that raw image-captioning ability is what matters.
Reach for this when you’re doing RL on a text-to-image or unified multimodal generator and don’t want to build a preference-annotation pipeline. Concretely: point any pretrained vision-language model at your generated images, teacher-force the prompt, average the log-probs, plug into Flow-GRPO / AWM / DiffusionNFT as the scalar reward. If your generator is a unified model, use its own understanding branch and skip serving a second large model entirely. That’s the practical win, one model on the GPUs instead of two.
Artifacts: the project page is linked in the paper. The paper does not explicitly commit to a code or checkpoint release in the text provided. Reward MLLMs tested include Qwen3-VL, Gemma 3, and InternVL 3.5, all off-the-shelf. Training was on 32 A100s for 380 steps at 512 resolution on the AlphaGRPO20k prompt set.
When you need a reward, ask the model to predict, not to judge. Likelihood is a signal the model was pretrained to produce; scores and verdicts are downstream skills you’re hoping it acquired. And if the generator and the judge can share weights, they should, because reward-policy alignment beats reward-model scale.
•
The reward only reflects what the prompt tokens explicitly say. Implicit consequences (“hot coffee” implies steam) get weak signal because “steam” isn’t in the prompt. Aesthetics and safety aren’t optimized at all; you’d need a complementary reward.
•
Non-monotonic scaling means picking the reward backbone is empirical. The 235B Qwen3-VL variant was worse than 30B on their setup, so “just use the biggest MLLM” is wrong.
•
Self-SpectraReward’s structural advantage depends on the generator being a Unified Multimodal Model with a genuinely capable understanding branch. On a diffusion-only generator like Stable Diffusion 3, you’re back to picking an external reward MLLM and inheriting its biases.