Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation

Noosaga Placements

• Computer Visionsubfield95%
  The paper focuses on text-to-image generation, a core computer vision task, and uses MLLMs for image-text alignment evaluation.
  SpectraReward measures how well the original prompt can be recovered from the generated image through a single image-conditioned, teacher-forced forward pass.Extensive experiments validate SpectraReward through a broad image-generation RL study covering two diffusion models, three RL algorithms, nine reward MLLM backbones from four MLLM families spanning 4B to 235B parameters, and five out-of-distribution text-to-image benchmarks.
• Diffusion Modelsframework90%
  The paper uses diffusion models (SD3.5-M, BAGEL) as the policy models for text-to-image generation.
  Extensive experiments validate SpectraReward through a broad image-generation RL study covering two diffusion models, three RL algorithms, nine reward MLLM backbones from four MLLM families spanning 4B to 235B parameters.On both SD3.5-M [10] and BAGEL [9], SpectraReward brings significant improvements on all downstream benchmarks.
• Reinforcement Learningsubfield90%
  The paper proposes a reward function for reinforcement learning in image generation, and the method is evaluated using RL algorithms (AWM, FlowGRPO, DiffusionNFT).
  SpectraReward, a training-free reward function that turns pretrained MLLMs into off-the-shelf reward models for image-generation reinforcement learning.We use the average image-conditioned prompt log-likelihood as the reward, directly reusing the MLLM's pretrained image-text alignment ability without preference labels, reward-model fine-tuning.
• Vision-Language Modelsframework90%
  The paper uses pretrained Multimodal Large Language Models (MLLMs) as reward models, which are vision-language models.
  SpectraReward, a training-free reward function that turns pretrained MLLMs into off-the-shelf reward models for image-generation reinforcement learning.We study nine reward MLLM backbones from four MLLM families, with external reward MLLMs spanning three families and 4B to 235B parameters.
• Transformer Architectureframework85%
  The paper uses MLLMs that are based on transformer architectures (e.g., Qwen3-VL, Gemma3, InternVL3.5) as reward models.
  We study nine reward MLLM backbones from four MLLM families, with external reward MLLMs spanning three families and 4B to 235B parameters.SpectraReward uses Qwen3-VL-30B-A3B [2] as the reward MLLM backbone.
• Deep Learningsubfield80%
  The paper uses pretrained MLLMs (deep learning models) and diffusion models, and the reward function is based on the MLLM's forward pass.
  SpectraReward measures how well the original prompt can be recovered from the generated image through a single image-conditioned, teacher-forced forward pass.We further introduce Self-SpectraReward, a special case for unified multimodal models where the policy's own understanding branch serves as the reward model for its generation branch.
• Actor-Critic Methodsframework80%
  The paper uses AWM (Advantage Weighted Matching) as the default RL algorithm, which is an actor-critic method.
  We use AWM as the default reinforcement learning algorithm, following the optimizer ablation in Section 4.4, where AWM achieves the best overall performance among the tested RL algorithms.
• Natural Language Processingsubfield70%
  The method uses MLLMs, which are multimodal models with a language component, and the reward is based on prompt token log-likelihood.
  We use the average image-conditioned prompt log-likelihood as the reward, directly reusing the MLLM's pretrained image-text alignment ability.The token-wise likelihoods form the semantic spectrum of the image-text pair.
• Representation Learningframework70%
  The paper reuses the MLLM's pretrained image-text alignment ability, which is a form of representation learning.
  directly reusing the MLLM's pretrained image-text alignment ability without preference labels, reward-model fine-tuning.

Abstract

In this paper, we propose SpectraReward, a training-free reward function that turns pretrained MLLMs into off-the-shelf reward models for image-generation reinforcement learning. Instead of asking the MLLM to judge a generated image or answer decomposed verification questions, SpectraReward measures how well the original prompt can be recovered from the generated image through a single image-conditioned, teacher-forced forward pass. We use the average image-conditioned prompt log-likelihood as the reward, directly reusing the MLLM's pretrained image-text alignment ability without preference labels, reward-model fine-tuning. We further introduce Self-SpectraReward, a special case for unified multimodal models where the policy's own understanding branch serves as the reward model for its generation branch, forming a closed-loop self-improving framework without external reward models or external knowledge. Extensive experiments validate SpectraReward through a broad image-generation RL study covering two diffusion models, three RL algorithms, nine reward MLLM backbones from four MLLM families spanning 4B to 235B parameters, and five out-of-distribution text-to-image benchmarks. Results show that both SpectraReward and Self-SpectraReward significantly and consistently improve generation performance and outperform prior MLLM-derived reward training methods. Further analysis reveals that larger reward MLLMs are not always better, while Self-SpectraReward can match or surpass much larger external reward models, suggesting that reward-policy alignment is a key factor for effective image-generation RL. Project Page: https://huangrh99.github.io/SpectraReward/

Paper Context