Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation
2026-07-13 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors propose SpectraReward, a new way to use pretrained multimodal large language models (MLLMs) as reward functions for training image generators without extra training steps. Instead of rating images directly, SpectraReward checks how well the original text prompt can be recovered from the image, using the MLLM's existing ability to link images and text. They also introduce Self-SpectraReward, where a single model improves itself by using its own understanding to guide image generation. Their experiments show that these methods improve image generation performance and sometimes smaller models can work better if their reward and generation parts are well aligned.
Multimodal Large Language Models (MLLMs)Reward FunctionImage GenerationReinforcement Learning (RL)Prompt RecoveryDiffusion ModelsSelf-Improving FrameworkImage-Text AlignmentTeacher-Forced Forward PassReward-Policy Alignment
Authors
Runhui Huang, Qihui Zhang, Zhe Liu, Yu Gao, Jie Wu, Hengshuang Zhao
Abstract
In this paper, we propose SpectraReward, a training-free reward function that turns pretrained MLLMs into off-the-shelf reward models for image-generation reinforcement learning. Instead of asking the MLLM to judge a generated image or answer decomposed verification questions, SpectraReward measures how well the original prompt can be recovered from the generated image through a single image-conditioned, teacher-forced forward pass. We use the average image-conditioned prompt log-likelihood as the reward, directly reusing the MLLM's pretrained image-text alignment ability without preference labels, reward-model fine-tuning. We further introduce Self-SpectraReward, a special case for unified multimodal models where the policy's own understanding branch serves as the reward model for its generation branch, forming a closed-loop self-improving framework without external reward models or external knowledge. Extensive experiments validate SpectraReward through a broad image-generation RL study covering two diffusion models, three RL algorithms, nine reward MLLM backbones from four MLLM families spanning 4B to 235B parameters, and five out-of-distribution text-to-image benchmarks. Results show that both SpectraReward and Self-SpectraReward significantly and consistently improve generation performance and outperform prior MLLM-derived reward training methods. Further analysis reveals that larger reward MLLMs are not always better, while Self-SpectraReward can match or surpass much larger external reward models, suggesting that reward-policy alignment is a key factor for effective image-generation RL. Project Page: https://huangrh99.github.io/SpectraReward/