Drifting Preference Optimization for One-Step Generative Models
2026-06-01 • Machine Learning
Machine LearningComputer Vision and Pattern Recognition
AI summaryⓘ
The authors propose a new method called Drifting Preference Optimization (DrPO) to improve one-step text-to-image generators based on user preferences. Instead of relying on complicated techniques like gradient calculations or multiple steps, DrPO ranks generated images by a reward and updates the model using differences between the best and worst ones, making the training simpler and faster. This approach works with black-box or non-differentiable rewards and keeps image generation efficient with just one model call. Their tests show that DrPO aligns better with target rewards and reduces training time compared to other methods.
text-to-image generationpreference finetuningone-step generatorsdeterministic modelsreward rankingfeature-space updatesnon-parametric methodsblack-box rewardsStable Diffusionmodel alignment
Authors
Zhou Jiang, Yandong Wen, Zhen Liu
Abstract
One-step text-to-image generators are attractive for deployment because they generate an image with a single forward pass, but preference finetuning them remains difficult: standard alignment methods often rely on policy likelihoods, denoising trajectories, differentiable reward gradients, or test-time optimization. We propose Drifting Preference Optimization (DrPO), an online preference-finetuning method for deterministic one-step generators. For each prompt, DrPO samples candidates from the current generator, ranks them with a target reward, and uses high- and low-scoring samples to synthesize a feature-space update direction. The update is a non-parametric dipole preference field plus a reference drift estimated from the frozen base generator, and is optimized through a detached feature-space regression target. The target reward is used only for ranking, so DrPO can train with large, black-box, or non-differentiable rewards while inference remains a single generator call. We evaluate DrPO on SD-Turbo and SDXL-Turbo with multiple target rewards and benchmarks, including HPSv3 and GenEval. DrPO improves alignment over reward-gradient-free one-step preference baselines and reduces HPSv3 training computation by $3.51\times$ under the matched effective-batch setting by removing reward-model backpropagation. Initial offline experiments suggest that sample-based gradient synthesis can also be used beyond online reward ranking.