Flow-Map GRPO: Reinforcement Learning for Few-Step Flow-Map Generators via Anchored Stochastic Composition
2026-07-01 • Machine Learning
Machine LearningArtificial IntelligenceComputer Vision and Pattern Recognition
AI summaryⓘ
The authors address the challenge of improving deterministic flow-map generators—models that quickly transform noise into data—in a way that works well with reinforcement learning (RL) techniques. They introduce a method called Flow-Map GRPO, which adds controlled randomness to these deterministic models without changing their fundamental structure, enabling RL optimization. Their approach uses a new technique called Anchored Stochastic Flow Map Composition to keep the original paths intact while allowing stochastic training. Experiments showed that their method enhances performance on text-to-image generation tasks without needing to retrain the models as inherently stochastic.
flow-map generatorsdeterministic modelsreinforcement learningstochastic trajectoriesconsistency modelsMeanFlowstochastic differential equationspath-preserving stochasticizationtext-to-image generationreward optimization
Authors
Zhiqi Li, Wen Zhang, Bo Zhu
Abstract
Few-step flow-map generators, such as consistency models and MeanFlow, accelerate sampling by directly learning long-range transport maps between noise and data. However, these models are typically deterministic, which makes them difficult to optimize with reinforcement learning (RL) post-training methods that require stochastic trajectories and well-defined likelihood ratios. Existing SDE-based stochasticization techniques are designed for velocity-based samplers with infinitesimal or finely discretized transitions, and therefore do not directly apply to long-range flow maps. In this work, we propose Flow-Map GRPO, an online RL post-training framework for deterministic few-step flow-map generators. The key component is Anchored Stochastic Flow Map Composition (ASFMC), a path-preserving stochasticization mechanism that introduces randomness through anchor-based conditional resampling while preserving the original marginal probability path of the deterministic flow map. We derive GRPO objectives for both single-time and two-time flow-map parameterizations. Experiments on few-step FLUX-based text-to-image generators, including MeanFlow and sCM, show that Flow-Map GRPO improves pretrained deterministic flow-map models across reward-based, perceptual, and task-level evaluation metrics. Our results demonstrate that deterministic few-step flow-map generators can be effectively aligned with RL post-training without modifying their original model parameterization or retraining them as native stochastic models.