Flow-DPPO: 面向流匹配模型的散度近端策略优化

1 Introduction

Reinforcement learning (RL) has emerged as a core paradigm for aligning models with downstream objectives. In language models, RL methods such as DPO (Rafailov et al., 2023) and GRPO (Shao et al., 2024) have substantially improved alignment (Ouyang et al., 2022) and reasoning capabilities (Guo et al., 2025). Recently, these advances have been extended to image and video generation Liu et al. (2025); Wallace et al. (2024); Wang and Yu (2025); Xue et al. (2025a); Zheng et al. (2026), where flow matching models (Lipman et al., 2023; Liu et al., 2023) represent the dominant generative framework. Among them, Flow-GRPO (Liu et al., 2025) and DanceGRPO (Xue et al., 2025b) demonstrated strong performance by transforming deterministic ODE sampling into stochastic SDE trajectories and introducing PPO-style ratio clipping to enforce trust-region optimization.

The theoretical foundation of trust-region methods originates from Trust Region Policy Optimization (TRPO) (Schulman et al., 2015), which establishes a policy improvement bound: monotonic improvement is guaranteed when policy updates remain within a trust region defined by the divergence between the old and new policies. PPO (Schulman et al., 2017) later introduced ratio clipping as a computationally efficient first-order approximation to TRPO. However, as noted by Qi et al. (2026), each clipping decision is based on a single-sample Monte Carlo estimate of the true Total Variation (TV) divergence, rather than the divergence itself. In the continuous and high-dimensional latent space of flow models, this estimation noise becomes substantially amplified, leading to a systematic left shift in the ratio distribution, with its mean falling below one (Wang et al., 2025). We show that this bias is intrinsic to Gaussian policies: the standard PPO clipping range therefore becomes effectively asymmetric, failing to adequately constrain over-optimization for positive-advantage samples while excessively clipping negative-advantage ones.

	FLUX.1-dev	Flow-GRPO	Flow-CPS	GRPO-Guard	Flow-DPPO
seven green croissants
a blue dog on top of three white sheep behind seven white candles
a blue giraffe behind seven pink clocks to the right of an elephant

To mitigate this bias, GRPO-Guard (Wang et al., 2025) proposed normalizing the ratio distribution. While this re-centering alleviates the symptom, it does not address the root cause: the ratio remains a noisy, per-sample proxy for the true policy divergence. We observe that flow models offer a structural advantage that sidesteps this problem entirely. Because each per-step policy is Gaussian with a mean determined by the velocity network and a fixed, schedule-dependent variance , the KL divergence between old and new policies reduces to , which is an exact, deterministic quantity that can be computed from two forward passes already performed during training. Unlike the LLM setting, where DPPO (Qi et al., 2026) must resort to approximate divergence reductions over large vocabularies, flow models admit exact divergence computation at no additional cost. This motivates replacing ratio clipping with a direct KL-proximal trust region constraint.

Building on this insight, we propose Flow-DPPO (Flow Divergence Proximal Policy Optimization), which replaces ratio clipping with a divergence-based mask. The mask blocks gradient updates only when two conditions are jointly met: (1) the advantage and ratio indicate that the update is moving the policy away from the old policy, and (2) the exact KL divergence already exceeds a threshold. This design directly enforces the trust region while preserving the beneficial asymmetric structure of PPO: updates that move the policy towards the old policy are never blocked, accelerating recovery from overshooting. Extensive experiments on various base models demonstrate that Flow-DPPO achieves superior reward optimization, improved KL-proximal efficiency, stronger robustness to catastrophic forgetting, balanced multi-objective optimization that mitigates reward hacking, and stable multi-epoch training that enables higher sample efficiency. Figure 1 presents qualitative generation results demonstrating that Flow-DPPO achieves competitive compositional accuracy while preserving notably higher visual quality than existing methods.

2 Preliminaries

Flow matching (Lipman et al., 2023; Liu et al., 2023) learns a continuous-time velocity field that transports samples from a simple source distribution to the data distribution. Specifically, let , and define an interpolating path with , where and determine the probability path between data and noise. This construction induces a conditional distribution . The goal of flow matching is to train a time-dependent vector field to match the target velocity, which is given by and the functinal . The model is then trained by minimizing the regression objective

where is a weighting function. After training, samples are generated by solving the ODE . In practice, simple numerical solvers such as Euler discretization are often sufficient for high-quality sampling (Karras et al., 2022; Lu et al., 2022; Song et al., 2021a). A notable special case is rectified flow (Liu et al., 2023), which uses the linear conditional path and . Under this choice, the target velocity reduces to . We adopt this linear schedule throughout the paper.

2.1 RL Fine-Tuning for Flow Matching Models

For text-conditional flow matching models, given a conditioning prompt , generation starts from a Gaussian latent and progressively transforms it into a clean sample . At each timestep , the flow model predicts a velocity field , which specifies a deterministic generation direction. Applying RL algorithms such as GRPO (Shao et al., 2024) to flow matching models requires a sampler-induced stochastic policy at each denoising step. Flow-GRPO (Liu et al., 2025) constructs such a policy via an ODE-to-SDE conversion, which transforms the probability-flow ODE into an equivalent SDE with the same marginals (Albergo and Vanden-Eijnden, 2023; Albergo et al., 2024; Song et al., 2021b): , where denotes Wiener process increments, , and is a scalar hyperparameter controlling the noise level. Applying Euler–Maruyama discretization yields the Flow-SDE sampler:

with . An alternative is Coefficients-Preserving Sampling (CPS) (Wang and Yu, 2025), which reduces the excessive noise injection in Flow-SDE and better preserves the interpolation structure of the scheduler. Let denote the predicted clean sample and noise component, respectively. CPS updates the latent as

where and controls the stochasticity. Both Flow-SDE and CPS therefore induce Gaussian per-step policies written as

where the specific forms of and depend on the sampler. The above generative process can be formulated as a finite-horizon Markov Decision Process (MDP) (Black et al., 2024; Fan et al., 2023; Liu et al., 2025). To distinguish the discrete decision process from the underlying continuous-time flow, we use for the MDP state index and for the reverse-time variable of the flow. Let be a discretization of reverse time, so that state corresponds to flow time . The state at step is . Note that . For , the action is the next latent sample, , drawn from the sampler-induced policy . Given the sampled action, the transition is deterministic, with next state . The rollout starts from and , and terminates at where .

After the full generative process, a scalar reward is provided. RL fine-tuning maximizes the expected terminal reward with a KL regularization term that penalizes deviation from the pretrained reference policy : , where denotes a trajectory induced by and controls the regularization strength. This KL penalty discourages reward hacking and mitigates catastrophic forgetting of the pretrained model’s capabilities.

Flow-GRPO (Liu et al., 2025) applies GRPO to the above MDP. Given a prompt , the current policy generates a group of samples . Their rewards are normalized within the group to obtain relative advantages: . In practice, each policy optimization iteration begins by rolling out a batch of data, which is then split into several minibatches for multiple gradient steps. This procedure introduces policy staleness: after the first update, the optimizing policy has already diverged from the behavior policy that generated the data. To control this off-policy drift, a trust region mechanism is applied. Following PPO (Schulman et al., 2017), the policy is optimized using the clipped surrogate objective

where we omit the KL penalty term for brevity, and the per-step importance ratio is defined as . Since both Flow-SDE and CPS define Gaussian per-step policies as in Eq. (4), the log-ratio admits the same closed-form expression:

Therefore, both samplers can be optimized within the same GRPO framework, differing only in the parameterization of the induced stochastic policy.

3 Methodology

In this section, we first derive a policy improvement bound that justifies trust-region methods for flow models. Then, we show that ratio clipping is a noisy proxy for the true divergence constraint. Finally, we present Flow-DPPO, which leverages exact KL computation to enforce a deterministic divergence mask, yielding a tighter and variance-free trust-region constraint.

3.1 Trust-Region Policy Optimization for Flow Matching Models

Inspired by Schulman et al. (2017); Qi et al. (2026), we adapt the trust region framework to the flow model fine-tuning setting defined in Section˜2.1. This setting differs from the classical discounted RL paradigm in two important ways. First, the problem is an undiscounted episodic task with a finite horizon of decision steps. Second, due to the terminal reward structure, advantages are estimated at the trajectory level rather than per step. These properties necessitate a tailored policy improvement guarantee. We follow the MDP defined in Section˜2.1.

Theorem 1(Performance Difference Identity for Flow Models).

In the finite-horizon flow model MDP with decision steps, let denote the expected reward. For any two policies and , the performance difference decomposes as: , where the surrogate objective is

and the error term is

The surrogate represents a first-order approximation to the true improvement, while the error term captures higher-order interactions between per-step policy changes. To yield a practical optimization objective, we bound this error term.

Theorem 2(Policy Improvement Bound for Flow Models).

In the finite-horizon flow model MDP with decision steps, the policy improvement is lower-bounded by:

where is the maximum per-step Total Variation divergence, and is the maximum absolute reward.

Please refer to Appendix B for the detailed derivation; a tighter bound linear in is given in Appendix B.3. This bound is structurally analogous to the policy improvement bound for LLMs derived in Qi et al. (2026). It provides a rigorous justification for trust-region methods in flow model fine-tuning: constraining the per-step divergence controls the penalty term and guarantees monotonic improvement. Similar to TRPO (Schulman et al., 2015), we can solve the following constrained optimization problem to ensure stable learning:

Remark 3(Exact Divergence in the Gaussian Setting).

For the Gaussian per-step policies in Eq. (4), the TV divergence is a monotone function of the mean displacement:

where is the standard normal CDF. Constraining the TV divergence below a threshold is therefore equivalent to constraining for an appropriate , which is precisely the divergence measure that Flow-DPPO employs. Moreover, the Pinsker inequality ensures that our KL-based constraint also upper-bounds the TV divergence: when the per-step , we have . In the Gaussian equal-covariance case, the converse also holds since KL and TV are both monotone functions of . Thus, our method is theoretically justified from both the KL and TV perspectives. Unlike the LLM setting, where the discrete vocabulary requires approximate divergence computations (Qi et al., 2026), the Gaussian structure of flow models provides exact per-step divergence at zero additional cost.

3.2 Pitfalls of Ratio Clipping in Flow-GRPO

Flow-GRPO adopts PPO-style ratio clipping to enforce a trust region. For consistency with the Flow-GRPO notation (Liu et al., 2025), in this and the following subsections we index denoising steps by the flow time (equivalently, in the MDP indexing of Section˜2.1). The clipping condition is intended to prevent the new policy from deviating too far from the old one. However, the probability ratio is a fundamentally noisy proxy for the true policy divergence. By definition of the Total Variation divergence,

so each individual is merely a single-sample Monte Carlo estimate of . While the policy improvement bound (Theorem˜2) calls for constraining , ratio clipping constrains this noisy per-sample surrogate instead. This issue was identified by Qi et al. (2026) in the LLM setting; we now show that the resulting pathology is particularly severe in flow models due to the high-dimensional continuous action space.

Recall from Eq. (6) that the log-ratio is:

Since is sampled from , we can write where . Substituting and letting :

The first term, , is a zero-mean random variable with variance . This reveals that the log-ratio is dominated by noise: the signal (the deterministic second term ) is exactly the negative of the KL divergence, but it is corrupted by a noise term whose standard deviation is of the same order as the signal itself. This analysis yields two key insights:

1. 1.

   High variance. The ratio is inherently noisy due to the stochastic sample . Even when the true KL divergence is moderate, individual ratio samples can be extreme (either very large or very small), triggering spurious clipping.

2. 2.

   Noise-dependent clipping. Whether an update is clipped depends heavily on the random noise drawn during sampling, rather than the true policy divergence. Two trajectories with identical policy parameters but different noise realizations may receive entirely different clipping decisions.

In contrast, the true KL divergence is a deterministic function of the policy parameters alone, unaffected by the sampling noise. This motivates our approach: replace the noisy ratio-based trust region with a direct divergence constraint. A detailed variance analysis is provided in Appendix D.

3.3 Divergence Proximal Policy Optimization for Flow Models

We now derive the divergence between old and new policies in the flow model setting and present our Flow-DPPO algorithm.

Exact KL divergence. Since both and are Gaussians with the same variance but different means, the KL divergence admits the closed form (see Appendix C for derivation):

For Flow-SDE (corresponding to Eq. (2)), , giving:

For CPS (corresponding to Eq. (3)), with :

Remark 4.

In the LLM setting, DPPO (Qi et al., 2026) must approximate the true divergence via Binary or Top-K reductions of the vocabulary distribution, as computing exact TV or KL over tokens is memory-prohibitive. In flow models, the Gaussian policy structure yields exact divergence at negligible cost, namely the squared difference between two forward passes of the velocity network. This makes divergence-based trust regions strictly more natural for flow models than for LLMs.

The Flow-DPPO mask. We define the Flow-DPPO objective as:

where the divergence-based mask is:

with and a divergence threshold.

Asymmetric design. The mask in Eq. (18) preserves the asymmetric structure that makes PPO effective. It only blocks updates that are already moving away from the old policy:

• •

  When and : the gradient is pushing the policy further from (increasing an already-increased action probability). The mask blocks this if the divergence exceeds .

• •

  When and : the gradient is decreasing an already-decreased action probability, again moving away from the old policy. The mask blocks this if divergence exceeds .

• •

  In all other cases ( or ): the gradient is moving the policy towards the old policy. These beneficial updates are never blocked, regardless of the divergence level.

This asymmetry ensures that the trust region constraint does not impede recovery: when the policy has drifted too far, corrective updates remain uninhibited. We provide a justification of this directional condition and discuss refined mask variants in Appendix E.

4 Experiments

Models and Baselines. We employ Stable Diffusion 3.5 Medium (Esser et al., 2024; Stability AI, 2024) (SD3.5), FLUX2-klein-base-9B (Black Forest Labs, 2026) (FLUX2-9B) and FLUX.1-dev (Black Forest Labs, 2024) as base models to cover diverse architectures and scales. We compare our method against four competitive baselines: Flow-GRPO (Liu et al., 2025), Flow-CPS (Wang and Yu, 2025), GRPO-Guard (Wang et al., 2025) and Diffusion-NFT (Zheng et al., 2026). Specifically, we evaluate two variants of our approach: Flow-DPPO (using SDE sampling from Flow-GRPO) and Flow-DPPO+CPS (using CPS-scheduled SDE sampling). Detailed configurations are deferred to Appendix F.

Metrics and Datasets. GenEval2 (Kamath et al., 2025) and PickScore (Kirstain et al., 2023) are selected as in-domain and out-of-domain (OOD) datasets, respectively. For GenEval2, we follow the official template to generate 20k synthetic training prompts and evaluate on the 800 officially released prompts. To monitor catastrophic forgetting under distribution shifts, we track PickScore (Kirstain et al., 2023), CLIP (Radford et al., 2021) score, and HPSv2 (Wu et al., 2023) during training. We report results for both single-reward optimization (GenEval2 only) and multi-reward training, where GDPO (Liu et al., 2026) aggregates advantages with equal reward weights.

4.1 Main results

Performance and Generalization. As summarized in Table 1, Flow-DPPO variants consistently outperform all baselines across both base models and all evaluation metrics, with particularly substantial gains in the GenEval2 reward. In the single-reward setting (optimizing GenEval2 only), Figure 2 demonstrates that our proposed variants not only achieve superior performance on FLUX2-9B compared to baselines but also exhibit a more stable training trajectory. These empirical advantages persist across SD3.5 (Figure 7) and FLUX.1-dev (Figure 9).

We attribute this superiority to the precise divergence-based mask in Flow-DPPO. By mitigating the influence of samples falling outside the trust region, which are susceptible to reward hacking, Flow-DPPO maintains a more robust optimization gradient. This constraint prevents the model from excessively exploiting individual rewards at the expense of others, thereby achieving a superior balance across multiple optimization objectives and fostering stable convergence. This is further corroborated by the multi-reward training curves in Figure 4, where Flow-DPPO variants consistently outperform all baselines across most metrics on SD3.5, without sacrificing any individual objective.

Out-of-domain Behavior and Catastrophic Forgetting. To investigate catastrophic forgetting, we analyze OOD metrics (PickScore, CLIP, and HPSv2) and the KL divergence from the pre-trained model. As illustrated in Figure 2, OOD metrics initially increase across all methods as RL optimization drives the model toward higher visual quality. However, as training progresses, these metrics decline, indicating that the model overfits the in-domain reward (GenEval2) at the expense of OOD knowledge. Notably, Flow-DPPO variants exhibit significantly less OOD degradation, suggesting that catastrophic forgetting is effectively mitigated. Qualitative results in Figure 4.2 further support this, demonstrating that our methods better preserve visual fidelity on OOD prompts. Consistently, Table 2 shows that Flow-DPPO variants maintain a lower KL divergence in most settings. This reduced distribution drift aligns with OOD metric trends, collectively indicating stronger resistance to reward hacking and forgetting. Ultimately, these results highlight that the divergence-based mask acts as a safety boundary, allowing the model to learn from rewards without losing its original generative quality or falling into distribution collapse.

4.2 Analysis

Asymmetric Masking and Divergence Threshold. We investigate the impact of the divergence threshold and asymmetric masking in Flow-DPPO using SD3.5 with CPS sampling (Figure 3). Without asymmetric masking, the training process collapses as the trust-region regularization becomes ineffective; specifically, samples falling outside the trust region are largely ignored, preventing optimization progress. Conversely, asymmetric masking constrains these samples back within the trust region, thereby stabilizing the trajectory. Regarding the divergence threshold, a looser threshold () results in diminished stability and suboptimal convergence. A tighter threshold () initially slows down learning but fosters superior stability and slightly better final performance due to more rigorous trust-region enforcement.

Multi-epoch Training and Sample Efficiency. Given the high computational cost of rollouts, we investigate how sample reuse frequency affects optimization efficiency on SD3.5. Specifically, we vary two factors: (i) the number of groups per rollout, and (ii) the number of training epochs per rollout (inner loops). The latter determines the reuse frequency of each sample. For instance, two inner loops imply that each rollout batch is utilized for two consecutive gradient steps.

媒体内容 · 前往原文查看

	FLUX2-9B			SD3.5
Method	Single	Multi	+CFG	Single	Multi
Flow-SDE schedule
Flow-GRPO	0.77	0.79	1.36	2.34	3.81
GRPO-Guard	1.07	1.01	1.63	2.05	3.33
Flow-DPPO	0.17	0.49	0.51	1.16	2.49
CPS schedule
Flow-CPS	0.24	1.66	1.51	2.41	3.18
Flow-DPPO + CPS	0.68	0.70	0.83	1.60	2.52

Figure 5: Multi-epoch training on SD3.5 (Left: Flow-SDE, Right: CPS). Flow-DPPO variants show consistent long-term gains under multi-epoch training (G64-I2 and G32-I2), while baselines plateau or even degrade.

	FLUX2-9B	Flow-GRPO	Flow-CPS	GRPO-Guard	Flow-DPPO	Flow-DPPO+CPS
In-Domain
a stone pig in background, two black cats in front of the pig, and six yellow horses in front
five colorful bicycles below, two stone guitars above them, and a penguin at the highest point
Out-of-Domain
a sun elf with a bow, facing the camera, in a jungle waterfall scene
people at a barbecue in Brazil, captured in HD, Canon EOS 5D Mark IV DSLR

Figure 6: Qualitative comparison on FLUX2-9B with single-reward setting and controlled seeds for each prompt at the same training iteration. Flow-DPPO and Flow-DPPO + CPS retain competitive in-domain performance with less reward hacking while exhibiting notably less catastrophic forgetting on out-of-domain prompts.

While our main experiments use 64 groups with 1 inner loop (G64-I1), we further explore two efficiency-oriented settings: G32-I2 (half the rollout computation with samples reused twice) and G64-I2 (standard rollout computation with doubled training intensity). As shown in Figure 4.2, baseline methods (Flow-GRPO, Flow-CPS) struggle to achieve sustained gains under multi-epoch training, often leading to performance plateaus or degradation. In contrast, Flow-DPPO variants successfully reuse rollout samples across multiple updates, yielding consistent long-term performance improvements. This advantage stems from the divergence-based mask, which constrains updates within the trust region, ensuring efficient sample utilization. This offers a promising direction for scenarios where rollouts are computationally expensive, such as long-video generation.

5 Conclusion

We show ratio clipping in flow models is a noisy, biased proxy for divergence. To address this, we propose a divergence-based mask using the exact KL at zero extra cost. Across multiple base models, sampling schedules, and reward objectives, Flow-DPPO consistently achieves superior performance than baselines in terms of reward optimization and catastrophic forgetting. Furthermore, Flow-DPPO enables stable multi-epoch training where ratio clipping degrades, offering a promising direction for scenarios with expensive rollouts, such as long-video generation.

References

• M. S. Albergo, M. Goldstein, N. M. Boffi, R. Ranganath, and E. Vanden-Eijnden (2024) Stochastic interpolants with data-dependent couplings. In International Conference on Machine Learning, pp. 921–937. Cited by: §2.1.
• M. S. Albergo and E. Vanden-Eijnden (2023) Building normalizing flows with stochastic interpolants. In The Eleventh International Conference on Learning Representations, Cited by: §2.1.
• Black Forest Labs (2024) FLUX.1: announcing black forest labs. Note: https://blackforestlabs.ai/announcing-black-forest-labs/ Cited by: Figure 1, Figure 1, §4.
• Black Forest Labs (2026) FLUX.2 [klein]: towards interactive visual intelligence. Note: https://bfl.ai/blog/flux2-klein-towards-interactive-visual-intelligenceModel weights: https://huggingface.co/black-forest-labs/FLUX.2-klein-base-9B Cited by: §4.
• K. Black, M. Janner, Y. Du, I. Kostrikov, and S. Levine (2024) Training diffusion models with reinforcement learning. In The Twelfth International Conference on Learning Representations, Cited by: §2.1.
• P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, et al. (2024) Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, Cited by: §4.
• Y. Fan, O. Watkins, Y. Du, H. Liu, M. Ryu, C. Boutilier, P. Abbeel, M. Ghavamzadeh, K. Lee, and K. Lee (2023) Reinforcement learning for fine-tuning text-to-image diffusion models. In Thirty-seventh Conference on Neural Information Processing Systems, Cited by: §2.1.
• D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. (2025) Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §1.
• S. Kakade and J. Langford (2002) Approximately optimal approximate reinforcement learning. In Proceedings of the nineteenth international conference on machine learning, pp. 267–274. Cited by: Appendix B.
• A. Kamath, K. Chang, R. Krishna, L. Zettlemoyer, Y. Hu, and M. Ghazvininejad (2025) GenEval 2: addressing benchmark drift in text-to-image evaluation. arXiv preprint arXiv:2512.16853. Cited by: Table 4, Table 4, Figure 1, Figure 1, §4.
• T. Karras, M. Aittala, T. Aila, and S. Laine (2022) Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems, Cited by: §2.
• Y. Kirstain, A. Polyak, U. Singer, S. Matiana, J. Penna, and O. Levy (2023) Pick-a-pic: an open dataset of user preferences for text-to-image generation. Advances in neural information processing systems 36, pp. 36652–36663. Cited by: §4.
• J. Li, Y. Cui, T. Huang, Y. Ma, C. Fan, Y. Cheng, M. Yang, Z. Zhong, and L. Bo (2025) Mixgrpo: unlocking flow-based grpo efficiency with mixed ode-sde. arXiv preprint arXiv:2507.21802. Cited by: §F.2.
• Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le (2023) Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, Cited by: §1, §2.
• J. Liu, G. Liu, J. Liang, Y. Li, J. Liu, X. Wang, P. Wan, D. Zhang, and W. Ouyang (2025) Flow-grpo: training flow matching models via online rl. arXiv preprint arXiv:2505.05470. Cited by: Figure 1, Figure 1, §1, §2.1, §2.1, §2.1, §3.2, §4.
• S. Liu, X. Dong, X. Lu, S. Diao, P. Belcak, M. Liu, M. Chen, H. Yin, Y. F. Wang, K. Cheng, et al. (2026) Gdpo: group reward-decoupled normalization policy optimization for multi-reward rl optimization. arXiv preprint arXiv:2601.05242. Cited by: §4.
• X. Liu, C. Gong, and qiang liu (2023) Flow straight and fast: learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, Cited by: §1, §2, §2.
• C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu (2022) DPM-solver: a fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In Advances in Neural Information Processing Systems, Cited by: §2.
• L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. (2022) Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, pp. 27730–27744. Cited by: §1.
• P. Qi, X. Zhou, Z. Liu, T. Pang, C. Du, M. Lin, and W. S. Lee (2026) Rethinking the trust region in llm reinforcement learning. arXiv preprint arXiv:2602.04879. Cited by: Appendix B, §1, §1, §3.1, §3.1, §3.2, Remark 3, Remark 4.
• A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. Cited by: §4.
• R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn (2023) Direct preference optimization: your language model is secretly a reward model. Advances in neural information processing systems 36, pp. 53728–53741. Cited by: §1.
• J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz (2015) Trust region policy optimization. In International conference on machine learning, pp. 1889–1897. Cited by: Appendix B, §1, §3.1.
• J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §1, §2.1, §3.1.
• Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §1, §2.1.
• J. Song, C. Meng, and S. Ermon (2021a) Denoising diffusion implicit models. In International Conference on Learning Representations, Cited by: §2.
• Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole (2021b) Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, Cited by: §2.1.
• Stability AI (2024) Stable diffusion 3.5. Note: https://stability.ai/news/introducing-stable-diffusion-3-5Model weights: https://huggingface.co/stabilityai/stable-diffusion-3.5-medium Cited by: §4.
• B. Wallace, M. Dang, R. Rafailov, L. Zhou, A. Lou, S. Purushwalkam, S. Ermon, C. Xiong, S. Joty, and N. Naik (2024) Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8228–8238. Cited by: §1.
• F. Wang and Z. Yu (2025) Coefficients-preserving sampling for reinforcement learning with flow matching. arXiv preprint arXiv:2509.05952. Cited by: Appendix A, §C.4, Figure 1, Figure 1, §1, §2.1, §4.
• J. Wang, J. Liang, J. Liu, H. Liu, G. Liu, J. Zheng, W. Pang, A. Ma, Z. Xie, X. Wang, et al. (2025) Grpo-guard: mitigating implicit over-optimization in flow matching via regulated clipping. arXiv preprint arXiv:2510.22319. Cited by: Figure 1, Figure 1, §1, §1, §4.
• X. Wu, Y. Hao, K. Sun, Y. Chen, F. Zhu, R. Zhao, and H. Li (2023) Human preference score v2: a solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09341. Cited by: §4.
• S. Xue, C. Ge, S. Zhang, Y. Li, and Z. Ma (2025a) Advantage weighted matching: aligning rl with pretraining in diffusion models. arXiv preprint arXiv:2509.25050. Cited by: §1.
• Z. Xue, J. Wu, Y. Gao, F. Kong, L. Zhu, M. Chen, Z. Liu, W. Liu, Q. Guo, W. Huang, et al. (2025b) Dancegrpo: unleashing grpo on visual generation. arXiv preprint arXiv:2505.07818. Cited by: §1.
• K. Zheng, H. Chen, H. Ye, H. Wang, Q. Zhang, K. Jiang, H. Su, S. Ermon, J. Zhu, and M. Liu (2026) DiffusionNFT: online diffusion reinforcement with forward process. In The Fourteenth International Conference on Learning Representations, Cited by: Figure 7, Figure 7, §G.3.1, §1, §4.

Appendix of Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models

Appendix A The Flow-DPPO Algorithm

We summarize the complete Flow-DPPO training procedure. The algorithm adopts the CPS sampling framework (Wang and Yu, 2025) for trajectory generation, uses group-relative advantage estimation, and applies the divergence-based mask during policy optimization.

Computational overhead. The divergence computation requires one additional forward pass of the velocity network to obtain at training time. However, this forward pass is already required for computing the log ratio, so the divergence comes at zero additional cost: it is simply the squared norm of a difference that is already computed.

Appendix B Policy Improvement Bound for Flow Models

We adapt the classical policy improvement theory (Kakade and Langford, 2002; Schulman et al., 2015) to the finite-horizon, undiscounted setting of flow model denoising, following the approach of Qi et al. (2026) for the LLM regime. We use the MDP notation introduced in Section˜2.1: decision steps indexed by , states , actions , and terminal reward .

B.1 Proof of Performance Difference Identity

Proof [Proof of Theorem˜1] We begin by expressing the performance difference via its definition. Since the reward is only a function of the terminal state and the prompt , we have:

where the integral is over all trajectories (we omit the deterministic transition structure for notational clarity).

The core of the proof is the telescoping identity for the difference in trajectory probabilities. Since , we apply the algebraic identity :

Substituting into the performance difference and converting to an expectation under :

We decompose this expression by adding and subtracting the term where the future ratio product is set to 1:

This completes the proof.

B.2 Proof of Policy Improvement Bound

Lemma 5(Bound on Trajectory-Level TV Divergence).

Let and be two policies for the flow model MDP. Let and denote the distributions over future sub-trajectories starting from state . Then:

where the expectation is over states visited under starting from .

Proof Let and , where . We have:

Applying the telescoping identity (which follows from the triangle inequality) and integrating:

For each term indexed by , integrating out the future actions yields 1 (since is normalized), leaving:

The inner integral is , and the outer integral defines an expectation over states under policy . Thus:

Proof [Proof of Theorem˜2] From Theorem˜1, we start with the exact performance difference identity:

Our goal is to upper-bound . We begin by bounding the reward by its maximum absolute value :

The inner expectation over future sub-trajectories is exactly twice the TV divergence between future trajectory distributions:

Applying Lemma˜5 and bounding each term by :

Substituting back into Eq. (19):

Evaluating the sum: . Therefore:

Substituting into the performance difference identity yields the desired bound:

This completes the proof.

B.3 A Tighter Policy Improvement Bound

The quadratic dependence on the horizon in Theorem˜2 can be overly pessimistic. By exploiting the fact that , we derive a tighter bound that is linear in .

Starting from the intermediate step in Eq. (19), the inner expectation is . Instead of applying Lemma˜5, we directly use the universal bound :

Combining both bounds, the policy improvement satisfies the composite guarantee:

where . The quadratic bound is tighter for small policy changes, while the linear bound is tighter for larger updates or longer horizons.

B.4 Connection to Gaussian Per-Step Divergence

For the Gaussian policies in Eq. (4), and , the TV divergence admits the closed form:

where is the standard normal CDF. Since is strictly monotonically increasing, the TV constraint is equivalent to:

This formally establishes that the Flow-DPPO mask, which blocks updates when , implements a trust-region constraint equivalent (up to a monotone rescaling) to constraining the per-step TV divergence. The policy improvement bound (Theorem˜2) thus provides a rigorous theoretical guarantee for Flow-DPPO: by enforcing a per-step divergence threshold, the penalty term remains controlled, ensuring monotonic policy improvement.

Appendix C KL Divergence Between Gaussian Policies

In this section, we derive the KL divergence between old and new policies in flow models and establish its connection to the TV divergence used in the policy improvement bound.

C.1 General Gaussian KL Divergence

Let and be two isotropic Gaussians in with the same covariance. The KL divergence is:

Note that this is symmetric in the means: when the covariances are identical.

C.2 Connection Between KL and TV in the Gaussian Setting

For the same pair of Gaussians, the TV divergence is:

Since both KL and TV are monotone functions of the single quantity , thresholding one is equivalent to thresholding the other. Specifically, the constraint is equivalent to , which in turn is equivalent to . This shows that the squared distance used in our mask is a unified divergence measure equivalent (up to monotone transformations) to both KL and TV divergences.

C.3 Application to Flow-SDE

For Flow-SDE (Eq. (2)), the per-step policy is where:

The difference in means is:

Substituting into Eq. (20) with :

C.4 Application to CPS

For CPS (Eq. (3)), the policy mean is and the variance is . Using and , the difference in means is:

Let . Then:

In previous work (Wang and Yu, 2025), the normalization is dropped for numerical stability, reducing the divergence to . We instead retain the full normalization in Eq. (22): because shrinks at later denoising steps, the factor amplifies the divergence where small velocity changes most affect the output, yielding a tighter constraint that prevents distribution collapse.

Appendix D Ratio Variance Analysis

We provide a detailed analysis of the variance of the log-ratio in flow models.

From Eq. (13), , where and . It follows that:

Thus . When the KL is moderate (e.g., ), the standard deviation of the log-ratio is , meaning that individual log-ratio samples fluctuate by around the mean of . In terms of the ratio itself, this corresponds to roughly a multiplicative spread.

Implication for clipping. With a typical clip parameter (i.e., clip range ), the log-clip range is . Comparing this narrow range with the log-ratio standard deviation of , we see that even for modest KL values, a significant fraction of samples will be clipped purely due to noise, not because the true divergence is excessive. This provides rigorous justification for replacing ratio-based clipping with direct divergence measurement.

Appendix E Towards a Predictive Divergence Mask

We recall the asymmetric mask in Flow-DPPO (Eq. (18)). The mask blocks the gradient (i.e., ) when two conditions hold simultaneously: (i) the divergence already exceeds the trust-region threshold, and (ii) a directional condition signals that the optimization would push the policy further away from . Concretely, the directional condition triggers when (the gradient would further increase an already-elevated ratio) or (the gradient would further decrease an already-reduced ratio). These two cases can be compactly unified as:

While this design is effective in practice, the directional indicator is a heuristic proxy for whether the upcoming gradient step will increase the divergence. In a ratio-based trust region (e.g., PPO clipping), this sign test is well-motivated: directly reflects the deviation of the single-sample Monte Carlo estimate of the importance ratio, so the sign of faithfully indicates whether the surrogate objective would drive the ratio further from unity. However, in a divergence-based trust region where the constraint is on , the connection is less direct. The ratio is a stochastic quantity evaluated at a single sampled action, whereas measures a distributional distance that integrates over all actions. A positive does not guarantee that the gradient step will increase , nor does a negative value guarantee a decrease.

In this section we exploit the Gaussian structure of flow model policies to derive a more principled masking criterion. We first predict how a single gradient step changes (§E.1), obtaining a closed-form expression that decomposes into a first-order directional term and a second-order magnitude term. The sign of the first-order term yields an exact directional criterion , which recovers the current sign test in the small-divergence regime but reveals a correction when the policy has already drifted. The full expression further accounts for the step size and gradient magnitude, leading to a predictive mask (§E.2) that directly forecasts whether the post-update divergence will exceed .

E.1 Predicting Post-Update Divergence

Fix a denoising step with state and suppress the time index for brevity. Write , , , and . The sampled action is with .

We derive how a single gradient step on the surrogate objective changes the divergence . The policy gradient with respect to is:

With effective learning rate , the updated mean is . Let . The predicted post-update divergence is:

From the ratio decomposition (Eq. (13)), , which gives . The first-order term thus simplifies to . The three terms in Eq. (24) have clear interpretations: (1) the current divergence ; (2) a first-order term whose sign determines whether the gradient step increases or decreases the divergence; (3) a non-negative second-order term that grows with the step size and gradient magnitude , always contributing positively to .

The first-order directional criterion. The direction of divergence change is mainly determined by the sign of the first-order term. Since and , this sign equals:

When this is positive, the gradient step increases ; when negative, it decreases . Equivalently, this is the sign of the inner product , confirming that the surrogate gradient projects onto the divergence-increasing direction.

Recovery of the current mask. In the small-divergence regime (which is the typical operating range when the trust region is effective), the correction and the criterion simplifies to . Since , this is equivalent to , which is exactly the directional condition in Eq. (23). Thus, the current Flow-DPPO mask implements the correct first-order divergence-increasing criterion in this regime.

The correction term. When is non-negligible (i.e., the policy has already drifted appreciably), the true divergence-change direction is rather than . The subtracted term shifts the decision boundary: a sample must have (rather than merely ) before the positive-advantage gradient is classified as divergence-increasing. Intuitively, when the policy has already moved away from , a moderately elevated ratio does not necessarily push it further; only sufficiently large ratios do. This yields a first natural refinement of the mask: replacing with as the directional indicator, which we call the first-order predictive mask:

This mask uses only quantities already computed during training (, , ) and requires no additional hyperparameters beyond the existing threshold .

E.2 The Predictive Mask

Based on Eq. (24), we define the (full) predictive mask that blocks updates whenever the predicted post-update divergence would exceed :

Comparison with the first-order mask. The first-order mask (Eq. (26)) only considers the direction of divergence change and still relies on the separate threshold condition . The full predictive mask unifies both into a single inequality: whether the gradient increases or decreases divergence is automatically encoded in the predicted value , and the threshold comparison is applied to the predicted (rather than current) divergence. This has two consequences. First, when , even a divergence-increasing step may be permitted if the predicted remains below . Second, when is close to , the second-order term may push above even when the first-order direction is “safe” (i.e., the first-order mask would not fire), correctly blocking large gradient steps near the trust-region boundary.

Recovery of the existing mask. In the limit , the second-order term vanishes and reduces to requiring that the first-order direction is positive and . Combined with the small-divergence approximation (), this exactly recovers the current Flow-DPPO mask (Eq. (23)).

E.3 Discussion on Mask Variants

Hierarchy of masks. The three masks form a natural hierarchy of increasing fidelity:

The current mask is the cheapest (no additional computation) and suffices when the trust region keeps small throughout training. The first-order mask refines the directional decision with zero additional hyperparameters. The full predictive mask additionally requires an effective learning rate estimate but provides quantitative divergence prediction.

Local approximation. The analysis treats as a free vector, whereas in practice it is the output of a neural network. The actual change in is coupled to changes at all other inputs through shared parameters. The predictive mask is thus a local approximation that is most accurate when the effective learning rate is small and the network Jacobian is approximately preserved across one step.

We leave empirical validation of the predictive masks to future work. The key contribution of this analysis is twofold: it provides a theoretical justification for the existing asymmetric condition (showing it is the correct first-order criterion in the small-divergence regime), and it charts a principled path toward more refined trust-region enforcement that exploits the Gaussian structure of flow model policies.

Appendix F Experimental Details

F.1 Computational Resources.

All experiments are conducted on NVIDIA H20 96GB GPUs. The main results in Table 1 require approximately 90K GPU hours in total (across SD3.5, FLUX2-klein-base-9B, and FLUX1-dev with all methods and reward configurations). Including all ablation studies, multi-epoch experiments, and auxiliary runs, the overall computational cost for all experiments reported in this paper is approximately 140K GPU hours.

F.2 Hyperparameters.

LoRA is used for all models. We use LoRA and for SD3.5, and for FLUX2-9B and FLUX.1-dev. The learning rate is set to for all models aligning to previous works. We set the training resolution to , number of denoising steps to 10 for SD3.5 and 14 for FLUX2-9B.

For GRPO setting, we use group size 16 and number of groups 64 per epoch for all methods. The PPO clip threshold is set to for Flow-GRPO and Flow-CPS, and for GRPO-Guard, following the official recommendation. The thresholds for KL-clipping are set to for Flow-DPPO and for Flow-DPPO+CPS due to their different KL-scaling factors. We applied the stragegy proposed in MixGRPO (Li et al., 2025) on all baselines and proposed methods for faster convergence and better performance. Specifically, we mix ODE and SDE sampling and randomly select 3 steps out of first half of the denoising steps for SDE sampling. The noise level for SDE sampling ( in CPS sampling) is set to .

For Diffusion-NFT, we follow the official implementation for SD3.5 for the rest of the hyperparameters, such as EMA schedule.

Appendix G Additional Experimental Results

G.1 Additional Training Curves

We provide the training curves on SD3.5 for the single-reward setting in Figure 7 (the multi-reward setting is in Figure 4 in the main body). We also provide the FLUX2-9B multi-reward training curves in Figure 8 and FLUX.1-dev in Figure 9.

We additionally provide training curves on FLUX.1-dev in Figure 9.

G.2 KL Divergence Curves

Figure 12 visualises the per-step KL divergence between the current and reference (pre-trained) model across all six training settings and two SDE schedules. The corresponding end-of-training values are reported in Table 2 of the main body.

G.3 Ablation Studies

G.3.1 Classifier-Free Guidance

Previous works found that CFG heavily affects the training convergence and performance (Zheng et al., 2026). Here, we study the effect of CFG on the training of Flow-DPPO on FLUX2-9B, as shown in Figure 10, where the CFG scale is set to 4.0 following the official recommendation. With CFG, Flow-DPPO variants still achieve state-of-the-art performance on the training reward (GenEval2) and mitigate catastrophic forgetting on the out-of-domain prompts, consistent with the observations in previous discussions. This shows that the divergence-based mask is robust under CFG and continues to deliver strong performance.

G.3.2 Reference KL Regularization Strength

We ablate the strength of the regularization term (controlled by ) on FLUX2-klein-base-9B under the multi-reward GDPO setting with CPS scheduling. Figure 11 shows the training reward curves and Figure 13 shows the KL divergence from the pretrained model. A moderate regularization strength () further mitigates early-stage reward hacking on auxiliary objectives (PickScore, HPSv2, etc.), thereby balancing the gradients across rewards and yielding an additional improvement in final GenEval2 performance over the unregularized baseline, without degrading end-of-training performance on any individual reward.

G.4 Quantitative Summary on GenEval2

To complement the per-setting training-curve figures above, Tables 4 and 3 report the end-of-training Soft TIFA score on GenEval2 for each method. Table 4 additionally reports end-of-training ancillary CLIP, PickScore, and HPSv2 rewards on both the in-domain GenEval2 prompt set and the held-out out-of-domain PickScore validation prompts, contextualising both SD3.5-medium and FLUX2-klein-base-9B by stacking six blocks: published reference numbers for state-of-the-art text-to-image systems, the corresponding pretrained-baseline scores (no RL), and the five RL fine-tuning algorithms applied to each base model under both the single-reward (GenEval2-only) and multi-reward (GenEval2 + CLIP + PickScore + HPSv2) configurations. Table 3 then expands the per-method Soft TIFA comparison to all five training settings reported in this paper.