强化学习 颠覆级 暂无讲解视频
发表时间
2026-10-08
arXiv
2610.12572

收录解读

在线强化学习微调扩散模型在图像和视频生成中面临效率和稳定性的双重挑战。现有反向过程方法如Flow-GRPO因信用分配不佳而效率低下,而前向过程方法如DiffusionNFT虽高效但依赖精心调参的EMA离策略训练,稳定性不足。

本文提出DiffusionART,通过聚合每个提示的奖励为共享的奖励改进方向,从构造上消除了前向-反向一致性力,并引入信任区域掩码的第二次更新实现漂移自适应恢复,无需EMA调参。方法不要求可微奖励模型,支持成对奖励模型,且样本效率更高。

全文支持从正反向一致性梯度干扰的分析导出奖励聚合与两步信赖域恢复,不依赖 EMA;在 SD3.5、FLUX 12B、Hunyuan 13B 的图像、文生视频和图生视频上验证。12.02/11.72 倍 GPU 小时改善及 28.8%/53% 奖励提升对应特定 VQ:MQ=1:1 训练设置,并非所有质量指标普遍提升;另有不同权重下的留出评估。复用训练机制与跨多类生成模型证据满足 disruptive,纠正机器 24 分仍判 breakthrough 的过度保守理由。

未达到 paradigm:预印本尚缺独立复现与长期采用证据,部分基线 EMA 参数未针对 FLUX 重调,奖励模型评价和随机种子覆盖仍有限。

原始摘要

Online reinforcement learning (RL) fine-tunes diffusion models with black-box rewards, but reverse-process methods such as Flow-GRPO are limited in efficiency by poor credit assignment, while forward-process methods such as DiffusionNFT are efficient yet unstable without carefully tuned exponential-moving-average (EMA) off-policy training. We show that NFT-style objectives contain a hidden forward--reverse consistency force that joint positive--negative training cancels only partially, and that EMA stabilizes training through restoration rather than off-policy rollouts. We introduce DiffusionART (Diffusion Reinforcement with Aggregation-First Reward Tilting), which aggregates each prompt's rewards into a shared reward-improving direction for reverse-process states, removing the consistency force by construction, and adds a trust-region-masked second update for drift-adaptive restoration without EMA tuning. It requires no differentiable reward model, remains effective with fewer samples per prompt, and supports pairwise reward models (DiffusionART-Pairwise). Across image and video generation, DiffusionART trains efficiently and stably without per-task tuning. In particular, on HunyuanVideo-13B with equally weighted visual and motion quality rewards, DiffusionART reaches the late-stage mean reward levels of DiffusionNFT and Flow-GRPO-Fast with 12.02 and 11.72 times fewer GPU hours, respectively, while achieving 28.8% and 53.0% higher mean training rewards.

相关论文

链接