DiffusionART fine-tunes diffusion image and video generators with RL by pooling each prompt’s rewards into one shared improvement direction applied to reverse-process states, then adding a drift-sized second update for stability, cutting GPU hours by roughly 12× versus prior forward-process RL on a 13B video model.
Suppose you have a text-to-image or text-to-video model and a reward model scoring outputs (visual quality, prompt following, human preference). You want to fine-tune the generator to score higher. Standard supervised training only matches the data distribution; it does not directly chase reward.
Two families of online RL exist for diffusion. Flow-GRPO optimizes each denoising transition using a trajectory-level reward signal, which gives poor credit assignment because any one step only partly determines the final image. DiffusionNFT is faster: it noises the generated clean sample and regresses velocities weighted by whether the sample scored high or low. But it only stays stable with two tricks glued on: joint positive-plus-negative training, and an Exponential moving average copy of the model whose decay rate must be retuned per task. In the authors’ own FLUX.1-dev experiments, carrying the GenEval-tuned settings over caused training collapse. That retuning cost is what blocks scaling.
The paper first diagnoses why NFT needs its stabilizers, then fixes the cause.
Diagnosis. NFT builds its regression target from a forward velocity (the ideal denoising direction implied by a noised clean sample), but the model actually generates samples using its reverse velocity. These two differ. The authors show NFT’s gradient decomposes into a real reward-improving part plus a hidden pull toward the forward velocity, which they call the forward–reverse consistency force. Joint positive-and-negative training cancels this force only on average across samples, not at each individual noisy state. EMA then papers over the residual by pulling the trained model back toward a lagged reference. Their ablations isolate that it’s the restoration pull from the lagged reference, not off-policy data per se, that stabilizes training.
Fix 1: aggregation-first guidance. Instead of per-sample forward-velocity targets, DiffusionART computes one direction per prompt: the covariance between rewards and generated endpoints across the prompt’s K samples, RMS-normalized to a fixed magnitude. Call this G(c). The target for every reverse-process state visited while generating for that prompt is simply the model’s own cached prediction plus G(c). Because the target never references a forward velocity, the consistency force is gone by construction, not by averaging.
Fix 2: two-update restoration. Split each rollout batch into two disjoint halves. Update once on the first half; the second update then automatically contains a restoring term proportional to how far the model drifted, with no decay schedule to tune.
Fix 3: trust-region mask. Since rollouts use a deterministic ODE sampler solver, there is no natural KL to clip against. The authors borrow Flow-GRPO’s ODE-to-SDE trick to define a reference Gaussian transition, then mask the second update at states where drift exceeds a fixed threshold and the update would push further outward. One threshold, no per-task tuning.
# One DiffusionART iteration
for prompt c in batch:
xs, rewards, v_old = rollout(policy, c, K=k) # ODE sampler
G = normalize(cov(rewards, xs)) # shared direction
v_target = v_old + G # frozen per state
B1, B2 = split(all_states)
update(policy, mse(policy(B1), v_target)) # pure guidance
mask = trust_region(drift(policy, v_old)) # KL proxy
update(policy, mask * mse(policy(B2), v_target)) # guidance + restoration
On Stable Diffusion 3.5 Medium with GenEval, OCR, and PickScore rewards separately and jointly, DiffusionART matches or beats DiffusionNFT with its original per-task tuning, and reaches Flow-GRPO’s step-500 reward with 10×, 5.2×, and 9.6× fewer training steps respectively.
Scaling to FLUX.1-dev (12B), where DiffusionNFT’s transferred hyperparameters fluctuate badly, DiffusionART learns fastest among all compared methods using the same recipe as the smaller model.
On HunyuanVideo-13B text-to-video with VideoReward scoring visual and motion quality, with equal weights DiffusionART reaches the late-stage reward of DiffusionNFT (EMA) and Flow-GRPO-Fast with 12.02× and 11.72× fewer GPU hours and 28.8% and 53.0% higher final training rewards. The pairwise-feedback variant performs comparably to the scalar version, and shrinking the per-prompt group from 8 to 2 cuts GPU hours 3.4× without hurting final quality.
An ablation on OCR shows the single-update variant improves fast but gradually deteriorates, while the two-update version stays stable. This supports the paper’s claim that the second update’s role is specifically restoration, not more guidance. The trust-region mask matters mostly for video; on image tasks training is stable without it.
•
If you are already running DiffusionNFT or Flow-GRPO-style post-training and spend time tuning EMA decay per reward model, the aggregation-first target plus two-update restoration is worth testing as a drop-in. The paper reports no per-task tuning across image and video, including Stable Diffusion 3.5 Medium, FLUX.1-dev, HunyuanVideo-13B, and Wan2.1-I2V-14B.
•
If your reward model only returns pairwise preferences, the DiffusionART-Pairwise variant swaps scalar rewards for {0, 1/2, 1} labels in the same covariance formula. For K=2 endpoints per prompt the guidance reduces to the winner-minus-loser direction, which may be attractive when generating many samples per prompt is expensive.
•
The paper’s analysis of why NFT needs EMA (restoration, not off-policy data) is useful even if you don’t adopt DiffusionART: it reframes EMA tuning as a drift-control knob, so a two-update split can substitute when per-task EMA tuning is painful.
•
The method does not require a differentiable reward model and uses no reference-policy KL regularization in the reported runs. The paper does not mention a public code release.
•
All comparisons are the authors’ own re-implementations and single-seed training curves; there is no third-party reproduction. The headline 12× GPU-hour gap is against DiffusionNFT with EMA on one video reward combination; other weightings give different multipliers.
•
The theoretical story (consistency force, restoration decomposition) is derived at the on-policy initialization point and under standard moment assumptions. It explains observed behavior rather than proving convergence.
•
Rewards are proxies. All gains are measured on the same reward models used for training, with VideoReward serving as both trainer and evaluator for video. The paper does not report independent human studies at scale.
•
Trust-region masking uses a fixed threshold of 1e-5 and noise coefficient 0.8; the paper does not sweep these or test transfer beyond the shown settings.