Repository navigation
fix(diffusion): keep replay transition means at SDE-native fp32 - #444
Merged
Merged
Conversation
Drop the trajectory-dtype cast on ReplayResult.prev_sample_means in the SD3/Qwen-Image serial replay and the shared batched replay mixin. The SDE kernel computes the mean in fp32 and log-prob scores that fp32 mean, but the cast narrowed it to bf16 before FlowDPPO's Gaussian KL — whose 1e-5 mask threshold sits below bf16 rounding — so the mask scored a different distribution than the one log-prob scored.
Yuanjia-Lee
requested review from
CjhHa1,
Jayce-Ping,
celve and
haonan3
as code owners
September 12, 2026 13:29
CjhHa1
requested review from
Ideny42,
KemingWu,
heguangxin and
leviking98z-rgb
as code owners
September 13, 2026 04:11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
FlowDPPO's KL mask and its log-prob score different distributions for the same
transition. The SDE kernel computes the transition mean μ_θ in fp32 and scores
log-prob against that mean, but
stage.replay()narrowed the returnedprev_sample_meanstotrajectory_dtype(typically bf16), and the mask(μ_θ − μ_old)²/(2σ_t²)compares those narrowed means againstkl_mask_threshold: 1e-5(sd3_flowdppo.yaml). One bf16 rounding of a|μ|≈1.6 value has std 2.3e-3, the same size as the μ-delta the threshold is
sized for (RMS ≈ 2e-3 at the recipe's σ window): small legitimate updates get
masked out, sub-ULP deltas read as KL = 0, and the β>0 reference penalty on the
same means reads 0.
The fix removes that cast from every diffusion replay implementation: SD3,
Qwen-Image/Edit-Plus, WAN 2.1/2.2, HunyuanVideo 1.0/1.5, HunyuanImage 3,
Z-Image, FLUX.2-klein, LTX2, Boogu-Image, Bagel, and SenseNova U1, plus the
shared
BatchedStepReplayMixin. It also preserves the rollout-owned old-meananchors in Bagel and SenseNova U1. MiniMax-H3 already returned un-narrowed
means.
log_probskeeps itslogprob_dtypecast.Measured through the real replay→loss→optimizer code at the SD3 recipe's
hyperparameters: replay's log-probs recomputed from the returned means match
exactly after the fix (max|Δ| = 0.0) and deviate by ~2e-5 with the cast. A
small weight perturbation flipped the mask on 5/24 sample-steps (native KL
~1e-6, narrowed up to 1.25e-5) and reversed 61/256 gradient components (cosine
0.76); with no flips the two arms' gradients are identical (cosine 1.0), so the
reversal is the mask, not noise.
Related Issue
N/A
Test Plan
SKIP=no-commit-to-branch pre-commit run --files <all 17 changed files> --show-diff-on-failure→ all hooks pass locally; the pre-push hook also passes on the follow-up commit.
sample/noise_predverifies both the returnedtransition and
prev_sample_meanremaintorch.float32.narrows
prev_sample_means/sde_meanstotrajectory_dtype.tests/tree by policy test: remove tests directory #99/test: remove tests directory #267): fullSD3DiffusionStage.diffuse/replay(serial and batched, bit-identical) andthe
FlowDPPOloss + AdamW (lr 3e-4, wd 0) at eta 0.7 / threshold 1e-5 /10-step shift-3 schedule / B=8, with a small deterministic transformer
standing in for the SD3.5 checkpoint; CPU and CUDA bf16-autocast on one 4090.
Numbers are in the Summary.
Compatibility / Risk
Consumers already align old/reference means to
new_means.dtypeor computethe KL in fp32; no dtype assertions exist. Numerics of mean-based KL paths
change by design;
log_probsare untouched.trajectory_precisionisbf16. For the SD3 recipe's
[8, 3, 16, 64, 64]micro-batch, the frozen CPUanchor is 6 MiB instead of 3 MiB. Bagel and SenseNova U1 also retain their
rollout-owned old-mean anchors in fp32 rather than bf16.
Reviewer Notes
ReplayResult.prev_sample_meansand inunirl/sde/README.md; serial andbatched replay no longer differ by dtype.
per-step batching and carried the same cast in its new code; feat(opd): harden teachers for video KL, meta-init, and resume #432, perf(bagel): pack FlowGRPO replay steps #384, and
refactor(diffusion): make role residency one choice per role, not per phase #428 do not touch this boundary.
Checklist
(Contract recorded in
unirl/sde/README.mdGotchas and onReplayResult.prev_sample_means.)