Question about Causal-rCM

[fy-zhang-ctrl]

Hi @zhengkw18 , nice work! About Causal-rCM, I have the following questions:

• Is Causal-rCM missing the DMD-1-like forward-KL cooperative term in DMD? More specifically, the contribution of rCM is to replace the forward-KL term in DMD-1, originally implemented via ODE regression, with sCM using JVP. However, it seems that the causal version of DMD no longer includes this term.
• Also, regarding APT2 in your figure, are the only two differences as follows: (1) Causal-rCM replaces the GAN in APT2 with DMD; and (2) Causal-rCM can support sCM? In addition, do you know where I can find an open-source implementation of APT2? Do you think causal-rCM-style DMD is better than APT2-style GAN?

[fy-zhang-ctrl]

Also, I noticed that the VBench scores you reported are much lower than those in papers such as Self Forcing, especially for Wan2.1（bidirectional） itself. Why is that? Did you use the official Self Forcing evaluation set, or did you modify VBench (e.g., use the extended prompts) yourself?

[zhengkw18]

Thanks for your attention to our work!

1. DMD-1 uses ODE-regression jointly with DMD loss. There are 2 differences (1) CM is significantly better than ODE-regression, the comparison will be in the paper (2) We also tried joint training of CM and DMD. In fact, the bidirectional rCM adopts joint training, but in the causal case, we find joint training has issues. Multiple stages is a better strategy.

2. In the algorithm aspect, you are correct. APT2 is not open-sourced, but you can refer to the self-forcing and teacher-forcing implementations in the Self-Forcing codebase (for example, https://github.com/guandeh17/Self-Forcing/blob/33593df3e81fa3ec10239271dd2c100facac6de1/model/diffusion.py#L30)

Apart from the algorithmic differences, another contribution is on the infrastructure part.

3. We have no idea how Self-Forcing tests the VBench score of Wan2.1 teacher. Possibly they are from the open leaderboard of VBench, but the leaderboard results may use some additional processing during sampling/evaluation. We cannot reproduce their numbers using the open VBench suite.

For our evaluation, we use the official GPT-4o-augmented prompts and official VBench evaluation suite. Typically, using augmented prompts will result in higher scores than non-augmented ones. But even in this case, we cannot get scores as high as what Self-Forcing reports. Our evaluation scripts are fully open-sourced and reproducible. You can see https://github.com/NVlabs/rcm/tree/main/evaluation/vbench_text2video

[fy-zhang-ctrl]

Thank you for such a timely response. This really clears up many of my questions!

BTW, why do you think it is necessary to replace the GAN in APT2 with DMD? The results shown on the APT2 project page seem to be quite strong. Do you think the difference is due to data or model choices, or is it caused by GAN itself? Why does Causal-rCM replace APT2’s GAN with DMD?

I understand that GAN training can be unstable, but is there concrete evidence showing that DMD is necessarily better than a well-optimized GAN?

[fy-zhang-ctrl]

Also I noticed that Guande He is in your author list, who is also the author of Self Forcing. Maybe ask him why there's such significant difference regrading VBench evaluation on Wan-Bidirectional?

[zhengkw18]

Thank you for such a timely response. This really clears up many of my questions!

BTW, why do you think it is necessary to replace the GAN in APT2 with DMD? The results shown on the APT2 project page seem to be quite strong. Do you think the difference is due to data or model choices, or is it caused by GAN itself? Why does Causal-rCM replace APT2’s GAN with DMD?

I understand that GAN training can be unstable, but is there concrete evidence showing that DMD is necessarily better than a well-optimized GAN?

This is a great question. I believe a well-optimized GAN can outperform DMD because GAN can utilize real data. However, GAN requires heavy architecture/hyperparameter tuning and has few open-sourced references, while DMD is simple to implement. For the current stage, we want to show that the simplest TF-CM+SF-DMD recipe can already bring impressive results. We leave the exploration of GAN for future work.

[zhengkw18]

Also I noticed that Guande He is in your author list, who is also the author of Self Forcing. Maybe ask him why there's such significant difference regrading VBench evaluation on Wan-Bidirectional?

He was not responsible for VBench evaluation

[fy-zhang-ctrl]

Thank you for such a timely response. This really clears up many of my questions!
BTW, why do you think it is necessary to replace the GAN in APT2 with DMD? The results shown on the APT2 project page seem to be quite strong. Do you think the difference is due to data or model choices, or is it caused by GAN itself? Why does Causal-rCM replace APT2’s GAN with DMD?
I understand that GAN training can be unstable, but is there concrete evidence showing that DMD is necessarily better than a well-optimized GAN?

This is a great question. I believe a well-optimized GAN can outperform DMD because GAN can utilize real data. However, GAN requires heavy architecture/hyperparameter tuning and has few open-sourced references, while DMD is simple to implement. For the current stage, we want to show that the simplest TF-CM+SF-DMD recipe can already bring impressive results. We leave the exploration of GAN for future work.

Thx a lot! Therefore, from my perspective, there is currently no evidence showing that Causal-rCM is better than APT2, since the latter is not open-sourced and we cannot rule out the possibility that it is a well-optimized GAN that outperforms Causal-rCM.

Anyway, thank you for your work.

[zhengkw18]

Thank you for such a timely response. This really clears up many of my questions!
BTW, why do you think it is necessary to replace the GAN in APT2 with DMD? The results shown on the APT2 project page seem to be quite strong. Do you think the difference is due to data or model choices, or is it caused by GAN itself? Why does Causal-rCM replace APT2’s GAN with DMD?
I understand that GAN training can be unstable, but is there concrete evidence showing that DMD is necessarily better than a well-optimized GAN?

This is a great question. I believe a well-optimized GAN can outperform DMD because GAN can utilize real data. However, GAN requires heavy architecture/hyperparameter tuning and has few open-sourced references, while DMD is simple to implement. For the current stage, we want to show that the simplest TF-CM+SF-DMD recipe can already bring impressive results. We leave the exploration of GAN for future work.

Thx a lot! Therefore, from my perspective, there is currently no evidence showing that Causal-rCM is better than APT2, since the latter is not open-sourced and we cannot rule out the possibility that it is a well-optimized GAN that outperforms Causal-rCM.

Anyway, thank you for your work.

Exactly! Maybe you can reproduce the APT2 work and surpass us😀