Why is RoPE not applied on T2V/T2A cross attention?

[wsntxxn]

Hello, thanks for you great work!

When I read the code, I have a quick question: according to the implementation here, the cross attention between audio/video and text does not use rope_apply on q and k. Why is RoPE not applied here?

[shanedeirmagic]

That is actually quite interesting! So after looking at it, I found that at line 107, the text cross-attention happens, x = flash_attention(q, k, v, k_lens=context_lens).

So rope isn't being aplied to q and k_target until line 118. Which means that only the cross-modal attention (between the audio and the video) get the benefits of the rope embeddings. Which is... strange. I'll test out the difference and get back to you! Because you miss out on the benefits of applying it to text cross-attention too! Could be a cool experiment!

Thanks for the insight!