Is it applicable to the DeepSeek MoE model?

[yanghailong-git]

Hi, I’m wondering if these algorithms are compatible with the DeepSeek MoE model, and if they could improve its performance?

[PiotrNawrot]

Hey, yes. I've checked out DeepSeek MoE model and in the paper report the authors discuss that for MoE models there's a different ratio between FFN and Attention parameters compared to non-MoE models. They believe that assigning more parameters to FFN Experts impacts performance on some tasks. Sparse Attention may further limit the attention expressivity, but it doesn't have to. We're currently evaluating Gemma 3 models to understand how Sparse Attention impacts models which applied some form of Attention compression during training, I believe that the insights from Gemma 3 evaluation may apply also to MoE models.

TLDR: It's worth trying, I believe that it would work well, but the maximum tolerable compression can be smaller compared to non-MoE models.

[yanghailong-git]

Hey, yes. I've checked out DeepSeek MoE model and in the paper report the authors discuss that for MoE models there's a different ratio between FFN and Attention parameters compared to non-MoE models. They believe that assigning more parameters to FFN Experts impacts performance on some tasks. Sparse Attention may further limit the attention expressivity, but it doesn't have to. We're currently evaluating Gemma 3 models to understand how Sparse Attention impacts models which applied some form of Attention compression during training, I believe that the insights from Gemma 3 evaluation may apply also to MoE models.

TLDR: It's worth trying, I believe that it would work well, but the maximum tolerable compression can be smaller compared to non-MoE models.

Thanks, I’d like to know whether these KV cache optimization algorithms are compatible with DeepSeek’s MLA, and if they would be effective?