it would be nice to support expert weight prefetch when SSD expert weight streaming is enabled. The tokens predicted by the MTP head could be used to deduce what expert weights are needed in next round, and this information could be used to prefetch weight on SSD before they are needed by the GPU, this in theory should improve throughput.
MTP might predict the wrong token, but it not necessarily means the wrong token will lead the wrong expert weight. The next expert weight prediction might be more forgiving than next token prediction.
there are maybe one or two people that have been working on this feature on llama.cpp, but so far, no working demo has been shown publicly.
it would be nice to support expert weight prefetch when SSD expert weight streaming is enabled. The tokens predicted by the MTP head could be used to deduce what expert weights are needed in next round, and this information could be used to prefetch weight on SSD before they are needed by the GPU, this in theory should improve throughput.
MTP might predict the wrong token, but it not necessarily means the wrong token will lead the wrong expert weight. The next expert weight prediction might be more forgiving than next token prediction.
there are maybe one or two people that have been working on this feature on llama.cpp, but so far, no working demo has been shown publicly.