[QST] Are prefetch slots shared across MoE layers or allocated per layer?

[Marsssqqq]

In a multi-layer MoE training setup, are the physical [E, E+B) prefetch
slots:

1. one B-slot bank shared by all MoE layers, or
2. a separate B-slot bank for every MoE layer?

The README says that every layer has an [E+B, H, H'] VMM range, while also
saying that the physical [E, E+B) prefetch-slot memory comes from one
process-global pool shared by all layers.

Does each layer's VMM range map its [E, E+B) tail to the same physical
B-slot bank?

If so, after a later layer overwrites that shared bank, how are the expert
weights required by backward restored? Does the training integration call
prefetch_weight() again before each layer's expert backward, or use another
mechanism?

[asp0ex]

The physical memory of prefetch slots and grad reduce slots are shared across layers to reduce memory overhead. in bwd, prefetch is called again to get the redundant experts.