Your MoE layer has eight feed-forward experts. A linear router scores every expert for each token, softmaxes the top two scores and mixes those two experts' outputs. Each expert has a fixed capacity, and tokens routed to a full expert skip the layer.

Left alone, early score advantages compound until most tokens pile onto a few experts, so you add an auxiliary loss ${\sum }_i{f}_i{P}_i$, where $f_i$ is the fraction of tokens sent to expert $i$ and $P_i$ is its mean router probability; it is smallest when load is even. You sweep its weight: at 0.01 the loads are close to even and validation loss is lowest, and at 1.0 the loads are almost perfectly even but validation loss is clearly worse.