Mixtral 8x7B has 47 billion parameters across 32 layers, all loaded in memory at 16 bits, yet each token uses about the compute of a 13B model and it runs about as fast as one.

In a dense model that size, every token would go through all 47 billion.