You serve an 8B chat model on one H100, and with 256 users about 1,000 tokens into their conversations, each gets around 70 tokens a second while the arithmetic units sit idle for most of each step.

The extra time goes to reading every user's KV cache from memory.