You serve an 8B chat model on one H100: each token reads all 16 GB of weights in about 5 ms, so one user gets about 200 tokens a second, and batching 32 users onto one weight read still gives each about 160.

At 256 users, each about 1,000 tokens into a conversation, everyone drops to around 70 while the arithmetic units sit idle for most of each step.