ServingYour LLM service batches prefill for new requests with decoding for running ones on the same GPUs. Whenever a long prompt arrives, time per output token spikes for every streaming request.
Your targets are separate: time to first token for new users, time per output token for everyone mid-answer.