Your chatbot runs long conversations on a model pretrained on 4K-token sequences. To bound memory, its KV cache holds only the most recent 4,000 tokens and evicts the oldest one each time a new token arrives.

Replies are fine until a conversation passes 4,000 tokens. Within a few tokens of the first evictions, which remove the opening tokens of a generic system prompt, perplexity shoots up and the output turns to gibberish.