Your model uses rotary embeddings, which rotate each query/key dimension pair by $m{\theta }_i$ at position $m$, and was pretrained on 4,096-token sequences, within which the slowest pairs never complete a full turn.

On 32,000-token documents perplexity is normal for the first 4,000 tokens and then shoots up, and you have budget for about 1,000 fine-tuning steps on long documents.