Your RNN character language model trains smoothly, but every few thousand steps one batch has a gradient norm about 100x the usual, and the loss jumps and takes a long time to recover.
Cutting the learning rate tenfold stops the jumps but makes training far slower.