Long contextYou extend a 4K-context model to 128K with continued pretraining on long documents, which are mostly books. Long-context loss improves, but short-text loss in other domains rises.
Long contextLong contextLong contextYou extend a 4K-context model to 128K with continued pretraining on long documents, which are mostly books. Long-context loss improves, but short-text loss in other domains rises.