Long Documents

We propose NextLevelBERT, a Masked Language Model that operates on text-embedding representations of whole chunks instead of tokens, and show it handles long-document tasks effectively while outperforming much larger embedding models.