---
title: "NeoBERT — paper by Breton et al. (2025) — summary and ideas"
description: "Encoder models such as BERT and RoBERTa still underpin many NLP pipelines, including retrieval… 57 references, cited by 20 papers here."
authors: ["Lola Le Breton", "Quentin Fournier", "Mariam El Mezouar", "John X. Morris", "Sarath Chandar"]
year: 2025
venue: "arXiv:2502.19587"
source: "https://arxiv.org/abs/2502.19587"
doi: "10.48550/arxiv.2502.19587"
arxiv: "2502.19587"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/breton-2025-neobert-next-generation-bert/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/breton-2025-neobert-next-generation-bert/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/5/) / NeoBERT: A Next-Generation BERT

# NeoBERT: A Next-Generation BERT

**Authors:** Lola Le Breton, Quentin Fournier, Mariam El Mezouar, [John X. Morris](https://md.research.elamur.ai/free/en/authors/john-x-morris/) and [Sarath Chandar](https://md.research.elamur.ai/free/en/authors/sarath-chandar/) · **Year:** 2025 · **Venue:** arXiv:2502.19587 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2502.19587) · [PDF](https://arxiv.org/pdf/2502.19587) · [DOI 10.48550/arxiv.2502.19587](https://doi.org/10.48550/arxiv.2502.19587) · [arXiv 2502.19587](https://arxiv.org/abs/2502.19587) · [OpenAlex](https://openalex.org/W4416149152)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [20](https://md.research.elamur.ai/free/en/papers/breton-2025-neobert-next-generation-bert/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Encoder models such as BERT and RoBERTa still underpin many NLP pipelines, including retrieval, classification and embeddings. Yet their pre-training has barely changed in years, while decoder LLMs advanced quickly. Most recent embedding work only refines fine-tuning on top of these aging backbones. NeoBERT addresses this by pre-training a new encoder from scratch with modern components:
- a deeper network of the same width (28 layers, hidden size 768);
- SwiGLU, RoPE and Pre-RMSNorm;
- the RefinedWeb corpus and a 20% MLM masking rate;
- a two-phase schedule that extends the context from 1,024 to 4,096 tokens.

The authors tested each change by fully pre-training ten successive models and scoring them on GLUE. To compare backbones fairly, they also designed one simple contrastive fine-tuning protocol and applied it identically to every model before MTEB evaluation. Under this protocol, the 250M-parameter NeoBERT beats BERT large, RoBERTa large, NomicBERT and ModernBERT, and it runs faster than ModernBERT on long inputs. Because NeoBERT keeps the base hidden size and all code, data and checkpoints are public, it can replace older encoders directly as a stronger, reproducible backbone.

## Key points
- **Architecture:** NeoBERT has about 250M parameters, 28 layers, hidden size 768 and 12 attention heads. Depth was increased rather than width, following the depth-to-width findings of Levine et al. It uses SwiGLU, RoPE and Pre-RMSNorm, and it is compatible with YaRN for further context extension.
- **Pre-training:** The model was trained on RefinedWeb (about 2.8TB) for roughly 2.1T tokens. The setup used MLM at a 20% masking rate, AdamW with cosine decay, and batches of about 2M tokens.
- **Long context:** The model was first trained at 1,024 tokens and already generalized to nearly 3,000 tokens. An extra 50k steps on sequences up to 4,096 tokens produced NeoBERT 4096. This window is 8× longer than RoBERTa's and 2× longer than NomicBERT's.
- **Results:** With the uniform fine-tuning protocol, NeoBERT leads all compared backbones on MTEB. On GLUE it surpasses BERT large and NomicBERT and roughly matches RoBERTa large. With more advanced fine-tuning, it reached the top of the MTEB leaderboard among models under 400M parameters.
- **Efficiency and release:** On a single A100, NeoBERT is 46.7% faster than ModernBERT base at 4,096 tokens. All code, data, checkpoints and training scripts are released (Hugging Face: `chandar-lab/NeoBERT`; GitHub: `chandar-lab/NeoBERT`).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Lola Le Breton, Quentin Fournier, Mariam El Mezouar, John X. Morris and Sarath Chandar. 2025. NeoBERT: A Next-Generation BERT. arXiv:2502.19587.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{breton2025neobert,
  title = {{NeoBERT: A Next-Generation BERT}},
  author = {Lola {Le Breton}, Quentin Fournier, Mariam {El Mezouar}, John X. Morris and Sarath Chandar},
  year = {2025},
  doi = {10.48550/arxiv.2502.19587},
  eprint = {2502.19587},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.19587}
}
```

- [← previous MLaKE: Multilingual Knowledge Editing Benchmark for Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2025-mlake-multilingual-knowledge-editing-benchmark/)
- [next → RARE: Retrieval-Augmented Reasoning Enhancement for Large Language Models](https://md.research.elamur.ai/free/en/papers/tran-2025-rare-retrieval-augmented-reasoning-enhancement/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/5/)

## References

57 references, 33 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, et al. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–15. IEEE, 2022. [find](https://scholar.google.com/scholar?q=Reza+Yazdani+Aminabadi%2C+Samyam+Rajbhandari%2C+Ammar+Ahmad+Awan%2C+et+al%2E+Deepspeed%2Dinference%3A+enabling+efficient+inference+of+transformer+models+at+unprecedented+scale%2E+In+SC22%3A+International+Conference+f)
2. Simran Arora, Patrick Lewis, Angela Fan, et al. Reasoning over public and private data in retrieval-based systems, 2022. URL https://arxiv.org/abs/2203.11027. [↗](https://arxiv.org/abs/2203.11027)
3. Alexei Baevski, Sergey Edunov, Yinhan Liu, et al. [Cloze-driven pretraining of self-attention networks](https://md.research.elamur.ai/free/en/papers/baevski-2019-cloze-driven-pretraining-self-attention/). In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on…
4. Payal Bajaj, Daniel Campos, Nick Craswell, et al. [Ms marco: A human generated machine reading comprehension dataset](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/), 2018. URL https://arxiv.org/abs/1611.09268. [↗](https://arxiv.org/abs/1611.09268)
5. Samuel R. Bowman, Gabor Angeli, Christopher Potts, et al. [A large annotated corpus for learning natural language inference](https://md.research.elamur.ai/free/en/papers/bowman-2015-large-annotated-corpus-learning-natural/), 2015. URL https://arxiv.org/abs/1508.05326. [↗](https://arxiv.org/abs/1508.05326)
6. Daniel Cer, Mona Diab, Eneko Agirre, et al. [Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation](https://md.research.elamur.ai/free/en/papers/cer-2017-semeval-2017-task-1-semantic/). In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Association for Computational Linguistics, 2017. doi: 10.18653/v1/s1…
7. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multilingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2024. [URL https://arxiv.org/abs/2402.03216](https://md.research.elamur.ai/free/en/papers/chen-2024-m3-embedding-multi-linguality/). [↗](https://arxiv.org/abs/2402.03216)
8. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, et al. [Unsupervised cross-lingual representation learning at scale](https://md.research.elamur.ai/free/en/papers/conneau-2020-unsupervised-cross-lingual-representation-learning/), 2020. URL https://arxiv.org/abs/1911.02116. [↗](https://arxiv.org/abs/1911.02116)
9. Tri Dao. [Flashattention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/), 2023. URL https://arxiv.org/abs/2307.08691. [↗](https://arxiv.org/abs/2307.08691)
10. DeepSeek-AI, Daya Guo, Dejian Yang, et al. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/), 2025. URL https://arxiv.org/abs/2501.12948. [↗](https://arxiv.org/abs/2501.12948)
11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, et al. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/), 2019. URL https://arxiv.org/abs/1810.04805. [↗](https://arxiv.org/abs/1810.04805)
12. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, et al. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/), 2024. URL https://arxiv.org/abs/2407.21783. [↗](https://arxiv.org/abs/2407.21783)
13. Katja Filippova and Yasemin Altun. [Overcoming the lack of parallel data in sentence compression](https://md.research.elamur.ai/free/en/papers/filippova-2013-overcoming-lack-parallel-data-sentence/), October 2013. URL https://aclanthology.org/D13-1155. [↗](https://aclanthology.org/D13-1155)
14. Tianyu Gao, Xingcheng Yao, and Danqi Chen. [Simcse: Simple contrastive learning of sentence embeddings](https://md.research.elamur.ai/free/en/papers/gao-2021-simcse-simple-contrastive-learning-sentence/), 2022. URL https://arxiv.org/abs/2104.08821. [↗](https://arxiv.org/abs/2104.08821)
15. Dirk Groeneveld, Iz Beltagy, Pete Walsh, et al. Olmo: Accelerating the science of language models, 2024. URL https://arxiv.org/abs/2402.00838. [↗](https://arxiv.org/abs/2402.00838)
16. Mansi Gupta, Nitish Kulkarni, Raghuveer Chanda, et al. Amazonqa: A review-based question answering task, 2019. URL https://arxiv.org/abs/1908.04364. — [arXiv:1908.04364](https://arxiv.org/abs/1908.04364) [↗](https://arxiv.org/abs/1908.04364)
17. Moonsu Han, Minki Kang, Hyunwoo Jung, et al. Episodic memory reader: Learning what to remember for question answering from streaming data, 2019. URL https://arxiv.org/abs/1903.06164. [↗](https://arxiv.org/abs/1903.06164)
18. Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, et al. [ToxiGen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection](https://md.research.elamur.ai/free/en/papers/hartvigsen-2022-toxigen-large-scale-machine-generated/). In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computationa…
19. Pengcheng He, Jianfeng Gao, and Weizhu Chen. [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://md.research.elamur.ai/free/en/papers/he-2021-debertav3-improving-deberta-using-electra/), March 2023. URL [http://arxiv.org/](http://arxiv.org/) abs/2111.09543. arXiv:2111.09543 [cs]. [↗](http://arxiv.org/)
20. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, et al. Mistral 7b, 2023. URL https://arxiv.org/abs/2310.06825. [↗](https://arxiv.org/abs/2310.06825)
21. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, et al. [Pubmedqa: A dataset for biomedical research question answering](https://md.research.elamur.ai/free/en/papers/jin-2019-pubmedqa-dataset-biomedical-research-question/), 2019. URL https://arxiv.org/abs/1909.06146. [↗](https://arxiv.org/abs/1909.06146)
22. Daniel Khashabi, Amos Ng, Tushar Khot, et al. Gooaq: Open question answering with diverse answer types, 2021. URL https://arxiv.org/abs/2104.08727. [↗](https://arxiv.org/abs/2104.08727)
23. Mahnaz Koupaee and William Yang Wang. Wikihow: A large scale text summarization dataset, 2018. URL https://arxiv.org/abs/1810.09305. [↗](https://arxiv.org/abs/1810.09305)
24. Achintya Kundu, Rhui Dih Lee, Laura Wynter, et al. Enhancing training efficiency using packing with flash attention, 2024. URL https://arxiv.org/abs/2407.09105. [↗](https://arxiv.org/abs/2407.09105)
25. Yoav Levine, Noam Wies, Or Sharir, et al. Limits to Depth Efficiencies of Self-Attention. In Advances in Neural Information Processing Systems, volume 33, pp. 22640–22651. Curran Associates, Inc., 2020. URL [https://papers.nips.cc/paper/2020/hash/ff4dfdf5904e920ce52b48c1cef97829-Abstract.html](https://papers.nips.cc/paper/2020/hash/ff4dfdf5904e920ce52b48c1cef97829-Abstract.html). [↗](https://papers.nips.cc/paper/2020/hash/ff4dfdf5904e920ce52b48c1cef97829-Abstract.html)
26. Xianming Li and Jing Li. [Angle-optimized text embeddings](https://md.research.elamur.ai/free/en/papers/li-2023-angle-optimized-text-embeddings/). arXiv preprint arXiv:2309.12871, 2023.
27. Zehan Li, Xin Zhang, Yanzhao Zhang, et al. [Towards general text embeddings with multi-stage contrastive learning](https://md.research.elamur.ai/free/en/papers/li-2023-towards-general-text-embeddings-multi/), 2023a. URL https://arxiv.org/abs/2308.03281. [↗](https://arxiv.org/abs/2308.03281)
28. Zehan Li, Xin Zhang, Yanzhao Zhang, et al. [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://md.research.elamur.ai/free/en/papers/li-2023-towards-general-text-embeddings-multi/), August 2023b. URL http://arxiv.org/abs/2308.03281. arXiv:2308.03281 [cs]. [↗](http://arxiv.org/abs/2308.03281)
29. Yinhan Liu, Myle Ott, Naman Goyal, et al. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/), 2019. URL https://arxiv.org/abs/1907.11692. [↗](https://arxiv.org/abs/1907.11692)
30. Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, 2017. URL https://arxiv.org/abs/1608.03983. [↗](https://arxiv.org/abs/1608.03983)
31. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL [https://arxiv.org/](https://arxiv.org/) abs/1711.05101. [↗](https://arxiv.org/)
32. Rui Meng, Ye Liu, Shafiq Rayhan Joty, et al. Sfr-embedding-2: Advanced text embedding with multi-stage training, 2024. URL [https://huggingface.co/Salesforce/SFR-Embedding-2%5FR](https://huggingface.co/Salesforce/SFR-Embedding-2%5FR). [↗](https://huggingface.co/Salesforce/SFR-Embedding-2%5FR)
33. John X. Morris and Alexander M. Rush. Contextual document embeddings, 2024. URL https://arxiv.org/abs/2410.02525. [↗](https://arxiv.org/abs/2410.02525)
34. Niklas Muennighoff, Nouamane Tazi, Loïc Magne, et al. [MTEB: Massive Text Embedding Benchmark](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-mteb-massive-text-embedding-benchmark/), March 2023. URL http://arxiv.org/abs/2210.07316. arXiv:2210.07316 [cs]. [↗](http://arxiv.org/abs/2210.07316)
35. Zach Nussbaum, John X. Morris, Brandon Duderstadt, et al. [Nomic embed: Training a reproducible long context text embedder](https://md.research.elamur.ai/free/en/papers/nussbaum-2024-nomic-embed-training-reproducible-long/), 2024. URL https://arxiv.org/abs/2402.01613. [↗](https://arxiv.org/abs/2402.01613)
36. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. [Medmcqa : A large-scale multi-subject multi-choice dataset for medical domain question answering](https://md.research.elamur.ai/free/en/papers/pal-2022-medmcqa-large-scale-multi-subject/), 2022. URL [https://arxiv.org/](https://arxiv.org/) abs/2203.14371. [↗](https://arxiv.org/)
37. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, et al. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only, 2023. URL https://arxiv.org/abs/2306. 01116. [↗](https://arxiv.org/abs/2306)
38. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, et al. Yarn: Efficient context window extension of large language models, 2023. URL https://arxiv.org/abs/2309.00071. [↗](https://arxiv.org/abs/2309.00071)
39. Alec Radford, Karthik Narasimhan, Tim Salimans, et al. Improving language understanding by generative pre-training, 2018. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+et+al%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2C+2018%2E)
40. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, et al. Zero: Memory optimizations toward training trillion parameter models, 2020. URL https://arxiv.org/abs/1910.02054. [↗](https://arxiv.org/abs/1910.02054)
41. Ori Ram, Yoav Levine, Itay Dalmedigos, et al. [In-context retrieval-augmented language models](https://md.research.elamur.ai/free/en/papers/ram-2023-context-retrieval-augmented-language-models/), 2023. URL https://arxiv.org/abs/2302.00083. [↗](https://arxiv.org/abs/2302.00083)
42. Noam Shazeer. Glu variants improve transformer, 2020. URL https://arxiv.org/abs/2002.05202. [↗](https://arxiv.org/abs/2002.05202)
43. Aivin V. Solatorio. Gistembed: Guided in-sample selection of training negatives for text embedding fine-tuning, 2024. URL https://arxiv.org/abs/2402.16829. — [arXiv:2402.16829](https://arxiv.org/abs/2402.16829) [↗](https://arxiv.org/abs/2402.16829)
44. Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, et al. [jina-embeddings-v3: Multilingual embeddings with task lora](https://md.research.elamur.ai/free/en/papers/sturua-2024-jina-embeddings-v3-multilingual-embeddings/), 2024. URL https://arxiv.org/abs/2409.10173. [↗](https://arxiv.org/abs/2409.10173)
45. Jianlin Su, Yu Lu, Shengfeng Pan, et al. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/), 2023. URL https://arxiv.org/abs/2104.09864. [↗](https://arxiv.org/abs/2104.09864)
46. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, et al. [Fever: a large-scale dataset for fact extraction and verification](https://md.research.elamur.ai/free/en/papers/thorne-2018-fever-large-scale-dataset-fact/), 2018. URL https://arxiv.org/abs/1803.05355. [↗](https://arxiv.org/abs/1803.05355)
47. Hugo Touvron, Louis Martin, Kevin Stone, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/), 2023. URL https://arxiv.org/abs/2307.09288. [↗](https://arxiv.org/abs/2307.09288)
48. Nikhil Vyas, Depen Morwani, Rosie Zhao, et al. Soap: Improving and stabilizing shampoo using adam, 2025. URL https://arxiv.org/abs/2409.11321. [↗](https://arxiv.org/abs/2409.11321)
49. Alex Wang, Amanpreet Singh, Julian Michael, et al. [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/), 2019. URL https://arxiv.org/abs/1804.07461. [↗](https://arxiv.org/abs/1804.07461)
50. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, et al. [Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference](https://md.research.elamur.ai/free/en/papers/warner-2024-smarter-better-faster-longer-modern/), 2024. URL https://arxiv.org/abs/2412.13663. [↗](https://arxiv.org/abs/2412.13663)
51. Alexander Wettig, Tianyu Gao, Zexuan Zhong, et al. Should You Mask 15% in Masked Language Modeling?, February 2023. URL http://arxiv.org/abs/2202.08005. arXiv:2202.08005 [cs]. [↗](http://arxiv.org/abs/2202.08005)
52. Adina Williams, Nikita Nangia, and Samuel R. Bowman. [A broad-coverage challenge corpus for sentence understanding through inference](https://md.research.elamur.ai/free/en/papers/williams-2018-broad-coverage-challenge-corpus-sentence/), 2018. URL https://arxiv.org/abs/1704.05426. [↗](https://arxiv.org/abs/1704.05426)
53. Ruibin Xiong, Yunchang Yang, Di He, et al. On layer normalization in the transformer architecture, 2020. URL https://arxiv.org/abs/2002.04745. [↗](https://arxiv.org/abs/2002.04745)
54. Biao Zhang and Rico Sennrich. Root mean square layer normalization, 2019. URL [https://arxiv.org/](https://arxiv.org/) abs/1910.07467. [↗](https://arxiv.org/)
55. Xiang Zhang, Junbo Zhao, and Yann LeCun. [Character-level convolutional networks for text classification](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/), 2016. URL https://arxiv.org/abs/1509.01626. [↗](https://arxiv.org/abs/1509.01626)
56. Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, et al. [mGTE: Generalized long-context text representation and reranking models for multilingual text retrieval](https://md.research.elamur.ai/free/en/papers/zhang-2024-mgte-generalized-long-context-text/). In Franck Dernoncourt, Daniel Preoţiuc-Pietro, and Anastasia Shimorina (eds.), Proceedings of the 2024 Conference on Empirical Me…
57. Yukun Zhu, Ryan Kiros, Richard Zemel, et al. [Aligning books and movies: Towards story-like visual explanations by watching movies and reading books](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/), 2015. URL https://arxiv.org/abs/1506.06724. [↗](https://arxiv.org/abs/1506.06724)

*Updated 2 November 2026*