---
title: "Jina Embeddings 2 — paper by Gunther et al. (2023)"
description: "The Jina AI team presents Jina Embeddings v2, an English embedding suite that accepts inputs of up… 36 references, cited by 62 papers here."
authors: ["Michael Gunther", "Jackmin Ong", "Isabelle Mohr", "Alaeddine Abdessalem", "Tanguy Abel", "Mohammad Kalim Akram", "Susana Guzman", "Georgios Mastrapas", "Saba Sturua", "Bo Wang", "Maximilian Werk", "Nan Wang", "Han Xiao"]
year: 2023
venue: "arXiv:2310.19923"
source: "https://arxiv.org/abs/2310.19923"
doi: "10.48550/arxiv.2310.19923"
arxiv: "2310.19923"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gunther-2023-jina-embeddings-2-8192-token/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gunther-2023-jina-embeddings-2-8192-token/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents

# Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents

**Authors:** Michael Gunther, Jackmin Ong, Isabelle Mohr, Alaeddine Abdessalem, Tanguy Abel, Mohammad Kalim Akram, Susana Guzman, Georgios Mastrapas, [Saba Sturua](https://md.research.elamur.ai/free/en/authors/saba-sturua/), [Bo Wang](https://md.research.elamur.ai/free/en/authors/bo-wang/), Maximilian Werk, Nan Wang and Han Xiao · **Year:** 2023 · **Venue:** arXiv:2310.19923 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2310.19923) · [PDF](https://arxiv.org/pdf/2310.19923) · [DOI 10.48550/arxiv.2310.19923](https://doi.org/10.48550/arxiv.2310.19923) · [arXiv 2310.19923](https://arxiv.org/abs/2310.19923) · [OpenAlex](https://openalex.org/W4388183278)

**Citations:** 13 citations (OpenAlex, 2026-10-14) · [62](https://md.research.elamur.ai/free/en/papers/gunther-2023-jina-embeddings-2-8192-token/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Most open-source text embedding models inherit BERT's 512-token input limit. Long documents must therefore be truncated or split into chunks, which fragments their meaning and multiplies the number of vectors to store and search. The Jina AI team presents Jina Embeddings v2, an English embedding suite that accepts inputs of up to 8192 tokens. Its backbone, Jina BERT, replaces learned position embeddings with a symmetric, bidirectional variant of ALiBi attention biases. It also uses GLU-style feedforward layers and post-layer normalization. The backbone is pre-trained from scratch with masked language modeling on C4. Two contrastive fine-tuning stages then turn it into an embedding model: the first uses text pairs, the second adds hard negatives. Fine-tuning only saw texts of up to 512 tokens, yet the models still work on much longer inputs. On MTEB, the base model performs close to OpenAI's proprietary text-embedding-ada-002 and to e5-base-v2. Longer context clearly improves results on tasks such as NarrativeQA. The work gives practitioners an open, compact option for embedding whole documents.

## Key points
- The architecture defines three sizes: Jina BERT Small (4 layers, 33M parameters), Base (12 layers, 137M) and Large (24 layers, 455M). Small and Base use GEGLU, while Large uses ReGLU because GEGLU training was unstable. The released embedding models are jina-embeddings-v2-small-en and jina-embeddings-v2-base-en on Hugging Face.
- Pre-training used English C4, about 365M web documents and roughly 170B tokens. The objective was MLM with 30% whole-word masking and no NSP task.
- Average MTEB scores were 60.37 for jina-base-v2 and 58.12 for jina-small-v2. For comparison, text-embedding-ada-002 scored 60.99, e5-base-v2 scored 61.50 and all-mpnet-base-v2 scored 57.78.
- The authors released new long-document evaluation datasets, including NarrativeQA-, BigPatent- and WikiCities-based tasks, and also tested on SciFact. Longer sequence lengths helped retrieval and BigPatent clustering but slightly hurt WikiCities clustering.
- The maximum input length is 16 times that of leading open-source embedding models, which are usually capped at 512 tokens.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

Also on the byline: [Jina AI](https://ror.org/02b1cd072).

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 523 works

## Cite

Cite as: Michael Gunther, Jackmin Ong, Isabelle Mohr, Alaeddine Abdessalem, Tanguy Abel, Mohammad Kalim Akram, Susana Guzman, Georgios Mastrapas, Saba Sturua, Bo Wang, Maximilian Werk, Nan Wang and Han Xiao. 2023. Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents. arXiv:2310.19923.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{gunther2023jina,
  title = {{Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents}},
  author = {Michael Gunther, Jackmin Ong, Isabelle Mohr, Alaeddine Abdessalem, Tanguy Abel, Mohammad Kalim Akram, Susana Guzman, Georgios Mastrapas, Saba Sturua, Bo Wang, Maximilian Werk, Nan Wang and Han Xiao},
  year = {2023},
  doi = {10.48550/arxiv.2310.19923},
  eprint = {2310.19923},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.19923}
}
```

- [← previous Dipping PLMs Sauce: Bridging Structure and Text for Effective Knowledge Graph Completion via Conditional Soft Prompting](https://md.research.elamur.ai/free/en/papers/chen-2023-dipping-plms-sauce-bridging-structure/)
- [next → SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling](https://md.research.elamur.ai/free/en/papers/kim-2023-solar-10-7b-scaling-large/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

36 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Nils Reimers and Iryna Gurevych. [Sentence-bert: Sentence embeddings using siamese bert-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982– 3992, 2019.
2. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. [Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models](https://md.research.elamur.ai/free/en/papers/thakur-2021-beir-heterogenous-benchmark-zero/), 2021.
3. Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. [Mteb: Massive text embedding benchmark](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-mteb-massive-text-embedding-benchmark/), 2023.
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/), 2019.
5. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019a. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2C+2019a%2E)
6. Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation, 2022. [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Noah+A%2E+Smith%2C+and+Mike+Lewis%2E+Train+short%2C+test+long%3A+Attention+with+linear+biases+enables+input+length+extrapolation%2C+2022%2E)
7. Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. [Language modeling with gated convolutional networks](https://md.research.elamur.ai/free/en/papers/dauphin-2016-language-modeling-gated-convolutional-networks/). CoRR, abs/1612.08083, 2016. URL http://arxiv.org/abs/1612.08083. [↗](http://arxiv.org/abs/1612.08083)
8. Michael Günther, Louis Milliken, Jonathan Geuter, Georgios Mastrapas, Bo Wang, and Han Xiao. Jina embeddings: A novel set of high-performance sentence embedding models. arXiv preprint arXiv:2307.11224, 2023. [find](https://scholar.google.com/scholar?q=Michael+G%C3%BCnther%2C+Louis+Milliken%2C+Jonathan+Geuter%2C+Georgios+Mastrapas%2C+Bo+Wang%2C+and+Han+Xiao%2E+Jina+embeddings%3A+A+novel+set+of+high%2Dperformance+sentence+embedding+models%2E+arXiv+preprint+arXiv%3A2307%2E11224)
9. Noam Shazeer. GLU variants improve transformer. CoRR, abs/2002.05202, 2020. URL https://arxiv.org/abs/2002.05202. [↗](https://arxiv.org/abs/2002.05202)
10. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. [Text embeddings by weakly-supervised contrastive pre-training](https://md.research.elamur.ai/free/en/papers/wang-2022-text-embeddings-weakly-supervised-contrastive/). arXiv preprint arXiv:2212.03533, 2022.
11. Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6):391–407, 1990. [find](https://scholar.google.com/scholar?q=Scott+Deerwester%2C+Susan+T%2E+Dumais%2C+George+W%2E+Furnas%2C+Thomas+K%2E+Landauer%2C+and+Richard+Harshman%2E+Indexing+by+latent+semantic+analysis%2E+Journal+of+the+American+Society+for+Information+Science%2C+41%286%29%3A391%E2%80%93)
12. David Blei, Andrew Ng, and Michael Jordan. [Latent dirichlet allocation](https://md.research.elamur.ai/free/en/papers/blei-2003-latent-dirichlet-allocation/). In T. Dietterich, S. Becker, and Z. Ghahramani, editors, Advances in Neural Information Processing Systems, volume 14. MIT Press, 2001. URL [https://proceedings.neurips.cc/paper_files/paper/2001/file/](https://proceedings.neurips.cc/paper_files/paper/2001/file/) 296472c9542ad4d4788d5435081… [↗](https://proceedings.neurips.cc/paper_files/paper/2001/file/)
13. Tianyu Gao, Xingcheng Yao, and Danqi Chen. [Simcse: Simple contrastive learning of sentence embeddings](https://md.research.elamur.ai/free/en/papers/gao-2021-simcse-simple-contrastive-learning-sentence/), 2022.
14. Luyu Gao and Jamie Callan. [Condenser: a pre-training architecture for dense retrieval](https://md.research.elamur.ai/free/en/papers/gao-2021-condenser-pre-training-architecture-dense/), 2021.
15. Shitao Xiao, Zheng Liu, Yingxia Shao, and Zhao Cao. [Retromae: Pre-training retrieval-oriented language models via masked auto-encoder](https://md.research.elamur.ai/free/en/papers/xiao-2022-retromae-pre-training-retrieval-oriented/), 2022.
16. Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. [Towards general text embeddings with multi-stage contrastive learning](https://md.research.elamur.ai/free/en/papers/li-2023-towards-general-text-embeddings-multi/), 2023.
17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). CoRR, abs/1706.03762, 2017. URL http://arxiv.org/abs/1706.03762. [↗](http://arxiv.org/abs/1706.03762)
18. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. CoRR, abs/1909.08053, 2019. URL [http://arxiv.org/](http://arxiv.org/) abs/1909.08053. [↗](http://arxiv.org/)
19. Toan Q. Nguyen and Julian Salazar. Transformers without tears: Improving the normalization of self-attention. CoRR, abs/1910.05895, 2019. URL http://arxiv.org/abs/1910.05895. [↗](http://arxiv.org/abs/1910.05895)
20. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). The Journal of Machine Learning Research, 21(1): 5485–5551, 2020.
21. Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. CoRR, abs/1711.05101, 2017. URL http://arxiv.org/abs/1711.05101. [↗](http://arxiv.org/abs/1711.05101)
22. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. In International Conference on Learning Representations, 2018. [find](https://scholar.google.com/scholar?q=Paulius+Micikevicius%2C+Sharan+Narang%2C+Jonah+Alben%2C+Gregory+Diamos%2C+Erich+Elsen%2C+David+Garcia%2C+Boris+Ginsburg%2C+Michael+Houston%2C+Oleksii+Kuchaiev%2C+Ganesh+Venkatesh%2C+et+al%2E+Mixed+precision+training%2E+In+In)
23. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3506, 2020. [find](https://scholar.google.com/scholar?q=Jeff+Rasley%2C+Samyam+Rajbhandari%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+Deepspeed%3A+System+optimizations+enable+training+deep+learning+models+with+over+100+billion+parameters%2E+In+Proceedings+of+the+26th+ACM+)
24. Zhuyun Dai, Vincent Y Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith Hall, and Ming-Wei Chang. [Promptagator: Few-shot dense retrieval from 8 examples](https://md.research.elamur.ai/free/en/papers/dai-2022-promptagator-few-shot-dense-retrieval/). In The Eleventh International Conference on Learning Representations, 2023. URL [https://openreview.net/forum?id=](https://openreview.net/forum?id=) gmL46YMp… [↗](https://openreview.net/forum?id=)
25. Aäron van den Oord, Yazhe Li, and Oriol Vinyals. [Representation learning with contrastive predictive coding](https://md.research.elamur.ai/free/en/papers/oord-2018-representation-learning-contrastive-predictive-coding/). CoRR, abs/1807.03748, 2018. URL http://arxiv.org/abs/1807.03748. [↗](http://arxiv.org/abs/1807.03748)
26. Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2495–2504. [find](https://scholar.google.com/scholar?q=Feng+Wang+and+Huaping+Liu%2E+Understanding+the+behaviour+of+contrastive+loss%2E+In+2021+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C+pages+2495%E2%80%932504%2E)
27. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. [Ms marco: A human generated machine reading comprehension dataset](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/). arXiv preprint arXiv:1611.09268, 2016.
28. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questi… — [Natural Questions: A Benchmark for Question Answering Research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
29. Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, 2015. [find](https://scholar.google.com/scholar?q=Samuel+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D+Manning%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+)
30. Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174, 2016. [find](https://scholar.google.com/scholar?q=Tianqi+Chen%2C+Bing+Xu%2C+Chiyuan+Zhang%2C+and+Carlos+Guestrin%2E+Training+deep+nets+with+sublinear+memory+cost%2E+arXiv+preprint+arXiv%3A1604%2E06174%2C+2016%2E)
31. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692, 2019b. URL http://arxiv.org/abs/1907.11692. [↗](http://arxiv.org/abs/1907.11692)
32. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). CoRR, abs/1804.07461, 2018. URL http://arxiv.org/abs/1804.07461. [↗](http://arxiv.org/abs/1804.07461)
33. Jason Phang, Thibault Févry, and Samuel R. Bowman. [Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks](https://md.research.elamur.ai/free/en/papers/phang-2018-sentence-encoders-stilts-supplementary-training/). CoRR, abs/1811.01088, 2018. URL [http://arxiv.org/](http://arxiv.org/) abs/1811.01088. [↗](http://arxiv.org/)
34. Eva Sharma, Chen Li, and Lu Wang. [BIGPATENT: A large-scale dataset for abstractive and coherent summarization](https://md.research.elamur.ai/free/en/papers/sharma-2019-bigpatent-large-scale-dataset-abstractive/). CoRR, abs/1906.03741, 2019. URL http://arxiv.org/abs/1906.03741. [↗](http://arxiv.org/abs/1906.03741)
35. Wikimedia Foundation. Wikimedia downloads, 2022. URL [https://dumps.wikimedia.org](https://dumps.wikimedia.org). [↗](https://dumps.wikimedia.org)
36. David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. [Fact or fiction: Verifying scientific claims](https://md.research.elamur.ai/free/en/papers/wadden-2020-fact-or-fiction-verifying-scientific/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534–7550, 2020.

*Updated 24 October 2026*