---
title: "Gecko — paper by Lee et al. (2024) — summary and ideas"
description: "General-purpose text embedding models need large, varied training data to work across many tasks… 49 references, cited by 62 papers here."
authors: ["Jinhyuk Lee", "Zhuyun Dai", "Xiaoqi Ren", "Blair Chen", "Daniel Cer", "Jeremy R. Cole", "Kai Hui", "Michael Boratko", "Rajvi Kapadia", "Wen Ding", "Yi Luan", "Sai Meher Karthik Duddu", "Gustavo Hernandez Abrego", "Weiqiang Shi", "Nithi Gupta", "Aditya Kusupati", "Prateek Jain", "Siddhartha Reddy Jonnalagadda", "Ming-Wei Chang", "Iftekhar Naim"]
year: 2024
venue: "arXiv:2403.20327"
source: "https://arxiv.org/abs/2403.20327"
doi: "10.48550/arxiv.2403.20327"
arxiv: "2403.20327"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lee-2024-gecko-versatile-text-embeddings-distilled/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lee-2024-gecko-versatile-text-embeddings-distilled/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Gecko: Versatile Text Embeddings Distilled from Large Language Models

# Gecko: Versatile Text Embeddings Distilled from Large Language Models

**Authors:** [Jinhyuk Lee](https://md.research.elamur.ai/free/en/authors/jinhyuk-lee/), [Zhuyun Dai](https://md.research.elamur.ai/free/en/authors/zhuyun-dai/), Xiaoqi Ren, Blair Chen, [Daniel Cer](https://md.research.elamur.ai/free/en/authors/daniel-cer/), [Jeremy R. Cole](https://md.research.elamur.ai/free/en/authors/jeremy-cole/), [Kai Hui](https://md.research.elamur.ai/free/en/authors/kai-hui/), Michael Boratko, Rajvi Kapadia, Wen Ding, [Yi Luan](https://md.research.elamur.ai/free/en/authors/yi-luan/), Sai Meher Karthik Duddu, [Gustavo Hernandez Abrego](https://md.research.elamur.ai/free/en/authors/gustavo-hernandez-abrego/), Weiqiang Shi, Nithi Gupta, Aditya Kusupati, Prateek Jain, Siddhartha Reddy Jonnalagadda, [Ming-Wei Chang](https://md.research.elamur.ai/free/en/authors/ming-wei-chang/) and Iftekhar Naim · **Year:** 2024 · **Venue:** arXiv:2403.20327 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2403.20327) · [PDF](https://arxiv.org/pdf/2403.20327) · [DOI 10.48550/arxiv.2403.20327](https://doi.org/10.48550/arxiv.2403.20327) · [arXiv 2403.20327](https://arxiv.org/abs/2403.20327) · [OpenAlex](https://openalex.org/W4393399864)

**Citations:** 10 citations (OpenAlex, 2026-10-14) · [62](https://md.research.elamur.ai/free/en/papers/lee-2024-gecko-versatile-text-embeddings-distilled/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
General-purpose text embedding models need large, varied training data to work across many tasks, and human-labeled collections are costly and limited in coverage. Gecko is a compact embedding model, about 1.2B parameters, that draws its training signal from a large language model. It is trained in two stages. First, pre-finetuning uses contrastive learning on community QA pairs and title-body pairs crawled from the web. Second, fine-tuning uses FRet, a synthetic dataset built in two LLM steps. In the first step, an LLM reads a random web passage and writes a task description plus a matching query. In the second step, a pretrained embedder retrieves neighboring passages for that query, and the same LLM reranks them to choose a positive and a hard negative. The chosen positive often turns out to be a passage other than the source passage. FRet is combined with academic datasets in a shared format. On MTEB, Gecko matches or beats much larger or higher-dimensional models, so strong embeddings are possible at modest cost.

## Key points
- On MTEB, Gecko with 768-dimensional embeddings averages 66.31. This rivals models about 7x larger and embeddings about 5x wider. The 256-dimensional version beats every existing 768-dimensional entry.
- FRet (Few-shot Prompted Retrieval) spans four kinds of tasks: answering questions, retrieving search results, verifying facts, and judging similarity between sentences. Trained on FRet alone, Gecko is already a strong zero-shot baseline.
- In ablations, the passage the LLM ranks highest always made a better positive than the original seed passage.
- On diversity, 300k examples sampled uniformly across all four tasks (75k each) generalized better than single-task subsets. The unified task-prefixed formatting also mattered.
- Two additions improved results without hurting other tasks much:
  - NLI data with a symmetric format and same-tower negatives raised STS by 1.6 points on average.
  - Merged classification datasets clearly improved classification.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 523 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works

## Cite

Cite as: Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R. Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, Yi Luan, Sai Meher Karthik Duddu, Gustavo Hernandez Abrego, Weiqiang Shi, Nithi Gupta, Aditya Kusupati, Prateek Jain, Siddhartha Reddy Jonnalagadda, Ming-Wei Chang and Iftekhar Naim. 2024. Gecko: Versatile Text Embeddings Distilled from Large Language Models. arXiv:2403.20327.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{lee2024gecko,
  title = {{Gecko: Versatile Text Embeddings Distilled from Large Language Models}},
  author = {Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R. Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, Yi Luan, Sai Meher Karthik Duddu, Gustavo Hernandez Abrego, Weiqiang Shi, Nithi Gupta, Aditya Kusupati, Prateek Jain, Siddhartha Reddy Jonnalagadda, Ming-Wei Chang and Iftekhar Naim},
  year = {2024},
  doi = {10.48550/arxiv.2403.20327},
  eprint = {2403.20327},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.20327}
}
```

- [← previous Agent Workflow Memory](https://md.research.elamur.ai/free/en/papers/wang-2024-agent-workflow-memory/)
- [next → Self-Preference Bias in LLM-as-a-Judge](https://md.research.elamur.ai/free/en/papers/wataoka-2024-self-preference-bias-llm-judge/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

49 references, 35 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, et al. [Palm 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/). arXiv preprint arXiv:2305.10403, 2023.
2. A. Asai, T. Schick, P. Lewis, X. Chen, G. Izacard, S. Riedel, H. Hajishirzi, and W.-t. Yih. [Task-aware retrieval with instructions](https://md.research.elamur.ai/free/en/papers/asai-2023-task-aware-retrieval-instructions/). arXiv preprint arXiv:2211.09260, 2022.
3. L. Bonifacio, H. Abonizio, M. Fadaee, and R. Nogueira. Inpars: Data augmentation for information retrieval using large language models. arXiv preprint arXiv:2202.05144, 2022. [find](https://scholar.google.com/scholar?q=L%2E+Bonifacio%2C+H%2E+Abonizio%2C+M%2E+Fadaee%2C+and+R%2E+Nogueira%2E+Inpars%3A+Data+augmentation+for+information+retrieval+using+large+language+models%2E+arXiv+preprint+arXiv%3A2202%2E05144%2C+2022%2E)
4. S. Bowman, G. Angeli, C. Potts, and C. D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, 2015. [find](https://scholar.google.com/scholar?q=S%2E+Bowman%2C+G%2E+Angeli%2C+C%2E+Potts%2C+and+C%2E+D%2E+Manning%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Natural+Language+Proc)
5. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Ber… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
6. D. Cer, Y. Yang, S.-y. Kong, N. Hua, N. Limtiaco, R. S. John, N. Constant, M. Guajardo-Cespedes, S. Yuan, C. Tar, et al. [Universal sentence encoder for english](https://md.research.elamur.ai/free/en/papers/cer-2018-universal-sentence-encoder-english/). In Proceedings of the 2018 conference on empirical methods in natural language processing: system demonstrations, pages 169–174, 2018.
7. G. V. Cormack, C. L. Clarke, and S. Buettcher. [Reciprocal rank fusion outperforms condorcet and individual rank learning methods](https://md.research.elamur.ai/free/en/papers/cormack-2009-reciprocal-rank-fusion-outperforms/). In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pages 758–759, 2009.
8. Z. Dai, V. Y. Zhao, J. Ma, Y. Luan, J. Ni, J. Lu, A. Bakalov, K. Guu, K. B. Hall, and M.-W. Chang. [Promptagator: Few-shot dense retrieval from 8 examples](https://md.research.elamur.ai/free/en/papers/dai-2022-promptagator-few-shot-dense-retrieval/). arXiv preprint arXiv:2209.11755, 2022.
9. A. Drozdov, H. Zhuang, Z. Dai, Z. Qin, R. Rahimi, X. Wang, D. Alon, M. Iyyer, A. McCallum, D. Metzler, and K. Hui. PaRaDe: Passage ranking using demonstrations with LLMs. In H. Bouamor, J. Pino, and K. Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14242–… [find](https://scholar.google.com/scholar?q=A%2E+Drozdov%2C+H%2E+Zhuang%2C+Z%2E+Dai%2C+Z%2E+Qin%2C+R%2E+Rahimi%2C+X%2E+Wang%2C+D%2E+Alon%2C+M%2E+Iyyer%2C+A%2E+McCallum%2C+D%2E+Metzler%2C+and+K%2E+Hui%2E+PaRaDe%3A+Passage+ranking+using+demonstrations+with+LLMs%2E+In+H%2E+Bouamor%2C+J%2E+Pino%2C+and+K)
10. T. Gao, X. Yao, and D. Chen. [Simcse: Simple contrastive learning of sentence embeddings](https://md.research.elamur.ai/free/en/papers/gao-2021-simcse-simple-contrastive-learning-sentence/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, 2021.
11. G. Izacard and E. Grave. Distilling knowledge from reader to retriever for question answering. In International Conference on Learning Representations, 2021. URL [https://openreview.net/](https://openreview.net/) forum?id=NTEz-6wysdb . [↗](https://openreview.net/)
12. G. Izacard, M. Caron, L. Hosseini, S. Riedel, P. Bojanowski, A. Joulin, and E. Grave. [Unsupervised dense information retrieval with contrastive learning](https://md.research.elamur.ai/free/en/papers/izacard-2021-unsupervised-dense-information-retrieval-contrastive/). Transactions on Machine Learning Research, 2022.
13. V. Jeronymo, L. Bonifacio, H. Abonizio, M. Fadaee, R. Lotufo, J. Zavrel, and R. Nogueira. Inpars-v2: Large language models as efficient dataset generators for information retrieval. arXiv preprint arXiv:2301.01820, 2023. — [arXiv:2301.01820](https://arxiv.org/abs/2301.01820) [find](https://scholar.google.com/scholar?q=V%2E+Jeronymo%2C+L%2E+Bonifacio%2C+H%2E+Abonizio%2C+M%2E+Fadaee%2C+R%2E+Lotufo%2C+J%2E+Zavrel%2C+and+R%2E+Nogueira%2E+Inpars%2Dv2%3A+Large+language+models+as+efficient+dataset+generators+for+information+retrieval%2E+arXiv+preprint+arX)
14. V. Karpukhin, B. Oğuz, S. Min, P. Lewis, L. Y. Wu, S. Edunov, D. Chen, and W. tau Yih. [Dense passage retrieval for open-domain question answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/). ArXiv, abs/2004.04906, 2020.
15. E. Khramtsova, S. Zhuang, M. Baktashmotlagh, and G. Zuccon. Leveraging llms for unsupervised dense retriever ranking. arXiv preprint arXiv:2402.04853, 2024. — [arXiv:2402.04853](https://arxiv.org/abs/2402.04853) [find](https://scholar.google.com/scholar?q=E%2E+Khramtsova%2C+S%2E+Zhuang%2C+M%2E+Baktashmotlagh%2C+and+G%2E+Zuccon%2E+Leveraging+llms+for+unsupervised+dense+retriever+ranking%2E+arXiv+preprint+arXiv%3A2402%2E04853%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E04853%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
16. A. Kusupati, G. Bhatt, A. Rege, M. Wallingford, A. Sinha, V. Ramanujan, W. Howard-Snyder, K. Chen, S. Kakade, P. Jain, et al. Matryoshka representation learning. Advances in Neural Information Processing Systems, 35:30233–30249, 2022. [find](https://scholar.google.com/scholar?q=A%2E+Kusupati%2C+G%2E+Bhatt%2C+A%2E+Rege%2C+M%2E+Wallingford%2C+A%2E+Sinha%2C+V%2E+Ramanujan%2C+W%2E+Howard%2DSnyder%2C+K%2E+Chen%2C+S%2E+Kakade%2C+P%2E+Jain%2C+et+al%2E+Matryoshka+representation+learning%2E+Advances+in+Neural+Information+Process)
17. T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, et al. [Natural questions: a benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). Transactions of the Association for Computational Linguistics, 7:453–466, 2019.
18. Q. Le and T. Mikolov. [Distributed representations of sentences and documents](https://md.research.elamur.ai/free/en/papers/le-2014-distributed-representations-sentences-documents/). In International conference on machine learning, pages 1188–1196. PMLR, 2014.
19. J. Lee, M. Sung, J. Kang, and D. Chen. [Learning dense representations of phrases at scale](https://md.research.elamur.ai/free/en/papers/lee-2021-learning-dense-representations-phrases-scale/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6634–6647, 2…
20. K. Lee, M.-W. Chang, and K. Toutanova. [Latent retrieval for weakly supervised open domain question answering](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086–6096, Florence, Italy, July 2019. Association for Computational Linguistics.
21. Z. Li, X. Zhang, Y. Zhang, D. Long, P. Xie, and M. Zhang. [Towards general text embeddings with multi-stage contrastive learning](https://md.research.elamur.ai/free/en/papers/li-2023-towards-general-text-embeddings-multi/). arXiv preprint arXiv:2308.03281, 2023.
22. X. Ma, L. Wang, N. Yang, F. Wei, and J. Lin. [Fine-tuning llama for multi-stage text retrieval](https://md.research.elamur.ai/free/en/papers/ma-2023-fine-tuning-llama-multi-stage/). arXiv preprint arXiv:2310.08319, 2023.
23. F. Moiseev, G. H. Abrego, P. Dornbach, I. Zitouni, E. Alfonseca, and Z. Dong. Samtone: Improving contrastive loss for dual encoder retrieval models with same tower negatives. arXiv preprint arXiv:2306.02516, 2023. [find](https://scholar.google.com/scholar?q=F%2E+Moiseev%2C+G%2E+H%2E+Abrego%2C+P%2E+Dornbach%2C+I%2E+Zitouni%2C+E%2E+Alfonseca%2C+and+Z%2E+Dong%2E+Samtone%3A+Improving+contrastive+loss+for+dual+encoder+retrieval+models+with+same+tower+negatives%2E+arXiv+preprint+arXiv%3A2306)
24. N. Muennighoff, N. Tazi, L. Magne, and N. Reimers. [Mteb: Massive text embedding benchmark](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-mteb-massive-text-embedding-benchmark/). In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2006–2029, 2023.
25. N. Muennighoff, H. Su, L. Wang, N. Yang, F. Wei, T. Yu, A. Singh, and D. Kiela. [Generative representational instruction tuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2024-generative-representational-instruction-tuning/). arXiv preprint arXiv:2402.09906, 2024.
26. A. Neelakantan, T. Xu, R. Puri, A. Radford, J. M. Han, J. Tworek, Q. Yuan, N. Tezak, J. W. Kim, C. Hallacy, et al. [Text and code embeddings by contrastive pre-training](https://md.research.elamur.ai/free/en/papers/neelakantan-2022-text-code-embeddings-contrastive-pre/). arXiv preprint arXiv:2201.10005, 2022.
27. J. Ni, C. Qu, J. Lu, Z. Dai, G. H. ’Abrego, J. Ma, V. Zhao, Y. Luan, K. B. Hall, M.-W. Chang, and Y. Yang. [Large dual encoders are generalizable retrievers](https://md.research.elamur.ai/free/en/papers/ni-2021-large-dual-encoders-generalizable-retrievers/). In Conference on Empirical Methods in Natural Language Processing, 2021.
28. J. Ni, G. H. Abrego, N. Constant, J. Ma, K. Hall, D. Cer, and Y. Yang. [Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models](https://md.research.elamur.ai/free/en/papers/ni-2021-sentence-t5-scalable-sentence-encoders/). In Findings of the Association for Computational Linguistics: ACL 2022, pages 1864–1874, 2022.
29. A. Pal, L. K. Umapathi, and M. Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pages 248–260. PMLR, 2022. — [MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering](https://md.research.elamur.ai/free/en/papers/pal-2022-medmcqa-large-scale-multi-subject/)
30. Y. Qu, Y. Ding, J. Liu, K. Liu, R. Ren, W. X. Zhao, D. Dong, H. Wu, and H. Wang. [Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering](https://md.research.elamur.ai/free/en/papers/qu-2021-rocketqa-optimized-training-approach-dense/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistic…
31. N. Reimers and I. Gurevych. [Sentence-bert: Sentence embeddings using siamese bert-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, 2019.
32. R. Ren, Y. Qu, J. Liu, W. X. Zhao, Q. She, H. Wu, H. Wang, and J.-R. Wen. [RocketQAv2: A joint training method for dense passage retrieval and passage re-ranking](https://md.research.elamur.ai/free/en/papers/ren-2021-rocketqav2-joint-training-method-dense/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2825–2835, Online and Punta Cana, Dominic…
33. D. Sachan, M. Lewis, M. Joshi, A. Aghajanyan, W.-t. Yih, J. Pineau, and L. Zettlemoyer. [Improving passage retrieval with zero-shot question generation](https://md.research.elamur.ai/free/en/papers/sachan-2022-improving-passage-retrieval-zero-shot/). In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 37…
34. D. S. Sachan, M. Lewis, D. Yogatama, L. Zettlemoyer, J. Pineau, and M. Zaheer. Questions are all you need to train a dense passage retriever. Transactions of the Association for Computational Linguistics, 11:600–616, 2023. [find](https://scholar.google.com/scholar?q=D%2E+S%2E+Sachan%2C+M%2E+Lewis%2C+D%2E+Yogatama%2C+L%2E+Zettlemoyer%2C+J%2E+Pineau%2C+and+M%2E+Zaheer%2E+Questions+are+all+you+need+to+train+a+dense+passage+retriever%2E+Transactions+of+the+Association+for+Computational+Linguist)
35. K. Santhanam, O. Khattab, J. Saad-Falcon, C. Potts, and M. Zaharia. [Colbertv2: Effective and efficient retrieval via lightweight late interaction](https://md.research.elamur.ai/free/en/papers/santhanam-2022-colbertv2-effective-efficient-retrieval-lightweight/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages…
36. J. M. Springer, S. Kotha, D. Fried, G. Neubig, and A. Raghunathan. Repetition improves language model embeddings. arXiv preprint arXiv:2402.15449, 2024. [find](https://scholar.google.com/scholar?q=J%2E+M%2E+Springer%2C+S%2E+Kotha%2C+D%2E+Fried%2C+G%2E+Neubig%2C+and+A%2E+Raghunathan%2E+Repetition+improves+language+model+embeddings%2E+arXiv+preprint+arXiv%3A2402%2E15449%2C+2024%2E)
37. H. Su, W. Shi, J. Kasai, Y. Wang, Y. Hu, M. Ostendorf, W.-t. Yih, N. A. Smith, L. Zettlemoyer, and T. Yu. [One embedder, any task: Instruction-finetuned text embeddings](https://md.research.elamur.ai/free/en/papers/su-2022-one-embedder-any-task-instruction/). arXiv preprint arXiv:2212.09741, 2022.
38. G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
39. N. Thakur, N. Reimers, A. Rücklé, A. Srivastava, and I. Gurevych. Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. [find](https://scholar.google.com/scholar?q=N%2E+Thakur%2C+N%2E+Reimers%2C+A%2E+R%C3%BCckl%C3%A9%2C+A%2E+Srivastava%2C+and+I%2E+Gurevych%2E+Beir%3A+A+heterogeneous+benchmark+for+zero%2Dshot+evaluation+of+information+retrieval+models%2E+In+Thirty%2Dfifth+Conference+on+Neural+Informa)
40. J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal. [Fever: a large-scale dataset for fact extraction and verification](https://md.research.elamur.ai/free/en/papers/thorne-2018-fever-large-scale-dataset-fact/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), p…
41. L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, D. Jiang, R. Majumder, and F. Wei. [Text embeddings by weakly-supervised contrastive pre-training](https://md.research.elamur.ai/free/en/papers/wang-2022-text-embeddings-weakly-supervised-contrastive/). arXiv preprint arXiv:2212.03533, 2022.
42. L. Wang, N. Yang, X. Huang, L. Yang, R. Majumder, and F. Wei. [Improving text embeddings with large language models](https://md.research.elamur.ai/free/en/papers/wang-2024-improving-text-embeddings-large-language/). arXiv preprint arXiv:2401.00368, 2023.
43. A. Williams, N. Nangia, and S. R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of NAACL-HLT, pages 1112–1122, 2018. [find](https://scholar.google.com/scholar?q=A%2E+Williams%2C+N%2E+Nangia%2C+and+S%2E+R%2E+Bowman%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+NAACL%2DHLT%2C+pages+1112%E2%80%931122%2C+2018%2E)
44. L. Xiong, C. Xiong, Y. Li, K.-F. Tang, J. Liu, P. Bennett, J. Ahmed, and A. Overwijk. [Approximate nearest neighbor negative contrastive learning for dense text retrieval](https://md.research.elamur.ai/free/en/papers/xiong-2020-approximate-nearest-neighbor-negative-contrastive/). arXiv preprint arXiv:2007.00808, 2020.
45. L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel. [mt5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol…
46. Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning. [Hotpotqa: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, 2018.
47. X. Zhang, N. Thakur, O. Ogundepo, E. Kamalloo, D. Alfonso-Hermelo, X. Li, Q. Liu, M. Rezagholizadeh, and J. Lin. [Miracl: A multilingual retrieval dataset covering 18 diverse languages](https://md.research.elamur.ai/free/en/papers/zhang-2023-miracl-multilingual-retrieval-dataset-covering/). Transactions of the Association for Computational Linguistics, 11:1114–1131, 2023.
48. H. Zhuang, Z. Qin, K. Hui, J. Wu, L. Yan, X. Wang, and M. Berdersky. Beyond yes and no: Improving zero-shot llm rankers via scoring fine-grained relevance labels. arXiv preprint arXiv:2310.14122, 2023. [find](https://scholar.google.com/scholar?q=H%2E+Zhuang%2C+Z%2E+Qin%2C+K%2E+Hui%2C+J%2E+Wu%2C+L%2E+Yan%2C+X%2E+Wang%2C+and+M%2E+Berdersky%2E+Beyond+yes+and+no%3A+Improving+zero%2Dshot+llm+rankers+via+scoring+fine%2Dgrained+relevance+labels%2E+arXiv+preprint+arXiv%3A2310%2E14122%2C+2023)
49. Author Contributions Jinhyuk Lee: Co-lead of FRet and Gecko. Coordinated the project, implemented the main functionality of FRet and Gecko, and led the paper writing. Zhuyun Dai: Co-lead of FRet. Implemented the main functionality of FRet and led the paper writing. Xiaoqi Ren: Co-lead of Gecko. Impl… [find](https://scholar.google.com/scholar?q=Author+Contributions+Jinhyuk+Lee%3A+Co%2Dlead+of+FRet+and+Gecko%2E+Coordinated+the+project%2C+implemented+the+main+functionality+of+FRet+and+Gecko%2C+and+led+the+paper+writing%2E+Zhuyun+Dai%3A+Co%2Dlead+of+FRet%2E+Impl)

*Updated 24 October 2026*