---
title: "Text and Code Embeddings by Contrastive — Neelakantan 2022"
description: "Text embeddings power semantic search, classification and similarity scoring."
authors: ["Arvind Neelakantan", "Tao Xu", "Raul Puri", "Alec Radford", "Jesse Michael Han", "Jerry Tworek", "Qiming Yuan", "Nikolas Tezak", "Jong Wook Kim", "Chris Hallacy", "Johannes Heidecke", "Pranav Shyam", "Boris Power", "Tyna Eloundou Nekoul", "Girish Sastry", "Gretchen Krueger", "David Schnurr", "Felipe Petroski Such", "Kenny Hsu", "Madeleine Thompson", "Tabarak Khan", "Toki Sherbakov", "Joanne Jang", "Peter Welinder", "Lilian Weng"]
year: 2022
venue: "arXiv:2201.10005"
source: "https://arxiv.org/abs/2201.10005"
doi: "10.48550/arxiv.2201.10005"
arxiv: "2201.10005"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/neelakantan-2022-text-code-embeddings-contrastive-pre/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/neelakantan-2022-text-code-embeddings-contrastive-pre/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / Text and Code Embeddings by Contrastive Pre-Training

# Text and Code Embeddings by Contrastive Pre-Training

**Authors:** [Arvind Neelakantan](https://md.research.elamur.ai/free/en/authors/arvind-neelakantan/), [Tao Xu](https://md.research.elamur.ai/free/en/authors/tao-xu/), [Raul Puri](https://md.research.elamur.ai/free/en/authors/raul-puri/), [Alec Radford](https://md.research.elamur.ai/free/en/authors/alec-radford/), [Jesse Michael Han](https://md.research.elamur.ai/free/en/authors/jesse-michael-han/), [Jerry Tworek](https://md.research.elamur.ai/free/en/authors/jerry-tworek/), [Qiming Yuan](https://md.research.elamur.ai/free/en/authors/qiming-yuan/), [Nikolas Tezak](https://md.research.elamur.ai/free/en/authors/nikolas-tezak/), [Jong Wook Kim](https://md.research.elamur.ai/free/en/authors/jong-wook-kim/), [Chris Hallacy](https://md.research.elamur.ai/free/en/authors/chris-hallacy/), [Johannes Heidecke](https://md.research.elamur.ai/free/en/authors/johannes-heidecke/), Pranav Shyam et al. (25 authors) · **Year:** 2022 · **Venue:** arXiv:2201.10005 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2201.10005) · [PDF](https://arxiv.org/pdf/2201.10005) · [DOI 10.48550/arxiv.2201.10005](https://doi.org/10.48550/arxiv.2201.10005) · [arXiv 2201.10005](https://arxiv.org/abs/2201.10005) · [OpenAlex](https://openalex.org/W4225576545)

**Citations:** 145 citations (openalex-snapshot · 2026-10-14) · [170](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Text embeddings power semantic search, classification and similarity scoring. In earlier work, each use case usually got its own dataset, training objective and architecture, and sentence-embedding research developed separately from neural retrieval. This OpenAI paper shows that one simple recipe can produce general-purpose embeddings for both text and code. A Transformer encoder starts from pre-trained generative weights: GPT for cpt-text and Codex for cpt-code. It is then trained contrastively with in-batch negatives on pairs that occur naturally and need no labels. For text, the pairs are adjacent text fragments from the Internet. For code, they are top-level docstrings matched with their function bodies from open-source repositories. The embedding is the final-layer state at a closing delimiter token, and similarity is measured by cosine. The authors found very large batches essential. On text tasks, quality grows steadily as models scale from 300M to 175B parameters. A single unsupervised model sets new records in linear-probe classification and in code search, and it clearly beats earlier unsupervised retrievers. Its weak spot is sentence similarity, which gets worse with longer training.

## Key points
- The cpt-text family has four sizes: S (300M), M (1.2B), L (6B) and XL (175B). Embedding dimensions run from 1024 to 12288, and batch sizes drop from 12288 for S to 4976 for XL.
- On SentEval, averaged over 7 linear-probe tasks, the best unsupervised model improves on the previous best unsupervised embeddings by 4% (relative) and on the best supervised ones (SimCSE) by 1.8%. Fine-tuning on NLI data adds another 2.2% over the previous best transfer method.
- Unsupervised search beats BM25 by 23.4% on MSMARCO (4M passages). Over 21M Wikipedia documents it beats Contriever by 14.7% on Natural Questions and 10.6% on TriviaQA. After fine-tuning on MS-MARCO, it gains 5.2% on zero-shot BEIR tasks.
- cpt-code is trained on (text, code) pairs and sets a new state of the art on CodeSearchNet, 20.8% above GraphCodeBERT. Growing it from 300M to 1.2B parameters brings no improvement.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

170 papers of our library cite it (of the 140673 whose bibliographies we read); 18 of them have a page here, 152 not yet.

- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://md.research.elamur.ai/free/en/papers/behnamghader-2024-llm2vec-large-language-models-secretly/)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://md.research.elamur.ai/free/en/papers/chen-2024-m3-embedding-multi-linguality/)
- [Promptagator: Few-shot Dense Retrieval From 8 Examples](https://md.research.elamur.ai/free/en/papers/dai-2022-promptagator-few-shot-dense-retrieval/)
- [Exploring Retrieval Augmented Generation in Arabic](https://md.research.elamur.ai/free/en/papers/el-2024-beltagy-exploring-retrieval-augmented-generation-arabic/)
- [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://md.research.elamur.ai/free/en/papers/lee-2024-nv-embed-improved-techniques-training/)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://md.research.elamur.ai/free/en/papers/li-2023-towards-general-text-embeddings-multi/)
- [Fine-Tuning LLaMA for Multi-Stage Text Retrieval](https://md.research.elamur.ai/free/en/papers/ma-2023-fine-tuning-llama-multi-stage/)
- [MTEB: Massive Text Embedding Benchmark](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-mteb-massive-text-embedding-benchmark/)
- [SGPT: GPT Sentence Embeddings for Semantic Search](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-sgpt-gpt-sentence-embeddings-semantic/)
- [Generative Representational Instruction Tuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2024-generative-representational-instruction-tuning/)
- [Nomic Embed: Training a Reproducible Long Context Text Embedder](https://md.research.elamur.ai/free/en/papers/nussbaum-2024-nomic-embed-training-reproducible-long/)
- [Prompting Instruction-tuned LLMs for Semantic Similarity Values](https://md.research.elamur.ai/free/en/papers/snelder-2026-prompting-instruction-tuned-llms-semantic/)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://md.research.elamur.ai/free/en/papers/wang-2022-text-embeddings-weakly-supervised-contrastive/)
- [Improving Text Embeddings with Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2024-improving-text-embeddings-large-language/)
- [C-Pack: Packed Resources For General Chinese Embeddings](https://md.research.elamur.ai/free/en/papers/xiao-2023-c-pack-packed-resources/)
- [mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval](https://md.research.elamur.ai/free/en/papers/zhang-2024-mgte-generalized-long-context-text/)
- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)
- [Large Language Models for Information Retrieval: A Survey](https://md.research.elamur.ai/free/en/papers/zhu-2023-large-language-models-information-retrieval/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{neelakantan2022text,
  title = {{Text and Code Embeddings by Contrastive Pre-Training}},
  author = {Neelakantan, Arvind and Xu, Tao and Puri, Raul and Radford, Alec and Han, Jesse Michael and Tworek, Jerry and Yuan, Qiming and Tezak, Nikolas and Kim, Jong Wook and Hallacy, Chris and Heidecke, Johannes and Shyam, Pranav and Power, Boris and Nekoul, Tyna Eloundou and Sastry, Girish and Krueger, Gretchen and Schnurr, David and Such, Felipe Petroski and Hsu, Kenny and Thompson, Madeleine and Khan, Tabarak and Sherbakov, Toki and Jang, Joanne and Welinder, Peter and Weng, Lilian},
  year = {2022},
  doi = {10.48550/arxiv.2201.10005},
  eprint = {2201.10005},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2201.10005}
}
```

- [← previous Teaching language models to support answers with verified quotes](https://md.research.elamur.ai/free/en/papers/menick-2022-teaching-language-models-support-answers/)
- [next → Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://md.research.elamur.ai/free/en/papers/wang-2022-text-embeddings-weakly-supervised-contrastive/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

82 references, 44 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bengio, Y., Courville, A. C., and Vincent, P. Representation learning: A review and new perspectives. Transactions on pattern analysis and machine intelligence, 35 (8), 2012. [find](https://scholar.google.com/scholar?q=Bengio%2C+Y%2E%2C+Courville%2C+A%2E+C%2E%2C+and+Vincent%2C+P%2E+Representation+learning%3A+A+review+and+new+perspectives%2E+Transactions+on+pattern+analysis+and+machine+intelligence%2C+35+%288%29%2C+2012%2E) [our card](https://md.research.elamur.ai/free/en/papers/bengio-2012-representation-learning-review-new-perspectives/)
2. Bolukbasi, T., Chang, K., Zou, J. Y., Saligrama, V., and Kalai, A. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. 29, 2016. [find](https://scholar.google.com/scholar?q=Bolukbasi%2C+T%2E%2C+Chang%2C+K%2E%2C+Zou%2C+J%2E+Y%2E%2C+Saligrama%2C+V%2E%2C+and+Kalai%2C+A%2E+Man+is+to+computer+programmer+as+woman+is+to+homemaker%3F+debiasing+word+embeddings%2E+29%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/bolukbasi-2016-man-computer-programmer-woman-homemaker/)
3. D. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020. [find](https://scholar.google.com/scholar?q=D%2E+Language+models+are+few%2Dshot+learners%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2020%2E)
4. Caliskan, A., Bryson, J. J., and Narayanan, A. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186, 2017. [find](https://scholar.google.com/scholar?q=Caliskan%2C+A%2E%2C+Bryson%2C+J%2E+J%2E%2C+and+Narayanan%2C+A%2E+Semantics+derived+automatically+from+language+corpora+contain+human%2Dlike+biases%2E+Science%2C+356%286334%29%3A183%E2%80%93186%2C+2017%2E)
5. Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C.… [find](https://scholar.google.com/scholar?q=Chen%2C+M%2E%2C+Tworek%2C+J%2E%2C+Jun%2C+H%2E%2C+Yuan%2C+Q%2E%2C+de+Oliveira+Pinto%2C+H%2E+P%2E%2C+Kaplan%2C+J%2E%2C+Edwards%2C+H%2E%2C+Burda%2C+Y%2E%2C+Joseph%2C+N%2E%2C+Brockman%2C+G%2E%2C+Ray%2C+A%2E%2C+Puri%2C+R%2E%2C+Krueger%2C+G%2E%2C+Petrov%2C+M%2E%2C+Khlaaf%2C+H%2E%2C+Sastry%2C+G%2E%2C+Mis) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
6. Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. E. A simple framework for contrastive learning of visual representations. In International conference on machine learning (ICML), 2020. [find](https://scholar.google.com/scholar?q=Chen%2C+T%2E%2C+Kornblith%2C+S%2E%2C+Norouzi%2C+M%2E%2C+and+Hinton%2C+G%2E+E%2E+A+simple+framework+for+contrastive+learning+of+visual+representations%2E+In+International+conference+on+machine+learning+%28ICML%29%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2020-simple-framework-contrastive-learning-visual/)
7. Chopra, S., Hadsell, R., and LeCun, Y. Learning a similarity metric discriminatively, with application to face verification. In Computer Vision and Pattern Recognition (CVPR). IEEE, 2005. [find](https://scholar.google.com/scholar?q=Chopra%2C+S%2E%2C+Hadsell%2C+R%2E%2C+and+LeCun%2C+Y%2E+Learning+a+similarity+metric+discriminatively%2C+with+application+to+face+verification%2E+In+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2E+IEEE%2C+2005%2E)
8. Conneau, A. and Kiela, D. Senteval: An evaluation toolkit for universal sentence representations. arXiv preprint arXiv:1803.05449, 2018. [find](https://scholar.google.com/scholar?q=Conneau%2C+A%2E+and+Kiela%2C+D%2E+Senteval%3A+An+evaluation+toolkit+for+universal+sentence+representations%2E+arXiv+preprint+arXiv%3A1803%2E05449%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2018-senteval-evaluation-toolkit-universal-sentence/)
9. Crawford, K. The trouble with bias. Keynote at NeurIPS, 2017. [find](https://scholar.google.com/scholar?q=Crawford%2C+K%2E+The+trouble+with+bias%2E+Keynote+at+NeurIPS%2C+2017%2E)
10. Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D. A large annotated corpus for learning natural language inference. In Conference on Empirical Methods in Natural Language Processing (EMNLP). ACL, 2015. [find](https://scholar.google.com/scholar?q=Bowman%2C+S%2E+R%2E%2C+Angeli%2C+G%2E%2C+Potts%2C+C%2E%2C+and+Manning%2C+C%2E+D%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E+AC)
11. Deerwester, S., Dumais, S., Furnas, G., Landauer, T., and Harshman, R. Indexing by latent semantic analysis. Journal of the American society for information science, 41(6):391–407, 1990. [find](https://scholar.google.com/scholar?q=Deerwester%2C+S%2E%2C+Dumais%2C+S%2E%2C+Furnas%2C+G%2E%2C+Landauer%2C+T%2E%2C+and+Harshman%2C+R%2E+Indexing+by+latent+semantic+analysis%2E+Journal+of+the+American+society+for+information+science%2C+41%286%29%3A391%E2%80%93407%2C+1990%2E)
12. Brown, P. F., Della Pietra, V. J., deSouza, P. V., Lai, J. C., and Mercer, R. L. Class-based n-gram models of natural language. Computational Linguistics, 18(4):467– 480, 1992. [find](https://scholar.google.com/scholar?q=Brown%2C+P%2E+F%2E%2C+Della+Pietra%2C+V%2E+J%2E%2C+deSouza%2C+P%2E+V%2E%2C+Lai%2C+J%2E+C%2E%2C+and+Mercer%2C+R%2E+L%2E+Class%2Dbased+n%2Dgram+models+of+natural+language%2E+Computational+Linguistics%2C+18%284%29%3A467%E2%80%93+480%2C+1992%2E)
13. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). ACL, 2019. [find](https://scholar.google.com/scholar?q=Devlin%2C+J%2E%2C+Chang%2C+M%2E%2DW%2E%2C+Lee%2C+K%2E%2C+and+Toutanova%2C+K%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Conference+of+the+North+American+Chapter+of+the+Association+fo)
14. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess… — [arXiv:2005.12766](https://arxiv.org/abs/2005.12766) [find](https://scholar.google.com/scholar?q=Brown%2C+T%2E%2C+Mann%2C+B%2E%2C+Ryder%2C+N%2E%2C+Subbiah%2C+M%2E%2C+Kaplan%2C+J%2E+D%2E%2C+Dhariwal%2C+P%2E%2C+Neelakantan%2C+A%2E%2C+Shyam%2C+P%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+Agarwal%2C+S%2E%2C+Herbert%2DVoss%2C+A%2E%2C+Krueger%2C+G%2E%2C+Henighan%2C+T%2E%2C+Child%2C+R%2E%2C+Rames)
15. Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., and Zhou, M. Codebert: A pre-trained model for programming and natural languages. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. [find](https://scholar.google.com/scholar?q=Feng%2C+Z%2E%2C+Guo%2C+D%2E%2C+Tang%2C+D%2E%2C+Duan%2C+N%2E%2C+Feng%2C+X%2E%2C+Gong%2C+M%2E%2C+Shou%2C+L%2E%2C+Qin%2C+B%2E%2C+Liu%2C+T%2E%2C+Jiang%2C+D%2E%2C+and+Zhou%2C+M%2E+Codebert%3A+A+pre%2Dtrained+model+for+programming+and+natural+languages%2E+In+Conference+on+Emp) [our card](https://md.research.elamur.ai/free/en/papers/feng-2020-codebert-pre-trained-model-programming/)
16. Formal, T., Lassance, C., Piwowarski, B., and Clinchant, S. SPLADE v2: Sparse lexical and expansion model for information retrieval. arXiv preprint arXiv:2109.10086, 2021. [find](https://scholar.google.com/scholar?q=Formal%2C+T%2E%2C+Lassance%2C+C%2E%2C+Piwowarski%2C+B%2E%2C+and+Clinchant%2C+S%2E+SPLADE+v2%3A+Sparse+lexical+and+expansion+model+for+information+retrieval%2E+arXiv+preprint+arXiv%3A2109%2E10086%2C+2021%2E)
17. Gao, T., Yao, X., and Chen, D. SimCSE: Simple contrastive learning of sentence embeddings. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021. [find](https://scholar.google.com/scholar?q=Gao%2C+T%2E%2C+Yao%2C+X%2E%2C+and+Chen%2C+D%2E+SimCSE%3A+Simple+contrastive+learning+of+sentence+embeddings%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/gao-2021-simcse-simple-contrastive-learning-sentence/)
18. Giorgi, J. M., Nitski, O., Bader, G. D., and Wang, B. Declutr: Deep contrastive learning for unsupervised textual representations. In Proceedings of ACL/IJCNLP, 2020. [find](https://scholar.google.com/scholar?q=Giorgi%2C+J%2E+M%2E%2C+Nitski%2C+O%2E%2C+Bader%2C+G%2E+D%2E%2C+and+Wang%2C+B%2E+Declutr%3A+Deep+contrastive+learning+for+unsupervised+textual+representations%2E+In+Proceedings+of+ACL%2FIJCNLP%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/giorgi-2021-declutr-deep-contrastive-learning-unsupervised/)
19. Goodman, N. Seven strictures on similarity. Bobbs Merrill, 1972. [find](https://scholar.google.com/scholar?q=Goodman%2C+N%2E+Seven+strictures+on+similarity%2E+Bobbs+Merrill%2C+1972%2E)
20. Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Liu, S., Zhou, L., Duan, N., Svyatkovskiy, A., Fu, S., Tufano, M., Deng, S. K., Clement, C. B., Drain, D., Sundaresan, N., Yin, J., Jiang, D., and Zhou, M. Graphcodebert: Pre-training code representations with data flow. In International Conference on Le… [find](https://scholar.google.com/scholar?q=Guo%2C+D%2E%2C+Ren%2C+S%2E%2C+Lu%2C+S%2E%2C+Feng%2C+Z%2E%2C+Tang%2C+D%2E%2C+Liu%2C+S%2E%2C+Zhou%2C+L%2E%2C+Duan%2C+N%2E%2C+Svyatkovskiy%2C+A%2E%2C+Fu%2C+S%2E%2C+Tufano%2C+M%2E%2C+Deng%2C+S%2E+K%2E%2C+Clement%2C+C%2E+B%2E%2C+Drain%2C+D%2E%2C+Sundaresan%2C+N%2E%2C+Yin%2C+J%2E%2C+Jiang%2C+D%2E%2C+and+Zhou%2C+M) [our card](https://md.research.elamur.ai/free/en/papers/guo-2021-graphcodebert-pre-training-code-representations/)
21. Gutmann, M. and Hyvärinen, A. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Conference on Artificial Intelligence and Statistics. PMLR, 2010. [find](https://scholar.google.com/scholar?q=Gutmann%2C+M%2E+and+Hyva%CC%88rinen%2C+A%2E+Noise%2Dcontrastive+estimation%3A+A+new+estimation+principle+for+unnormalized+statistical+models%2E+In+Conference+on+Artificial+Intelligence+and+Statistics%2E+PMLR%2C+2010%2E)
22. Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M. REALM: retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. [find](https://scholar.google.com/scholar?q=Guu%2C+K%2E%2C+Lee%2C+K%2E%2C+Tung%2C+Z%2E%2C+Pasupat%2C+P%2E%2C+and+Chang%2C+M%2E+REALM%3A+retrieval%2Daugmented+language+model+pre%2Dtraining%2E+arXiv+preprint+arXiv%3A2002%2E08909%2C+2020%2E)
23. Hadsell, R., Chopra, S., and LeCun, Y. Dimensionality reduction by learning an invariant mapping. In Computer Vision and Pattern Recognition (CVPR), volume 2, pp. 1735–1742. IEEE, 2006. [find](https://scholar.google.com/scholar?q=Hadsell%2C+R%2E%2C+Chopra%2C+S%2E%2C+and+LeCun%2C+Y%2E+Dimensionality+reduction+by+learning+an+invariant+mapping%2E+In+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C+volume+2%2C+pp%2E+1735%E2%80%931742%2E+IEEE%2C+2006%2E)
24. He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. B. Momentum contrast for unsupervised visual representation learning. In Computer Vision and Pattern Recognition (CVPR), 2020. [find](https://scholar.google.com/scholar?q=He%2C+K%2E%2C+Fan%2C+H%2E%2C+Wu%2C+Y%2E%2C+Xie%2C+S%2E%2C+and+Girshick%2C+R%2E+B%2E+Momentum+contrast+for+unsupervised+visual+representation+learning%2E+In+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/he-2019-momentum-contrast-unsupervised-visual-representation/)
25. Izacard, G., Caron, M., Hosseini, L., Riedel, S., Bojanowski, P., Joulin, A., and Grave, E. Towards unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118, 2021. [find](https://scholar.google.com/scholar?q=Izacard%2C+G%2E%2C+Caron%2C+M%2E%2C+Hosseini%2C+L%2E%2C+Riedel%2C+S%2E%2C+Bojanowski%2C+P%2E%2C+Joulin%2C+A%2E%2C+and+Grave%2C+E%2E+Towards+unsupervised+dense+information+retrieval+with+contrastive+learning%2E+arXiv+preprint+arXiv%3A2112%2E09118%2C) [our card](https://md.research.elamur.ai/free/en/papers/izacard-2021-unsupervised-dense-information-retrieval-contrastive/)
26. Jain, P., Jain, A., Zhang, T., Abbeel, P., Gonzalez, J. E., and Stoica, I. Contrastive code representation learning. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021. [find](https://scholar.google.com/scholar?q=Jain%2C+P%2E%2C+Jain%2C+A%2E%2C+Zhang%2C+T%2E%2C+Abbeel%2C+P%2E%2C+Gonzalez%2C+J%2E+E%2E%2C+and+Stoica%2C+I%2E+Contrastive+code+representation+learning%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+2021%2E)
27. Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., and Duerig, T. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning (ICML), 2021. [find](https://scholar.google.com/scholar?q=Jia%2C+C%2E%2C+Yang%2C+Y%2E%2C+Xia%2C+Y%2E%2C+Chen%2C+Y%2E%2DT%2E%2C+Parekh%2C+Z%2E%2C+Pham%2C+H%2E%2C+Le%2C+Q%2E+V%2E%2C+Sung%2C+Y%2E%2C+Li%2C+Z%2E%2C+and+Duerig%2C+T%2E+Scaling+up+visual+and+vision%2Dlanguage+representation+learning+with+noisy+text+supervision%2E+In) [our card](https://md.research.elamur.ai/free/en/papers/jia-2021-scaling-up-visual-vision-language/)
28. Johnson, J., Douze, M., and Jégou, H. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 2019. [find](https://scholar.google.com/scholar?q=Johnson%2C+J%2E%2C+Douze%2C+M%2E%2C+and+Je%CC%81gou%2C+H%2E+Billion%2Dscale+similarity+search+with+gpus%2E+IEEE+Transactions+on+Big+Data%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/johnson-2017-billion-scale-similarity-search-gpus/)
29. Joshi, M., Choi, E., Weld, D., and Zettlemoyer, L. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Conference of the Association for Computational Linguistics (ACL). ACL, 2017. [find](https://scholar.google.com/scholar?q=Joshi%2C+M%2E%2C+Choi%2C+E%2E%2C+Weld%2C+D%2E%2C+and+Zettlemoyer%2C+L%2E+TriviaQA%3A+A+large+scale+distantly+supervised+challenge+dataset+for+reading+comprehension%2E+In+Conference+of+the+Association+for+Computational+Linguist) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)
30. Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-t. Dense passage retrieval for open-domain question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020a. [find](https://scholar.google.com/scholar?q=Karpukhin%2C+V%2E%2C+Oguz%2C+B%2E%2C+Min%2C+S%2E%2C+Lewis%2C+P%2E%2C+Wu%2C+L%2E%2C+Edunov%2C+S%2E%2C+Chen%2C+D%2E%2C+and+Yih%2C+W%2E%2Dt%2E+Dense+passage+retrieval+for+open%2Ddomain+question+answering%2E+In+Conference+on+Empirical+Methods+in+Natural+Lang) [our card](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/)
31. Karpukhin, V., Oguz, B., Min, S., Wu, L., Edunov, S., Chen, D., and Yih, W. Dense passage retrieval for open-domain question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020b. [find](https://scholar.google.com/scholar?q=Karpukhin%2C+V%2E%2C+Oguz%2C+B%2E%2C+Min%2C+S%2E%2C+Wu%2C+L%2E%2C+Edunov%2C+S%2E%2C+Chen%2C+D%2E%2C+and+Yih%2C+W%2E+Dense+passage+retrieval+for+open%2Ddomain+question+answering%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processin) [our card](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/)
32. Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., and Krishnan, D. Supervised contrastive learning. arXiv preprint arXiv:2004.11362, 2020. — [arXiv:2004.11362](https://arxiv.org/abs/2004.11362) [find](https://scholar.google.com/scholar?q=Khosla%2C+P%2E%2C+Teterwak%2C+P%2E%2C+Wang%2C+C%2E%2C+Sarna%2C+A%2E%2C+Tian%2C+Y%2E%2C+Isola%2C+P%2E%2C+Maschinot%2C+A%2E%2C+Liu%2C+C%2E%2C+and+Krishnan%2C+D%2E+Supervised+contrastive+learning%2E+arXiv+preprint+arXiv%3A2004%2E11362%2C+2020%2E+%E2%80%94+%5BarXiv%3A2004%2E11362)
33. Kim, W., Son, B., and Kim, I. Vilt: Vision-and-language transformer without convolution or region supervision. In International Conference on Machine Learning (ICML), 2021. [find](https://scholar.google.com/scholar?q=Kim%2C+W%2E%2C+Son%2C+B%2E%2C+and+Kim%2C+I%2E+Vilt%3A+Vision%2Dand%2Dlanguage+transformer+without+convolution+or+region+supervision%2E+In+International+Conference+on+Machine+Learning+%28ICML%29%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/kim-2021-vilt-vision-language-transformer-convolution/)
34. Hofstätter, S., Lin, S., Yang, J., Lin, J., and Hanbury, A. Efficiently teaching an effective dense retriever with balanced topic aware sampling. arXiv preprint arXiv:2104.06967, 2021. [find](https://scholar.google.com/scholar?q=Hofsta%CC%88tter%2C+S%2E%2C+Lin%2C+S%2E%2C+Yang%2C+J%2E%2C+Lin%2C+J%2E%2C+and+Hanbury%2C+A%2E+Efficiently+teaching+an+effective+dense+retriever+with+balanced+topic+aware+sampling%2E+arXiv+preprint+arXiv%3A2104%2E06967%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/hofstatter-2021-efficiently-teaching-effective-dense-retriever/)
35. Kingma, D. P. and Welling, M. Auto-Encoding Variational Bayes. In International Conference on Learning Representation (ICLR), 2014. [find](https://scholar.google.com/scholar?q=Kingma%2C+D%2E+P%2E+and+Welling%2C+M%2E+Auto%2DEncoding+Variational+Bayes%2E+In+International+Conference+on+Learning+Representation+%28ICLR%29%2C+2014%2E) [our card](https://md.research.elamur.ai/free/en/papers/kingma-2013-auto-encoding-variational-bayes/)
36. Husain, H., Wu, H.-H., Gazit, T., Allamanis, M., and Brockschmidt, M. CodeSearchNet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436, 2020. [find](https://scholar.google.com/scholar?q=Husain%2C+H%2E%2C+Wu%2C+H%2E%2DH%2E%2C+Gazit%2C+T%2E%2C+Allamanis%2C+M%2E%2C+and+Brockschmidt%2C+M%2E+CodeSearchNet+challenge%3A+Evaluating+the+state+of+semantic+code+search%2E+arXiv+preprint+arXiv%3A1909%2E09436%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/husain-2020-codesearchnet-challenge-evaluating-state-semantic/)
37. Kiros, J., Zhu, Y., Salakhutdinov, R., Zemel, R. S., Torralba, A., Urtasun, R., and Fidler, S. Skip-thought vectors. In Advances in Neural Information Processing Systems (NeuriPS), 2015. [find](https://scholar.google.com/scholar?q=Kiros%2C+J%2E%2C+Zhu%2C+Y%2E%2C+Salakhutdinov%2C+R%2E%2C+Zemel%2C+R%2E+S%2E%2C+Torralba%2C+A%2E%2C+Urtasun%2C+R%2E%2C+and+Fidler%2C+S%2E+Skip%2Dthought+vectors%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NeuriPS%29%2C+2015%2E)
38. Kobayashi, S. Contextual augmentation: Data augmentation by words with paradigmatic relations. arXiv preprint arXiv:1805.06201, 2018. [find](https://scholar.google.com/scholar?q=Kobayashi%2C+S%2E+Contextual+augmentation%3A+Data+augmentation+by+words+with+paradigmatic+relations%2E+arXiv+preprint+arXiv%3A1805%2E06201%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/kobayashi-2018-contextual-augmentation-data-augmentation-words/)
39. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Kelcey, M., Devlin, J., Lee, K., Toutanova, K. N., Jones, L., Chang, M.-W., Dai, A., Uszkoreit, J., Le, Q., and Petrov, S. Natural questions: a benchmark for question answering research. T… [find](https://scholar.google.com/scholar?q=Kwiatkowski%2C+T%2E%2C+Palomaki%2C+J%2E%2C+Redfield%2C+O%2E%2C+Collins%2C+M%2E%2C+Parikh%2C+A%2E%2C+Alberti%2C+C%2E%2C+Epstein%2C+D%2E%2C+Polosukhin%2C+I%2E%2C+Kelcey%2C+M%2E%2C+Devlin%2C+J%2E%2C+Lee%2C+K%2E%2C+Toutanova%2C+K%2E+N%2E%2C+Jones%2C+L%2E%2C+Chang%2C+M%2E%2DW%2E%2C+Dai%2C+A%2E%2C+Usz) [our card](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
40. Lee, K., Chang, M., and Toutanova, K. Latent retrieval for weakly supervised open domain question answering. In Korhonen, A., Traum, D. R., and Màrquez, L. (eds.), Conference of the Association for Computational Linguistics (ACL), pp. 6086–6096. ACL. Lewis, P. S. H., Perez, E., Piktus, A., Petroni,… [find](https://scholar.google.com/scholar?q=Lee%2C+K%2E%2C+Chang%2C+M%2E%2C+and+Toutanova%2C+K%2E+Latent+retrieval+for+weakly+supervised+open+domain+question+answering%2E+In+Korhonen%2C+A%2E%2C+Traum%2C+D%2E+R%2E%2C+and+Ma%CC%80rquez%2C+L%2E+%28eds%2E%29%2C+Conference+of+the+Association+for+C) [our card](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/)
41. Li, B., Zhou, H., He, J., Wang, M., Yang, Y., and Li, L. On the sentence embeddings from pre-trained language models. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. [find](https://scholar.google.com/scholar?q=Li%2C+B%2E%2C+Zhou%2C+H%2E%2C+He%2C+J%2E%2C+Wang%2C+M%2E%2C+Yang%2C+Y%2E%2C+and+Li%2C+L%2E+On+the+sentence+embeddings+from+pre%2Dtrained+language+models%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/li-2020-sentence-embeddings-pre-trained-language/)
42. Liang, P. P., Li, I. M., Zheng, E., Lim, Y. C., Salakhutdinov, R., and Morency, L. Towards debiasing sentence representations. In Conference of the Association for Computational Linguistics (ACL), 2020. [find](https://scholar.google.com/scholar?q=Liang%2C+P%2E+P%2E%2C+Li%2C+I%2E+M%2E%2C+Zheng%2C+E%2E%2C+Lim%2C+Y%2E+C%2E%2C+Salakhutdinov%2C+R%2E%2C+and+Morency%2C+L%2E+Towards+debiasing+sentence+representations%2E+In+Conference+of+the+Association+for+Computational+Linguistics+%28ACL%29%2C+202) [our card](https://md.research.elamur.ai/free/en/papers/liang-2020-debiasing-sentence-representations/)
43. Lin, J., Nogueira, R., and Yates, A. Pretrained transformers for text ranking: BERT and beyond. Synthesis Lectures on Human Language Technologies, 14(4):1–325, 2021. [find](https://scholar.google.com/scholar?q=Lin%2C+J%2E%2C+Nogueira%2C+R%2E%2C+and+Yates%2C+A%2E+Pretrained+transformers+for+text+ranking%3A+BERT+and+beyond%2E+Synthesis+Lectures+on+Human+Language+Technologies%2C+14%284%29%3A1%E2%80%93325%2C+2021%2E)
44. Logeswaran, L. and Lee, H. An efficient framework for learning sentence representations. In International Conference on Learning Representation (ICLR), 2018. [find](https://scholar.google.com/scholar?q=Logeswaran%2C+L%2E+and+Lee%2C+H%2E+An+efficient+framework+for+learning+sentence+representations%2E+In+International+Conference+on+Learning+Representation+%28ICLR%29%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/logeswaran-2018-efficient-framework-learning-sentence-representations/)
45. Lu, J., Batra, D., Parikh, D., and Lee, S. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. arXiv preprint arXiv:1908.02265, 2019. [find](https://scholar.google.com/scholar?q=Lu%2C+J%2E%2C+Batra%2C+D%2E%2C+Parikh%2C+D%2E%2C+and+Lee%2C+S%2E+Vilbert%3A+Pretraining+task%2Dagnostic+visiolinguistic+representations+for+vision%2Dand%2Dlanguage+tasks%2E+arXiv+preprint+arXiv%3A1908%2E02265%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/lu-2019-vilbert-pretraining-task-agnostic-visiolinguistic/)
46. Nguyen, T., Rosenberg, M., Song, X., Gao, J., Tiwary, S., Majumder, R., and Deng, L. MS MARCO: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268, 2016. [find](https://scholar.google.com/scholar?q=Nguyen%2C+T%2E%2C+Rosenberg%2C+M%2E%2C+Song%2C+X%2E%2C+Gao%2C+J%2E%2C+Tiwary%2C+S%2E%2C+Majumder%2C+R%2E%2C+and+Deng%2C+L%2E+MS+MARCO%3A+A+human+generated+machine+reading+comprehension+dataset%2E+arXiv+preprint+arXiv%3A1611%2E09268%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/)
47. Nogueira, R., Lin, J., and Epistemic, A. From doc2query to doctttttquery. Online preprint, 2019a. [find](https://scholar.google.com/scholar?q=Nogueira%2C+R%2E%2C+Lin%2C+J%2E%2C+and+Epistemic%2C+A%2E+From+doc2query+to+doctttttquery%2E+Online+preprint%2C+2019a%2E)
48. Nogueira, R., Yang, W., Cho, K., and Lin, J. Multi-stage document ranking with BERT. arXiv preprint arXiv:1910.14424, 2019b. — [arXiv:1910.14424](https://arxiv.org/abs/1910.14424) [find](https://scholar.google.com/scholar?q=Nogueira%2C+R%2E%2C+Yang%2C+W%2E%2C+Cho%2C+K%2E%2C+and+Lin%2C+J%2E+Multi%2Dstage+document+ranking+with+BERT%2E+arXiv+preprint+arXiv%3A1910%2E14424%2C+2019b%2E+%E2%80%94+%5BarXiv%3A1910%2E14424%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1910%2E14424%29)
49. Park, J. H., Shin, J., and Fung, P. Reducing gender bias in abusive language detection. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018. [find](https://scholar.google.com/scholar?q=Park%2C+J%2E+H%2E%2C+Shin%2C+J%2E%2C+and+Fung%2C+P%2E+Reducing+gender+bias+in+abusive+language+detection%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/park-2018-reducing-gender-bias-abusive-language/)
50. Pennington, J., Socher, R., and Manning, C. GloVe: Global vectors for word representation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014. [find](https://scholar.google.com/scholar?q=Pennington%2C+J%2E%2C+Socher%2C+R%2E%2C+and+Manning%2C+C%2E+GloVe%3A+Global+vectors+for+word+representation%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+2014%2E)
51. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. Deep contextualized word representations. In Proceedings of NCAAL/IJCNLP, 2018. [find](https://scholar.google.com/scholar?q=Peters%2C+M%2E+E%2E%2C+Neumann%2C+M%2E%2C+Iyyer%2C+M%2E%2C+Gardner%2C+M%2E%2C+Clark%2C+C%2E%2C+Lee%2C+K%2E%2C+and+Zettlemoyer%2C+L%2E+Deep+contextualized+word+representations%2E+In+Proceedings+of+NCAAL%2FIJCNLP%2C+2018%2E)
52. Qu, Y., Ding, Y., Liu, J., Liu, K., Ren, R., Zhao, X., Dong, D., Wu, H., and Wang, H. Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering. In Conference of the Association for Computational Linguistics (ACL), 2021. [find](https://scholar.google.com/scholar?q=Qu%2C+Y%2E%2C+Ding%2C+Y%2E%2C+Liu%2C+J%2E%2C+Liu%2C+K%2E%2C+Ren%2C+R%2E%2C+Zhao%2C+X%2E%2C+Dong%2C+D%2E%2C+Wu%2C+H%2E%2C+and+Wang%2C+H%2E+Rocketqa%3A+An+optimized+training+approach+to+dense+passage+retrieval+for+open%2Ddomain+question+answering%2E+In+Confere) [our card](https://md.research.elamur.ai/free/en/papers/qu-2021-rocketqa-optimized-training-approach-dense/)
53. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020, 2021. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Kim%2C+J%2E+W%2E%2C+Hallacy%2C+C%2E%2C+Ramesh%2C+A%2E%2C+Goh%2C+G%2E%2C+Agarwal%2C+S%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+Mishkin%2C+P%2E%2C+Clark%2C+J%2E%2C+Krueger%2C+G%2E%2C+and+Sutskever%2C+I%2E+Learning+transferable+visual+models+from+natural+) [our card](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/)
54. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019. [find](https://scholar.google.com/scholar?q=Raffel%2C+C%2E%2C+Shazeer%2C+N%2E%2C+Roberts%2C+A%2E%2C+Lee%2C+K%2E%2C+Narang%2C+S%2E%2C+Matena%2C+M%2E%2C+Zhou%2C+Y%2E%2C+Li%2C+W%2E%2C+and+Liu%2C+P%2E+J%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext+transformer%2E+arXiv+preprin)
55. Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092, 2021. [find](https://scholar.google.com/scholar?q=Ramesh%2C+A%2E%2C+Pavlov%2C+M%2E%2C+Goh%2C+G%2E%2C+Gray%2C+S%2E%2C+Voss%2C+C%2E%2C+Radford%2C+A%2E%2C+Chen%2C+M%2E%2C+and+Sutskever%2C+I%2E+Zero%2Dshot+text%2Dto%2Dimage+generation%2E+arXiv+preprint+arXiv%3A2102%2E12092%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/ramesh-2021-zero-shot-text-image-generation/)
56. May, C., Wang, A., Bordia, S., Bowman, S. R., and Rudinger, R. On measuring social biases in sentence encoders. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019. [find](https://scholar.google.com/scholar?q=May%2C+C%2E%2C+Wang%2C+A%2E%2C+Bordia%2C+S%2E%2C+Bowman%2C+S%2E+R%2E%2C+and+Rudinger%2C+R%2E+On+measuring+social+biases+in+sentence+encoders%2E+In+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguis) [our card](https://md.research.elamur.ai/free/en/papers/may-2019-measuring-social-biases-sentence-encoders/)
57. Reimers, N. and Gurevych, I. Sentence-bert: Sentence embeddings using siamese bert-networks. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019. [find](https://scholar.google.com/scholar?q=Reimers%2C+N%2E+and+Gurevych%2C+I%2E+Sentence%2Dbert%3A+Sentence+embeddings+using+siamese+bert%2Dnetworks%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+2019%2E)
58. Mikolov, T., Chen, K., Corrado, G. S., and Dean, J. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. [find](https://scholar.google.com/scholar?q=Mikolov%2C+T%2E%2C+Chen%2C+K%2E%2C+Corrado%2C+G%2E+S%2E%2C+and+Dean%2C+J%2E+Efficient+estimation+of+word+representations+in+vector+space%2E+arXiv+preprint+arXiv%3A1301%2E3781%2C+2013%2E)
59. Robertson, S. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends® in Information Retrieval, 2009. [find](https://scholar.google.com/scholar?q=Robertson%2C+S%2E+The+Probabilistic+Relevance+Framework%3A+BM25+and+Beyond%2E+Foundations+and+Trends%C2%AE+in+Information+Retrieval%2C+2009%2E)
60. Rudinger, R., Naradowsky, J., Leonard, B., and Durme, B. V. Gender bias in coreference resolution. arXiv preprint arXiv:1804.09301, 2018. [find](https://scholar.google.com/scholar?q=Rudinger%2C+R%2E%2C+Naradowsky%2C+J%2E%2C+Leonard%2C+B%2E%2C+and+Durme%2C+B%2E+V%2E+Gender+bias+in+coreference+resolution%2E+arXiv+preprint+arXiv%3A1804%2E09301%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/rudinger-2018-gender-bias-coreference-resolution/)
61. Sachan, D. S., Patwary, M., Shoeybi, M., Kant, N., Ping, W., Hamilton, W. L., and Catanzaro, B. End-to-end training of neural retrievers for open-domain question answering. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Proceedings of ACL/IJCNLP, pp. 6648–6662. ACL, 2021. [find](https://scholar.google.com/scholar?q=Sachan%2C+D%2E+S%2E%2C+Patwary%2C+M%2E%2C+Shoeybi%2C+M%2E%2C+Kant%2C+N%2E%2C+Ping%2C+W%2E%2C+Hamilton%2C+W%2E+L%2E%2C+and+Catanzaro%2C+B%2E+End%2Dto%2Dend+training+of+neural+retrievers+for+open%2Ddomain+question+answering%2E+In+Zong%2C+C%2E%2C+Xia%2C+F%2E%2C+Li%2C+W)
62. Santhanam, K., Khattab, O., Saad-Falcon, J., Potts, C., and Zaharia, M. Colbertv2: Effective and efficient retrieval via lightweight late interaction. arXiv preprint arXiv:2112.01488, 2021. [find](https://scholar.google.com/scholar?q=Santhanam%2C+K%2E%2C+Khattab%2C+O%2E%2C+Saad%2DFalcon%2C+J%2E%2C+Potts%2C+C%2E%2C+and+Zaharia%2C+M%2E+Colbertv2%3A+Effective+and+efficient+retrieval+via+lightweight+late+interaction%2E+arXiv+preprint+arXiv%3A2112%2E01488%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/santhanam-2022-colbertv2-effective-efficient-retrieval-lightweight/)
63. Schroff, F., Kalenichenko, D., and Philbin, J. Facenet: A unified embedding for face recognition and clustering. In Computer Vision and Pattern Recognition (CVPR), 2015. [find](https://scholar.google.com/scholar?q=Schroff%2C+F%2E%2C+Kalenichenko%2C+D%2E%2C+and+Philbin%2C+J%2E+Facenet%3A+A+unified+embedding+for+face+recognition+and+clustering%2E+In+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/schroff-2015-facenet-unified-embedding-face-recognition/)
64. Shen, D., Zheng, M., Shen, Y., Qu, Y., and Chen, W. A simple but tough-to-beat data augmentation approach for natural language understanding and generation. arXiv preprint arXiv:2009.13818, 2020. — [arXiv:2009.13818](https://arxiv.org/abs/2009.13818) [find](https://scholar.google.com/scholar?q=Shen%2C+D%2E%2C+Zheng%2C+M%2E%2C+Shen%2C+Y%2E%2C+Qu%2C+Y%2E%2C+and+Chen%2C+W%2E+A+simple+but+tough%2Dto%2Dbeat+data+augmentation+approach+for+natural+language+understanding+and+generation%2E+arXiv+preprint+arXiv%3A2009%2E13818%2C+2020%2E+%E2%80%94+%5Ba)
65. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2013. [find](https://scholar.google.com/scholar?q=Socher%2C+R%2E%2C+Perelygin%2C+A%2E%2C+Wu%2C+J%2E%2C+Chuang%2C+J%2E%2C+Manning%2C+C%2E+D%2E%2C+Ng%2C+A%2E%2C+and+Potts%2C+C%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In+Conference+on+Empirical+Methods+i)
66. Sohn, K. Improved deep metric learning with multi-class n-pair loss objective. In Advances in Neural Information Processing Systems (NeuriPS), 2016. [find](https://scholar.google.com/scholar?q=Sohn%2C+K%2E+Improved+deep+metric+learning+with+multi%2Dclass+n%2Dpair+loss+objective%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NeuriPS%29%2C+2016%2E)
67. Solaiman, I. and Dennison, C. Process for adapting language models to society (PALMS) with values-targeted datasets. arXiv preprint arXiv:2106.10328, 2021. [find](https://scholar.google.com/scholar?q=Solaiman%2C+I%2E+and+Dennison%2C+C%2E+Process+for+adapting+language+models+to+society+%28PALMS%29+with+values%2Dtargeted+datasets%2E+arXiv+preprint+arXiv%3A2106%2E10328%2C+2021%2E)
68. Su, J., Cao, J., Liu, W., and Ou, Y. Whitening sentence representations for better semantics and faster retrieval. arXiv preprint arXiv:2103.15316, 2021. — [arXiv:2103.15316](https://arxiv.org/abs/2103.15316) [find](https://scholar.google.com/scholar?q=Su%2C+J%2E%2C+Cao%2C+J%2E%2C+Liu%2C+W%2E%2C+and+Ou%2C+Y%2E+Whitening+sentence+representations+for+better+semantics+and+faster+retrieval%2E+arXiv+preprint+arXiv%3A2103%2E15316%2C+2021%2E+%E2%80%94+%5BarXiv%3A2103%2E15316%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F210)
69. Sun, C., Myers, A., Vondrick, C., Murphy, K., and Schmid, C. Videobert: A joint model for video and language representation learning. In International Conference on Computer Vision (ICCV), 2019. [find](https://scholar.google.com/scholar?q=Sun%2C+C%2E%2C+Myers%2C+A%2E%2C+Vondrick%2C+C%2E%2C+Murphy%2C+K%2E%2C+and+Schmid%2C+C%2E+Videobert%3A+A+joint+model+for+video+and+language+representation+learning%2E+In+International+Conference+on+Computer+Vision+%28ICCV%29%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/sun-2019-videobert-joint-model-video-language/)
70. Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., and Gurevych, I. BEIR: A heterogenous benchmark for zero-shot evaluation of information retrieval models. In Advances in Neural Information Processing Systems (NeuriPS), 2021. [find](https://scholar.google.com/scholar?q=Thakur%2C+N%2E%2C+Reimers%2C+N%2E%2C+Ru%CC%88ckle%CC%81%2C+A%2E%2C+Srivastava%2C+A%2E%2C+and+Gurevych%2C+I%2E+BEIR%3A+A+heterogenous+benchmark+for+zero%2Dshot+evaluation+of+information+retrieval+models%2E+In+Advances+in+Neural+Information+Proce) [our card](https://md.research.elamur.ai/free/en/papers/thakur-2021-beir-heterogenous-benchmark-zero/)
71. Tian, Y., Krishnan, D., and Isola, P. Contrastive multi-view coding. European Conference on Computer Vision (ECCV), 2019. van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. Wavenet: A generative model for raw audio. arXiv… [find](https://scholar.google.com/scholar?q=Tian%2C+Y%2E%2C+Krishnan%2C+D%2E%2C+and+Isola%2C+P%2E+Contrastive+multi%2Dview+coding%2E+European+Conference+on+Computer+Vision+%28ECCV%29%2C+2019%2E+van+den+Oord%2C+A%2E%2C+Dieleman%2C+S%2E%2C+Zen%2C+H%2E%2C+Simonyan%2C+K%2E%2C+Vinyals%2C+O%2E%2C+Graves%2C+A%2E) [our card](https://md.research.elamur.ai/free/en/papers/oord-2016-wavenet-generative-model-raw-audio/)
72. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems (NeuriPS), 2017. [find](https://scholar.google.com/scholar?q=Vaswani%2C+A%2E%2C+Shazeer%2C+N%2E%2C+Parmar%2C+N%2E%2C+Uszkoreit%2C+J%2E%2C+Jones%2C+L%2E%2C+Gomez%2C+A%2E+N%2E%2C+Kaiser%2C+L%2E+u%2E%2C+and+Polosukhin%2C+I%2E+Attention+is+all+you+need%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NeuriPS)
73. Vervaeke, J., Lillicrap, T. P., and Richards, B. A. Relevance realization and the emerging framework in cognitive science. Journal of logic and computation, 22(1):79–99, 2012. [find](https://scholar.google.com/scholar?q=Vervaeke%2C+J%2E%2C+Lillicrap%2C+T%2E+P%2E%2C+and+Richards%2C+B%2E+A%2E+Relevance+realization+and+the+emerging+framework+in+cognitive+science%2E+Journal+of+logic+and+computation%2C+22%281%29%3A79%E2%80%9399%2C+2012%2E)
74. Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., and Zhou, M. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. arXiv preprint arXiv:2002.10957, 2020. [find](https://scholar.google.com/scholar?q=Wang%2C+W%2E%2C+Wei%2C+F%2E%2C+Dong%2C+L%2E%2C+Bao%2C+H%2E%2C+Yang%2C+N%2E%2C+and+Zhou%2C+M%2E+Minilm%3A+Deep+self%2Dattention+distillation+for+task%2Dagnostic+compression+of+pre%2Dtrained+transformers%2E+arXiv+preprint+arXiv%3A2002%2E10957%2C+2020%2E)
75. Wei, J. W. and Zou, K. EDA: easy data augmentation techniques for boosting performance on text classification tasks. arXiv preprint arXiv:1901.11196, 2019. [find](https://scholar.google.com/scholar?q=Wei%2C+J%2E+W%2E+and+Zou%2C+K%2E+EDA%3A+easy+data+augmentation+techniques+for+boosting+performance+on+text+classification+tasks%2E+arXiv+preprint+arXiv%3A1901%2E11196%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/wei-2019-eda-easy-data-augmentation-techniques/)
76. Williams, A., Nangia, N., and Bowman, S. A broad-coverage challenge corpus for sentence understanding through inference. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). ACL, 2018. [find](https://scholar.google.com/scholar?q=Williams%2C+A%2E%2C+Nangia%2C+N%2E%2C+and+Bowman%2C+S%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Conference+of+the+North+American+Chapter+of+the+Association+for+Computationa)
77. Wu, Z., Xiong, Y., Yu, S. X., and Lin, D. Unsupervised feature learning via non-parametric instance-level discrimination. In Computer Vision and Pattern Recognition (CVPR), 2018. [find](https://scholar.google.com/scholar?q=Wu%2C+Z%2E%2C+Xiong%2C+Y%2E%2C+Yu%2C+S%2E+X%2E%2C+and+Lin%2C+D%2E+Unsupervised+feature+learning+via+non%2Dparametric+instance%2Dlevel+discrimination%2E+In+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C+2018%2E)
78. Xiong, L., Xiong, C., Li, Y., Tang, K., Liu, J., Bennett, P. N., Ahmed, J., and Overwijk, A. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00808, 2020. [find](https://scholar.google.com/scholar?q=Xiong%2C+L%2E%2C+Xiong%2C+C%2E%2C+Li%2C+Y%2E%2C+Tang%2C+K%2E%2C+Liu%2C+J%2E%2C+Bennett%2C+P%2E+N%2E%2C+Ahmed%2C+J%2E%2C+and+Overwijk%2C+A%2E+Approximate+nearest+neighbor+negative+contrastive+learning+for+dense+text+retrieval%2E+arXiv+preprint+arXiv%3A2) [our card](https://md.research.elamur.ai/free/en/papers/xiong-2020-approximate-nearest-neighbor-negative-contrastive/)
79. Yih, W.-t., Toutanova, K., Platt, J. C., and Meek, C. Learning discriminative projections for text similarity measures. In Conference on Computational Natural Language Learning (CoNLL). ACL, 2011. [find](https://scholar.google.com/scholar?q=Yih%2C+W%2E%2Dt%2E%2C+Toutanova%2C+K%2E%2C+Platt%2C+J%2E+C%2E%2C+and+Meek%2C+C%2E+Learning+discriminative+projections+for+text+similarity+measures%2E+In+Conference+on+Computational+Natural+Language+Learning+%28CoNLL%29%2E+ACL%2C+2011%2E) [our card](https://md.research.elamur.ai/free/en/papers/yih-2011-learning-discriminative-projections-text-similarity/)
80. Zbontar, J., Jing, L., Misra, I., LeCun, Y., and Deny, S. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning (ICML), 2021. [find](https://scholar.google.com/scholar?q=Zbontar%2C+J%2E%2C+Jing%2C+L%2E%2C+Misra%2C+I%2E%2C+LeCun%2C+Y%2E%2C+and+Deny%2C+S%2E+Barlow+twins%3A+Self%2Dsupervised+learning+via+redundancy+reduction%2E+In+International+Conference+on+Machine+Learning+%28ICML%29%2C+2021%2E)
81. Zhang, Y., He, R., Liu, Z., Lim, K. H., and Bing, L. An unsupervised sentence embedding method by mutual information maximization. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. [find](https://scholar.google.com/scholar?q=Zhang%2C+Y%2E%2C+He%2C+R%2E%2C+Liu%2C+Z%2E%2C+Lim%2C+K%2E+H%2E%2C+and+Bing%2C+L%2E+An+unsupervised+sentence+embedding+method+by+mutual+information+maximization%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EM) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2020-unsupervised-sentence-embedding-method-mutual/)
82. Zhao, J., Wang, T., Yatskar, M., Ordonez, V., and Chang, K. Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv preprint arXiv:1804.06876, 2018. [find](https://scholar.google.com/scholar?q=Zhao%2C+J%2E%2C+Wang%2C+T%2E%2C+Yatskar%2C+M%2E%2C+Ordonez%2C+V%2E%2C+and+Chang%2C+K%2E+Gender+bias+in+coreference+resolution%3A+Evaluation+and+debiasing+methods%2E+arXiv+preprint+arXiv%3A1804%2E06876%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2018-gender-bias-coreference-resolution-evaluation/)