---
title: "Diffusion vs. Autoregressive Language Models: A Text"
description: "The authors propose DiffEmbed, which uses a diffusion language model as the embedding backbone. 46 references, cited by 8 papers here."
authors: ["Siyue Zhang", "Yilun Zhao", "Liyuan Geng", "Arman Cohan", "Anh Tuan Luu", "Chen Zhao"]
year: 2025
venue: "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2025.emnlp-main.213/"
doi: "10.18653/v1/2025.emnlp-main.213"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2025-diffusion-vs-autoregressive-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2025-diffusion-vs-autoregressive-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/8/) / Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective

# Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective

**Authors:** [Siyue Zhang](https://md.research.elamur.ai/free/en/authors/siyue-zhang/), [Yilun Zhao](https://md.research.elamur.ai/free/en/authors/yilun-zhao/), Liyuan Geng, [Arman Cohan](https://md.research.elamur.ai/free/en/authors/arman-cohan/), [Anh Tuan Luu](https://md.research.elamur.ai/free/en/authors/anh-tuan-luu/) and [Chen Zhao](https://md.research.elamur.ai/free/en/authors/chen-zhao/) · **Year:** 2025 · **Venue:** [Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2025](https://md.research.elamur.ai/free/en/venues/emnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.emnlp-main.213/) · [PDF](https://aclanthology.org/2025.emnlp-main.213.pdf) · [DOI 10.18653/v1/2025.emnlp-main.213](https://doi.org/10.18653/v1/2025.emnlp-main.213) · [OpenAlex](https://openalex.org/W4416036849)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [8](https://md.research.elamur.ai/free/en/papers/zhang-2025-diffusion-vs-autoregressive-language-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Embedding models built on decoder-only LLMs have become very strong. However, they inherit causal attention from pre-training: each token's representation ignores everything that follows it. This is a poor fit for embedding tasks, which benefit from seeing the whole text at once. The authors propose DiffEmbed, which uses a diffusion language model as the embedding backbone. Such models are pre-trained at scale with a masked-denoising objective and are bidirectional by design. DiffEmbed uses Dream-7B, mean-pools its final-layer token states, and fine-tunes with contrastive learning. The authors compare it against LLM-based embedders, including Llama3-8B with LLM2Vec, Mistral and Qwen2.5. Evaluation covers long-document, reasoning-heavy, instruction-following and general embedding benchmarks. Because no suitable training data existed for theorem-level retrieval, the authors also built ReasonAug, a dataset generated entirely with LLMs. DiffEmbed clearly wins on long and logically complex inputs and stays on par on standard tasks. This makes diffusion LMs a credible alternative backbone for retrieval and RAG systems, without the bidirectional adaptation that LLM embedders need.

## Key points
- Against the Llama3-8B-based embedder, DiffEmbed scores about 20% higher on LongEmbed and 8% higher on BRIGHT. It is about 2% better on FollowIR and competitive on MTEB.
- ReasonAug contains 10,896 LLM-generated query pairs, each with a logically related positive and an unrelated hard negative. Topics range from math and physics theorems to code. With this data, DiffEmbed beats the previous best on BRIGHT's TheoremQA tasks by 16.4%.
- The authors ablated reverse attention at test time, leaving only causal attention. DiffEmbed collapses on TheoQ., dropping from 48.3 to 0.7. Mistral falls less sharply, from 33.7 to 9.6. Leet. barely changes, so bidirectional context matters most for logical reasoning tasks.
- Fine-tuning data stayed below 20k samples per task:
  - 16k from public E5 data;
  - 10k from ReasonAug;
  - 16k from MS MARCO with instructions.
- Limitations: Dream was the only diffusion LM tested, for example not LLaDA. ReasonAug may still contain some noisy documents. The authors also report annotation noise in BRIGHT's Leet. and AoPS subsets.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) — University, New York, United States · 201 works · 100,180 citations
- [Nanyang Technological University](https://md.research.elamur.ai/free/en/universities/nanyang-technological-university/) — University, Singapore, Singapore · 156 works · 25,644 citations
- [Yale University](https://md.research.elamur.ai/free/en/universities/yale-university/) — University, New Haven, United States · 95 works · 12,765 citations
- [New York University Shanghai](https://md.research.elamur.ai/free/en/universities/new-york-university-shanghai/) — University, Shanghai, China · 8 works · 523 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Siyue Zhang, Yilun Zhao, Liyuan Geng, Arman Cohan, Anh Tuan Luu and Chen Zhao. 2025. Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2025diffusion,
  title = {{Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective}},
  author = {Siyue Zhang, Yilun Zhao, Liyuan Geng, Arman Cohan, Anh Tuan Luu and Chen Zhao},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.213},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.213}
}
```

- [← previous DeepSolution: Boosting Complex Engineering Solution Design via Tree-based Exploration and Bi-point Thinking](https://md.research.elamur.ai/free/en/papers/li-2025-deepsolution-boosting-complex-engineering-solution/)
- [next → Disentangling Language and Culture for Evaluating Multilingual Large Language Models](https://md.research.elamur.ai/free/en/papers/ying-2025-disentangling-language-culture-evaluating-multilingual/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/8/)

## References

46 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. 2021. [Structured denoising diffusion models in discrete state-spaces](https://md.research.elamur.ai/free/en/papers/austin-2021-structured-denoising-diffusion-models-discrete/). In Thirty-Fifth Annual Conference on Neural Information Processing Systems.
2. Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. [LLM2Vec: Large language models are secretly powerful text encoders](https://md.research.elamur.ai/free/en/papers/behnamghader-2024-llm2vec-large-language-models-secretly/). In First Conference on Language Modeling.
3. Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. [M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation](https://md.research.elamur.ai/free/en/papers/chen-2024-m3-embedding-multi-linguality/). In Findings of the Association for Computational Linguistics: ACL 2024.
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
5. Enjun Du, Siyi Liu, and Yongqi Zhang. 2025. Mixture of length and pruning experts for knowledge graphs reasoning. ArXiv preprint arXiv:2507.20498. [find](https://scholar.google.com/scholar?q=Enjun+Du%2C+Siyi+Liu%2C+and+Yongqi+Zhang%2E+2025%2E+Mixture+of+length+and+pruning+experts+for+knowledge+graphs+reasoning%2E+ArXiv+preprint+arXiv%3A2507%2E20498%2E)
6. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez,… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
7. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. [SimCSE: Simple contrastive learning of sentence embeddings](https://md.research.elamur.ai/free/en/papers/gao-2021-simcse-simple-contrastive-learning-sentence/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
8. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. [The curious case of neural text degeneration](https://md.research.elamur.ai/free/en/papers/holtzman-2019-curious-case-neural-text-degeneration/). In The Eighth International Conference on Learning Representations.
9. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. [Unsupervised dense information retrieval with contrastive learning](https://md.research.elamur.ai/free/en/papers/izacard-2021-unsupervised-dense-information-retrieval-contrastive/). Transactions on Machine Learning Research.
10. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
11. Jina. 2024. Jina reranker v2 for agentic rag: Ultra-fast, multilingual, function-calling and code search. [https://jina.ai/news/jina-reranker-v2-for](https://jina.ai/news/jina-reranker-v2-for) -agentic-rag-ultra-fast-multilingual-fun ction-calling-and-code-search/. Accessed: 2024-11-26. [↗](https://jina.ai/news/jina-reranker-v2-for)
12. Greg Kamradt. 2024. Llm test - needle in a haystack. [https://github.com/gkamradt/LLMTest_Need](https://github.com/gkamradt/LLMTest_Need) leInAHaystack. Accessed: 2025-05-19. [↗](https://github.com/gkamradt/LLMTest_Need)
13. Rabeeh Karimi Mahabadi, Hamish Ivison, Jaesung Tae, James Henderson, Iz Beltagy, Matthew Peters, and Arman Cohan. 2024. TESS: Text-to-text self-conditioned simplex diffusion. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Lon… [find](https://scholar.google.com/scholar?q=Rabeeh+Karimi+Mahabadi%2C+Hamish+Ivison%2C+Jaesung+Tae%2C+James+Henderson%2C+Iz+Beltagy%2C+Matthew+Peters%2C+and+Arman+Cohan%2E+2024%2E+TESS%3A+Text%2Dto%2Dtext+self%2Dconditioned+simplex+diffusion%2E+In+Proceedings+of+the+18t)
14. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. [Dense passage retrieval for open-domain question answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
15. Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. [Nv-embed: Improved techniques for training llms as generalist embedding models](https://md.research.elamur.ai/free/en/papers/lee-2024-nv-embed-improved-techniques-training/).
16. Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Yaniv+Leviathan%2C+Matan+Kalman%2C+and+Yossi+Matias%2E+2023%2E+Fast+inference+from+transformers+via+speculative+decoding%2E+In+Proceedings+of+the+40th+International+Conference+on+Machine+Learning%2E)
17. Chaofan Li, Zheng Liu, Shitao Xiao, and Yingxia Shao. 2023. Making large language models a better foundation for dense retrieval. [find](https://scholar.google.com/scholar?q=Chaofan+Li%2C+Zheng+Liu%2C+Shitao+Xiao%2C+and+Yingxia+Shao%2E+2023%2E+Making+large+language+models+a+better+foundation+for+dense+retrieval%2E)
18. Aaron Lou, Chenlin Meng, and Stefano Ermon. 2024. Discrete diffusion language modeling by estimating the ratios of the data distribution. [find](https://scholar.google.com/scholar?q=Aaron+Lou%2C+Chenlin+Meng%2C+and+Stefano+Ermon%2E+2024%2E+Discrete+diffusion+language+modeling+by+estimating+the+ratios+of+the+data+distribution%2E)
19. Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2024. [Fine-tuning llama for multi-stage text retrieval](https://md.research.elamur.ai/free/en/papers/ma-2023-fine-tuning-llama-multi-stage/). In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval.
20. Sean MacAvaney, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto, Nazli Goharian, and Ophir Frieder. 2020. Efficient document re-ranking for transformers by precomputing term representations. In Proceedings of the 43rd International ACM SI-GIR Conference on Research and Development in Inform… [find](https://scholar.google.com/scholar?q=Sean+MacAvaney%2C+Franco+Maria+Nardini%2C+Raffaele+Perego%2C+Nicola+Tonellotto%2C+Nazli+Goharian%2C+and+Ophir+Frieder%2E+2020%2E+Efficient+document+re%2Dranking+for+transformers+by+precomputing+term+representations%2E+)
21. Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. 2025. [Generative representational instruction tuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2024-generative-representational-instruction-tuning/). In The Thirteenth International Conference on Learning Representations.
22. Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cull… [find](https://scholar.google.com/scholar?q=Jake+McNeil%2C+David+Medina%2C+Aalok+Mehta%2C+Jacob+Menick%2C+Luke+Metz%2C+Andrey+Mishchenko%2C+Pamela+Mishkin%2C+Vinnie+Monaco%2C+Evan+Morikawa%2C+Daniel+Mossing%2C+Tong+Mu%2C+Mira+Murati%2C+Oleg+Murk%2C+David+M%C3%A9ly%2C+Ashvin+Na)
23. Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. [MTEB: Massive text embedding benchmark](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-mteb-massive-text-embedding-benchmark/). In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics.
24. Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. 2025. [Large language diffusion models](https://md.research.elamur.ai/free/en/papers/nie-2025-large-language-diffusion-models/).
25. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… [find](https://scholar.google.com/scholar?q=OpenAI%2C+Josh+Achiam%2C+Steven+Adler%2C+Sandhini+Agarwal%2C+Lama+Ahmad%2C+Ilge+Akkaya%2C+Florencia+Leoni+Aleman%2C+Diogo+Almeida%2C+Janko+Altenschmidt%2C+Sam+Altman%2C+Shyamal+Anadkat%2C+Red+Avila%2C+Igor+Babuschkin%2C+Suchir)
26. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research.
27. Subham Sekhar Sahoo, Marianne Arriola, Aaron Gokaslan, Edgar Mariano Marroquin, Alexander M Rush, Yair Schiff, Justin T Chiu, and Volodymyr Kuleshov. 2024. [Simple and effective masked diffusion language models](https://md.research.elamur.ai/free/en/papers/sahoo-2024-simple-effective-masked-diffusion-language/).
28. Rulin Shao, Rui Qiao, Varsha Kishore, Niklas Muennighoff, Xi Victoria Lin, Daniela Rus, Bryan Kian Hsiang Low, Sewon Min, Wen tau Yih, Pang Wei Koh, and Luke Zettlemoyer. 2025. Reasonir: Training retrievers for reasoning tasks. [find](https://scholar.google.com/scholar?q=Rulin+Shao%2C+Rui+Qiao%2C+Varsha+Kishore%2C+Niklas+Muennighoff%2C+Xi+Victoria+Lin%2C+Daniela+Rus%2C+Bryan+Kian+Hsiang+Low%2C+Sewon+Min%2C+Wen+tau+Yih%2C+Pang+Wei+Koh%2C+and+Luke+Zettlemoyer%2E+2025%2E+Reasonir%3A+Training+retr)
29. Zhang Siyue, Xue Yuxiang, Zhang Yiming, Wu Xiaobao, Luu Anh Tuan, and Zhao Chen. 2024. Mrag: A modular retrieval framework for time-sensitive question answering. [find](https://scholar.google.com/scholar?q=Zhang+Siyue%2C+Xue+Yuxiang%2C+Zhang+Yiming%2C+Wu+Xiaobao%2C+Luu+Anh+Tuan%2C+and+Zhao+Chen%2E+2024%2E+Mrag%3A+A+modular+retrieval+framework+for+time%2Dsensitive+question+answering%2E)
30. Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. 2015. [Deep unsupervised learning using nonequilibrium thermodynamics](https://md.research.elamur.ai/free/en/papers/sohl-dickstein-2015-deep-unsupervised-learning-using-nonequilibrium/). In Proceedings of the 32nd International Conference on Machine Learning.
31. Tingyu Song, Guo Gan, Mingsheng Shang, and Yilun Zhao. 2025. Ifir: A comprehensive benchmark for evaluating instruction-following in expert-domain information retrieval. In Proceedings of 2025 Annual Conference of the North American Chapter of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tingyu+Song%2C+Guo+Gan%2C+Mingsheng+Shang%2C+and+Yilun+Zhao%2E+2025%2E+Ifir%3A+A+comprehensive+benchmark+for+evaluating+instruction%2Dfollowing+in+expert%2Ddomain+information+retrieval%2E+In+Proceedings+of+2025+Annual+)
32. Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 2021. [Score-based generative modeling through stochastic differential equations](https://md.research.elamur.ai/free/en/papers/song-2020-score-based-generative-modeling-through/). In International Conference on Learning Representations.
33. Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, and Aditi Raghunathan. 2025. Repetition improves language model embeddings. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Jacob+Mitchell+Springer%2C+Suhas+Kotha%2C+Daniel+Fried%2C+Graham+Neubig%2C+and+Aditi+Raghunathan%2E+2025%2E+Repetition+improves+language+model+embeddings%2E+In+The+Thirteenth+International+Conference+on+Learning+Re)
34. Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han yu Wang, Haisu Liu, Quan Shi, Zachary S. Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan O. Arik, Danqi Chen, and Tao Yu. 2025. [Bright: A realistic and challenging benchmark for reasoning-intensive retrieval](https://md.research.elamur.ai/free/en/papers/su-2024-bright-realistic-challenging-benchmark-reasoning/). In The Thir…
35. Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. 2021. [Augmented SBERT: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks](https://md.research.elamur.ai/free/en/papers/thakur-2021-augmented-sbert-data-augmentation-method/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics:…
36. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2024a. [Text embeddings by weakly-supervised contrastive pre-training](https://md.research.elamur.ai/free/en/papers/wang-2022-text-embeddings-weakly-supervised-contrastive/).
37. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024b. [Improving text embeddings with large language models](https://md.research.elamur.ai/free/en/papers/wang-2024-improving-text-embeddings-large-language/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics.
38. Orion Weller, Benjamin Chang, Sean MacAvaney, Kyle Lo, Arman Cohan, Benjamin Van Durme, Dawn Lawrie, and Luca Soldaini. 2024. [Followir: Evaluating and teaching information retrieval models to follow instructions](https://md.research.elamur.ai/free/en/papers/weller-2025-followir-evaluating-teaching-information-retrieval/).
39. Orion Weller, Benjamin Van Durme, Dawn Lawrie, Ashwin Paranjape, Yuhao Zhang, and Jack Hessel. 2025. Promptriever: Instruction-trained retrievers can be prompted like language models. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Orion+Weller%2C+Benjamin+Van+Durme%2C+Dawn+Lawrie%2C+Ashwin+Paranjape%2C+Yuhao+Zhang%2C+and+Jack+Hessel%2E+2025%2E+Promptriever%3A+Instruction%2Dtrained+retrievers+can+be+prompted+like+language+models%2E+In+The+Thirteent)
40. Tong Wu, Zhihao Fan, Xiao Liu, Yeyun Gong, Yelong Shen, Jian Jiao, Hai-Tao Zheng, Juntao Li, Zhongyu Wei, Jian Guo, Nan Duan, and Weizhu Chen. 2023. Ar-diffusion: Auto-regressive diffusion model for text generation. In Thirty-Seventh Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Tong+Wu%2C+Zhihao+Fan%2C+Xiao+Liu%2C+Yeyun+Gong%2C+Yelong+Shen%2C+Jian+Jiao%2C+Hai%2DTao+Zheng%2C+Juntao+Li%2C+Zhongyu+Wei%2C+Jian+Guo%2C+Nan+Duan%2C+and+Weizhu+Chen%2E+2023%2E+Ar%2Ddiffusion%3A+Auto%2Dregressive+diffusion+model+for+t)
41. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Juny… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chang+Zhou%2C+Chengpeng+Li%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Guanting+Dong%2C+Haoran+Wei%2C+Huan+Lin%2C+Jialong+Tang%2C+Jialin+Wang%2C+Jian+Yang%2C+Jian)
42. Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. 2025. Dream 7b: Introducing dream 7b, the most powerful open diffusion large language model to date. [find](https://scholar.google.com/scholar?q=Jiacheng+Ye%2C+Zhihui+Xie%2C+Lin+Zheng%2C+Jiahui+Gao%2C+Zirui+Wu%2C+Xin+Jiang%2C+Zhenguo+Li%2C+and+Lingpeng+Kong%2E+2025%2E+Dream+7b%3A+Introducing+dream+7b%2C+the+most+powerful+open+diffusion+large+language+model+to+date%2E)
43. Honghua Zhang, Meihua Dang, Nanyun Peng, and Guy Van Den Broeck. 2023. Tractable control for autoregressive language generation. In Proceedings of the 40th International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Honghua+Zhang%2C+Meihua+Dang%2C+Nanyun+Peng%2C+and+Guy+Van+Den+Broeck%2E+2023%2E+Tractable+control+for+autoregressive+language+generation%2E+In+Proceedings+of+the+40th+International+Conference+on+Machine+Learning)
44. Dawei Zhu, Liang Wang, Nan Yang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. 2024. [LongEmbed: Extending embedding models for long context retrieval](https://md.research.elamur.ai/free/en/papers/zhu-2024-longembed-extending-embedding-models-long/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.
45. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. 2024. [Smarter, better, faster, longer: A modern bidirectional encoder for fast,…](https://md.research.elamur.ai/free/en/papers/warner-2024-smarter-better-faster-longer-modern/)
46. Task ArguAna Banking77Classification BiorxivClusteringS2S EmotionClassification MassiveIntentClassification MedrxivClusteringS2S NFCorpus SciDocsRR SciFact SICK-R SprintDuplicateQuestions StackOverflowDupQuestions STS17 STSBenchmark TwentyNewsgroupsClustering Average Llama3 57.33 87.35 38.45 47.81 7… [find](https://scholar.google.com/scholar?q=Task+ArguAna+Banking77Classification+BiorxivClusteringS2S+EmotionClassification+MassiveIntentClassification+MedrxivClusteringS2S+NFCorpus+SciDocsRR+SciFact+SICK%2DR+SprintDuplicateQuestions+StackOverflo)

*Updated 1 November 2026*