---
title: "Zero-Shot Listwise Document Reranking with a Large Language"
description: "They propose LRL (Listwise Reranker with a Large Language Model). 28 references, cited by 84 papers here."
authors: ["Xueguang Ma", "Xinyu Zhang", "Ronak Pradeep", "Jimmy Lin"]
year: 2023
venue: "arXiv:2305.02156"
source: "https://arxiv.org/abs/2305.02156"
doi: "10.48550/arxiv.2305.02156"
arxiv: "2305.02156"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ma-2023-zero-shot-listwise-document-reranking/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ma-2023-zero-shot-listwise-document-reranking/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / Zero-Shot Listwise Document Reranking with a Large Language Model

# Zero-Shot Listwise Document Reranking with a Large Language Model

**Authors:** [Xueguang Ma](https://md.research.elamur.ai/free/en/authors/xueguang-ma/), [Xinyu Zhang](https://md.research.elamur.ai/free/en/authors/xinyu-zhang/), [Ronak Pradeep](https://md.research.elamur.ai/free/en/authors/ronak-pradeep/) and [Jimmy Lin](https://md.research.elamur.ai/free/en/authors/jimmy-lin/) · **Year:** 2023 · **Venue:** arXiv:2305.02156 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2305.02156) · [PDF](https://arxiv.org/pdf/2305.02156) · [DOI 10.48550/arxiv.2305.02156](https://doi.org/10.48550/arxiv.2305.02156) · [arXiv 2305.02156](https://arxiv.org/abs/2305.02156) · [OpenAlex](https://openalex.org/W4368755400)

**Citations:** 21 citations (OpenAlex, 2026-10-14) · [84](https://md.research.elamur.ai/free/en/papers/ma-2023-zero-shot-listwise-document-reranking/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

Strong neural rerankers usually depend on large collections of human relevance labels such as MS MARCO. The authors ask whether a competitive ranking pipeline can be built without any task-specific training. They propose LRL (Listwise Reranker with a Large Language Model). It prompts GPT-3 with a query and up to ten candidate passages and asks the model to output the passage identifiers sorted by relevance. Because the model sees all candidates at once, it can judge them relative to each other instead of scoring each one in isolation. For candidate lists longer than the model's context allows, a sliding-window procedure reranks overlapping chunks from the tail of the list toward the head. On TREC DL19, DL20 and DL21, LRL beats the zero-shot pointwise rerankers UPR and PRL. It also improves results when applied as a final stage on top of them. On three MIRACL languages, it clearly outperforms lexical and hybrid first-stage retrieval. The work shows that LLMs can act directly as label-free rerankers, although contamination of training data cannot be ruled out.

## Key points

- **Models and prompting:** LRL uses text-davinci-003 on TREC DL and code-davinci-002 on MIRACL; the latter was chosen for its longer input limit, since GPT-3 over-tokenizes many non-English languages.
  - The prompt leaves the output list's bracket open so the model fills in ranked IDs.
  - PRL is the authors' pointwise baseline: it uses the probability of the token "True" as the relevance score.
- **Reranking BM25 top-100 (nDCG@10):** LRL averages about 6 points above UPR, the previous zero-shot SoTA, and about 3 points above PRL. For example, it reaches 0.6580 on DL19, against 0.5910 for UPR.
- **As a final stage:** reranking the top-20 PRL results with LRL adds roughly 5 points nDCG@10 and 7 points MRR@10 on average. It reaches 0.6650 on DL19 and 0.6260 on DL21.
- **Against supervised systems:** LRL surpasses the supervised DPR retriever but still trails stronger pipelines such as TCT_ColBERT and MonoBERT.
- **MIRACL (Chinese, Swahili, Yoruba):** reranking the top-20 results of a BM25 + mContriever hybrid gives an average gain of about 15 points nDCG@10 over BM25. The authors tested only these three languages because of API cost.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Waterloo](https://md.research.elamur.ai/free/en/universities/university-of-waterloo/) — University, Waterloo, Canada · 80 works · 15,030 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 552 works

## Cite

Cite as: Xueguang Ma, Xinyu Zhang, Ronak Pradeep and Jimmy Lin. 2023. Zero-Shot Listwise Document Reranking with a Large Language Model. arXiv:2305.02156.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ma2023zero,
  title = {{Zero-Shot Listwise Document Reranking with a Large Language Model}},
  author = {Xueguang Ma, Xinyu Zhang, Ronak Pradeep and Jimmy Lin},
  year = {2023},
  doi = {10.48550/arxiv.2305.02156},
  eprint = {2305.02156},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.02156}
}
```

- [← previous Studying Large Language Model Generalization with Influence Functions](https://md.research.elamur.ai/free/en/papers/grosse-2023-studying-large-language-model-generalization/)
- [next → A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity](https://md.research.elamur.ai/free/en/papers/longpre-2023-pretrainer-s-guide-training-data/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

28 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2016. [MS MARCO: A human generated machine reading comprehension dataset](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/). arXiv:arXiv:1611…
2. Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. 2022. InPars: Unsupervised dataset generation for information retrieval. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’22, page 2387–2392. [find](https://scholar.google.com/scholar?q=Luiz+Bonifacio%2C+Hugo+Abonizio%2C+Marzieh+Fadaee%2C+and+Rodrigo+Nogueira%2E+2022%2E+InPars%3A+Unsupervised+dataset+generation+for+information+retrieval%2E+In+Proceedings+of+the+45th+International+ACM+SIGIR+Confere)
3. Luiz Bonifacio, Vitor Jeronymo, Hugo Queiroz Abonizio, Israel Campiotti, Marzieh Fadaee, Roberto Lotufo, and Rodrigo Nogueira. 2021. mMARCO: A multilingual version of the MS MARCO passage ranking dataset. arXiv:2108.13897. — [arXiv:2108.13897](https://arxiv.org/abs/2108.13897) [find](https://scholar.google.com/scholar?q=Luiz+Bonifacio%2C+Vitor+Jeronymo%2C+Hugo+Queiroz+Abonizio%2C+Israel+Campiotti%2C+Marzieh+Fadaee%2C+Roberto+Lotufo%2C+and+Rodrigo+Nogueira%2E+2021%2E+mMARCO%3A+A+multilingual+version+of+the+MS+MARCO+passage+ranking+data)
4. Leonid Boytsov, Preksha Patel, Vivek Sourabh, Riddhi Nisar, Sayani Kundu, Ramya Ramanathan, and Eric Nyberg. 2023. InPars-Light: Cost-effective unsupervised training of efficient rankers. arXiv:2301.02998. — [arXiv:2301.02998](https://arxiv.org/abs/2301.02998) [find](https://scholar.google.com/scholar?q=Leonid+Boytsov%2C+Preksha+Patel%2C+Vivek+Sourabh%2C+Riddhi+Nisar%2C+Sayani+Kundu%2C+Ramya+Ramanathan%2C+and+Eric+Nyberg%2E+2023%2E+InPars%2DLight%3A+Cost%2Deffective+unsupervised+training+of+efficient+rankers%2E+arXiv%3A2301%2E0)
5. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
6. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. [Reading Wikipedia to answer open-domain questions](https://md.research.elamur.ai/free/en/papers/chen-2017-reading-wikipedia-answer-open-domain/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870– 1879, Vancouver, Canada.
7. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2021. Overview of the TREC 2020 deep learning track. arXiv:2102.07662. — [arXiv:2102.07662](https://arxiv.org/abs/2102.07662) [find](https://scholar.google.com/scholar?q=Nick+Craswell%2C+Bhaskar+Mitra%2C+Emine+Yilmaz%2C+and+Daniel+Campos%2E+2021%2E+Overview+of+the+TREC+2020+deep+learning+track%2E+arXiv%3A2102%2E07662%2E+%E2%80%94+%5BarXiv%3A2102%2E07662%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2102%2E07662%29)
8. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Jimmy Lin. 2022. Overview of the TREC 2021 deep learning track. In Proceedings of the Thirtieth Text REtrieval Conference (TREC 2021). [find](https://scholar.google.com/scholar?q=Nick+Craswell%2C+Bhaskar+Mitra%2C+Emine+Yilmaz%2C+Daniel+Campos%2C+and+Jimmy+Lin%2E+2022%2E+Overview+of+the+TREC+2021+deep+learning+track%2E+In+Proceedings+of+the+Thirtieth+Text+REtrieval+Conference+%28TREC+2021%29%2E)
9. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. 2020. [Overview of the TREC 2019 deep learning track](https://md.research.elamur.ai/free/en/papers/craswell-2020-overview-trec-2019-deep-learning/). arXiv:2003.07820.
10. Zhuyun Dai, Vincent Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, and Ming-Wei Chang. 2022. [Promptagator: Few-shot dense retrieval from 8 examples](https://md.research.elamur.ai/free/en/papers/dai-2022-promptagator-few-shot-dense-retrieval/). arXiv:2209.11755.
11. Luyu Gao, Zhuyun Dai, and Jamie Callan. 2021. Rethink training of BERT rerankers in multi-stage retrieval pipeline. In The 43rd European Conference on Information Retrieval (ECIR). [find](https://scholar.google.com/scholar?q=Luyu+Gao%2C+Zhuyun+Dai%2C+and+Jamie+Callan%2E+2021%2E+Rethink+training+of+BERT+rerankers+in+multi%2Dstage+retrieval+pipeline%2E+In+The+43rd+European+Conference+on+Information+Retrieval+%28ECIR%29%2E)
12. Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2022. [Precise zero-shot dense retrieval without relevance labels](https://md.research.elamur.ai/free/en/papers/gao-2022-precise-zero-shot-dense-retrieval/). arXiv:2212.10496.
13. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. [Unsupervised dense information retrieval with contrastive learning](https://md.research.elamur.ai/free/en/papers/izacard-2021-unsupervised-dense-information-retrieval-contrastive/). arXiv:2112.09118.
14. Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. 2021. In-batch negatives for knowledge distillation with tightly-coupled teachers for dense retrieval. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021), pages 163– 173. [find](https://scholar.google.com/scholar?q=Sheng%2DChieh+Lin%2C+Jheng%2DHong+Yang%2C+and+Jimmy+Lin%2E+2021%2E+In%2Dbatch+negatives+for+knowledge+distillation+with+tightly%2Dcoupled+teachers+for+dense+retrieval%2E+In+Proceedings+of+the+6th+Workshop+on+Representa)
15. Rodrigo Nogueira and Kyunghyun Cho. 2019. [Passage re-ranking with BERT](https://md.research.elamur.ai/free/en/papers/nogueira-2019-passage-re-ranking-bert/). arXiv:1901.04085.
16. Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with a pre-trained sequence-to-sequence model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 708–718. [find](https://scholar.google.com/scholar?q=Rodrigo+Nogueira%2C+Zhiying+Jiang%2C+Ronak+Pradeep%2C+and+Jimmy+Lin%2E+2020%2E+Document+ranking+with+a+pre%2Dtrained+sequence%2Dto%2Dsequence+model%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP)
17. Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019. Multi-stage document ranking with BERT. arXiv:1910.14424. [find](https://scholar.google.com/scholar?q=Rodrigo+Nogueira%2C+Wei+Yang%2C+Kyunghyun+Cho%2C+and+Jimmy+Lin%2E+2019%2E+Multi%2Dstage+document+ranking+with+BERT%2E+arXiv%3A1910%2E14424%2E)
18. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Low… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
19. Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. [KILT: a benchmark for knowledge intensive language tasks](https://md.research.elamur.ai/free/en/papers/petroni-2020-kilt-benchmark-knowledge-intensive-language/). In Proceedings of th…
20. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21(140):1–67.
21. Stephen E. Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389. [find](https://scholar.google.com/scholar?q=Stephen+E%2E+Robertson+and+Hugo+Zaragoza%2E+2009%2E+The+probabilistic+relevance+framework%3A+BM25+and+beyond%2E+Foundations+and+Trends+in+Information+Retrieval%2C+3%284%29%3A333%E2%80%93389%2E)
22. Devendra Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. 2022. [Improving passage retrieval with zero-shot question generation](https://md.research.elamur.ai/free/en/papers/sachan-2022-improving-passage-retrieval-zero-shot/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3781–3797, Abu Dhabi,…
23. Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan C… — [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
24. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). [find](https://scholar.google.com/scholar?q=Nandan+Thakur%2C+Nils+Reimers%2C+Andreas+R%C3%BCckl%C3%A9%2C+Abhishek+Srivastava%2C+and+Iryna+Gurevych%2E+2021%2E+BEIR%3A+A+heterogeneous+benchmark+for+zero%2Dshot+evaluation+of+information+retrieval+models%2E+In+Thirty%2Dfifth+Co)
25. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). In The Tenth International Conference on Learning Representations.
26. Xinyu Zhang, Kelechi Ogueji, Xueguang Ma, and Jimmy Lin. 2022a. Towards best practices for training multilingual dense retrieval models. arXiv:2204.02363. — [arXiv:2204.02363](https://arxiv.org/abs/2204.02363) [find](https://scholar.google.com/scholar?q=Xinyu+Zhang%2C+Kelechi+Ogueji%2C+Xueguang+Ma%2C+and+Jimmy+Lin%2E+2022a%2E+Towards+best+practices+for+training+multilingual+dense+retrieval+models%2E+arXiv%3A2204%2E02363%2E+%E2%80%94+%5BarXiv%3A2204%2E02363%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F22)
27. Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. 2022b. Making a MIRACL: Multilingual information retrieval across a continuum of languages. arXiv:2210.09984. — [arXiv:2210.09984](https://arxiv.org/abs/2210.09984) [find](https://scholar.google.com/scholar?q=Xinyu+Zhang%2C+Nandan+Thakur%2C+Odunayo+Ogundepo%2C+Ehsan+Kamalloo%2C+David+Alfonso%2DHermelo%2C+Xiaoguang+Li%2C+Qun+Liu%2C+Mehdi+Rezagholizadeh%2C+and+Jimmy+Lin%2E+2022b%2E+Making+a+MIRACL%3A+Multilingual+information+retrie)
28. Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni, Xuanhui Wang, and Michael Bendersky. 2022. RankT5: Fine-tuning T5 for text ranking with ranking losses. arXiv:2210.10634. — [arXiv:2210.10634](https://arxiv.org/abs/2210.10634) [find](https://scholar.google.com/scholar?q=Honglei+Zhuang%2C+Zhen+Qin%2C+Rolf+Jagerman%2C+Kai+Hui%2C+Ji+Ma%2C+Jing+Lu%2C+Jianmo+Ni%2C+Xuanhui+Wang%2C+and+Michael+Bendersky%2E+2022%2E+RankT5%3A+Fine%2Dtuning+T5+for+text+ranking+with+ranking+losses%2E+arXiv%3A2210%2E10634%2E+%E2%80%94)

*Updated 2 November 2026*