---
title: "Large Language Models are Zero-Shot Rankers for"
description: "The authors ask how well a general-purpose LLM can rank recommendation candidates without… 16 references, cited by 85 papers here."
authors: ["Yupeng Hou", "Junjie Zhang", "Zihan Lin", "Hongyu Lu", "Ruobing Xie", "Julian McAuley", "Wayne Xin Zhao"]
year: 2023
venue: "arXiv:2305.08845"
source: "https://arxiv.org/abs/2305.08845"
doi: "10.48550/arxiv.2305.08845"
arxiv: "2305.08845"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hou-2023-large-language-models-zero-shot/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hou-2023-large-language-models-zero-shot/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/2/) / Large Language Models are Zero-Shot Rankers for Recommender Systems

# Large Language Models are Zero-Shot Rankers for Recommender Systems

**Authors:** [Yupeng Hou](https://md.research.elamur.ai/free/en/authors/yupeng-hou/), [Junjie Zhang](https://md.research.elamur.ai/free/en/authors/junjie-zhang/), Zihan Lin, [Hongyu Lu](https://md.research.elamur.ai/free/en/authors/hongyu-lu/), [Ruobing Xie](https://md.research.elamur.ai/free/en/authors/ruobing-xie/), [Julian McAuley](https://md.research.elamur.ai/free/en/authors/julian-mcauley/) and [Wayne Xin Zhao](https://md.research.elamur.ai/free/en/authors/wayne-xin-zhao/) · **Year:** 2023 · **Venue:** [arXiv:2305.08845](https://md.research.elamur.ai/free/en/venues/lecture-notes-in-computer-science/) ([Lecture notes in computer science 2023](https://md.research.elamur.ai/free/en/venues/lecture-notes-in-computer-science/2023/)) · **Publisher:** [Springer Science+Business Media](https://md.research.elamur.ai/free/en/publishers/springer-science-business-media/) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2305.08845) · [PDF](https://arxiv.org/pdf/2305.08845) · [DOI 10.48550/arxiv.2305.08845](https://doi.org/10.48550/arxiv.2305.08845) · [arXiv 2305.08845](https://arxiv.org/abs/2305.08845) · [OpenAlex](https://openalex.org/W4392846385)

**Citations:** 309 citations (OpenAlex, 2026-10-14) · [85](https://md.research.elamur.ai/free/en/papers/hou-2023-large-language-models-zero-shot/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The authors ask how well a general-purpose LLM can rank recommendation candidates without task-specific training, and what helps or hurts that ability. They treat recommendation as conditional ranking. A user's chronologically ordered history serves as the condition, and a small pool of items from candidate generators is the set to reorder. Both parts are written into a natural-language instruction for ChatGPT (gpt-3.5-turbo), and the answer is mapped back to item titles by string matching. Experiments on MovieLens-1M and the Amazon Games category use only item titles as text.

The study finds three weaknesses. The LLM largely ignores the order of past interactions. It favours popular items. Its rankings shift depending on where a candidate appears in the prompt. Two fixes help: prompts that highlight the most recent item or add self-derived in-context examples, and bootstrapping, which reranks shuffled candidate lists several times. With these, the zero-shot ranker beats earlier zero-shot recommenders. On candidates pooled from several generators, it also outperforms trained baselines such as Pop and BPRMF. This makes LLMs credible rankers for settings where little in-domain training data exists.

## Key points
- **Datasets:**
  - ML-1M: 6,040 users, 3,706 items, about 1M interactions.
  - Games: 50,547 users, 16,859 items, 389,718 interactions.
  - Any user or item with under five interactions was filtered out.
- **Evaluation setup:**
  - Leave-one-out protocol scored with NDCG@K, implemented in RecBole.
  - Histories capped at 50 items; usually 20 candidates (m = 20).
  - Temperature 0.2; 6,000 sampled users for Games.
- **Three history encodings were tested:**
  - Plain sequential prompting.
  - Recency-focused prompting.
  - In-context learning built from prefixes of the user's own sequence.
- **Best configuration:** recency-focused prompting with 5 history items plus 3 bootstrapping rounds. On 21-item multi-generator pools, it was second-best on most metrics.
- **Practical notes:**
  - GPT-3.5 named items outside the candidate set in roughly 3% of cases.
  - Open models such as LLaMA-2-70B-Chat clearly lagged behind ChatGPT.
  - Code and processed data are available at RUCAIBox/LLMRank.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tencent](https://md.research.elamur.ai/free/en/companies/tencent/) — Company, Shenzhen, China · 163 works · 21,032 citations
- [Renmin University of China](https://md.research.elamur.ai/free/en/universities/renmin-university-of-china/) — University, Beijing, China · 139 works · 17,752 citations
- [University of California San Diego](https://md.research.elamur.ai/free/en/universities/university-of-california-san-diego/) — University, San Diego, United States · 89 works · 54,032 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley and Wayne Xin Zhao. 2023. Large Language Models are Zero-Shot Rankers for Recommender Systems. arXiv:2305.08845.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{hou2023large,
  title = {{Large Language Models are Zero-Shot Rankers for Recommender Systems}},
  author = {Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley and Wayne Xin Zhao},
  year = {2023},
  doi = {10.48550/arxiv.2305.08845},
  eprint = {2305.08845},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.08845}
}
```

- [← previous Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://md.research.elamur.ai/free/en/papers/inan-2023-llama-guard-llm-based-input/)
- [next → Decomposed Prompting: A Modular Approach for Solving Complex Tasks](https://md.research.elamur.ai/free/en/papers/khot-2022-decomposed-prompting-modular-approach/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/2/)

## References

16 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 1. Bao, K., Zhang, J., Zhang, Y., Wang, W., Feng, F., He, X.: [Tallrec: An effective and efficient tuning framework to align large language model with recommendation](https://md.research.elamur.ai/free/en/papers/bao-2023-tallrec-effective-efficient-tuning-framework/). arXiv preprint arXiv:2305.00447 (2023) 2. Barkan, O., Koenigstein, N.: ITEM2VEC: neural item embedding for collaborative filtering. In…
2. IEEE (2016). https://doi.org/10.1109/MLSP.2016.7738886, https://doi.org/10.1109/ MLSP.2016.7738886 3. Bonab, H.R., Aliannejadi, M., Vardasbi, A., Kanoulas, E., Allan, J.: Cross-market product recommendation. In: Demartini, G., Zuccon, G., Culpepper, J.S., Huang, Z., Tong, H. (eds.) CIKM. pp. 110–119… — [doi:10.1109/mlsp.2016.7738886](https://doi.org/10.1109/mlsp.2016.7738886) [↗](https://doi.org/10.1109/MLSP.2016.7738886)
3. ACM (2021). https://doi.org/10.1145/3459637.3482493, https://doi.org/10.1145/3459637. 3482493 4. Cao, D., He, X., Nie, L., Wei, X., Hu, X., Wu, S., Chua, T.: Cross-platform app recommendation by jointly modeling ratings and texts. ACM Trans. Inf. Syst. 35(4), 37:1–37:27 (2017). https://doi.org/10.11… — [doi:10.1145/3459637.3482493](https://doi.org/10.1145/3459637.3482493) [↗](https://doi.org/10.1145/3459637.3482493)
4. IEEE (2021). https://doi.org/10.1109/ICDM51629.2021.00015, https://doi.org/ 10.1109/ICDM51629.2021.00015 9. Chiang, W.L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J.E., et al.: Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt qualit… — [doi:10.1109/icdm51629.2021.00015](https://doi.org/10.1109/icdm51629.2021.00015) [↗](https://doi.org/10.1109/ICDM51629.2021.00015)
5. CEUR-WS.org (2020), [https://ceur-ws.org/Vol-2715/paper6.pdf](https://ceur-ws.org/Vol-2715/paper6.pdf) 12. Cui, Z., Ma, J., Zhou, C., Zhou, J., Yang, H.: M6-rec: Generative pretrained language models are open-ended recommender systems. arXiv preprint arXiv:2205.08084 (2022) 13. Dai, S., Shao, N., Zhao, H., Yu, W., Si, Z., Xu, C., Sun, Z., Zh… — [arXiv:2205.08084](https://arxiv.org/abs/2205.08084) [↗](https://ceur-ws.org/Vol-2715/paper6.pdf)
6. Large Language Models are Zero-Shot Rankers for Recommender Systems 14. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In: NAACL (2019) 15. Ding, H., Ma, Y., Deoras, A., Wang, Y., Wang, H.: Zero-shot recommender syst…
7. ACM (2018). https://doi.org/10.1145/3219819.3219885, https://doi.org/10. 1145/3219819.3219885 23. Guo, Q., Zhuang, F., Qin, C., Zhu, H., Xie, X., Xiong, H., He, Q.: A survey on knowledge graph-based recommender systems. TKDE 34(8), 3549–3568 (2020) 24. Harper, F.M., Konstan, J.A.: The movielens data… — [doi:10.1145/3219819.3219885](https://doi.org/10.1145/3219819.3219885) [↗](https://doi.org/10.1145/3219819.3219885)
8. Hou et al. 32. Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., Casas, D.d.l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al.: Mistral 7b. arXiv preprint arXiv:2310.06825 (2023) 33. Kang, W., McAuley, J.: Self-attentive sequential recommendation. In: ICDM (2018) 34… [find](https://scholar.google.com/scholar?q=Hou+et+al%2E+32%2E+Jiang%2C+A%2EQ%2E%2C+Sablayrolles%2C+A%2E%2C+Mensch%2C+A%2E%2C+Bamford%2C+C%2E%2C+Chaplot%2C+D%2ES%2E%2C+Casas%2C+D%2Ed%2El%2E%2C+Bressand%2C+F%2E%2C+Lengyel%2C+G%2E%2C+Lample%2C+G%2E%2C+Saulnier%2C+L%2E%2C+et+al%2E%3A+Mistral+7b%2E+arXiv+preprint+arXiv%3A2310%2E)
9. Large Language Models are Zero-Shot Rankers for Recommender Systems 51. Roitero, K., Carterette, B., Mehrotra, R., Lalmas, M.: Leveraging behavioral heterogeneity across markets for cross-market training of recommender systems. In: Seghrouchni, A.E.F., Sukthankar, G., Liu, T., van Steen, M. (eds.) W… — [doi:10.1145/3366424.3384362](https://doi.org/10.1145/3366424.3384362) [find](https://scholar.google.com/scholar?q=Large+Language+Models+are+Zero%2DShot+Rankers+for+Recommender+Systems+51%2E+Roitero%2C+K%2E%2C+Carterette%2C+B%2E%2C+Mehrotra%2C+R%2E%2C+Lalmas%2C+M%2E%3A+Leveraging+behavioral+heterogeneity+across+markets+for+cross%2Dmarket+train)
10. ACM (2018). [https://doi.org/10.1145/3159652.3159656, https://doi.org/10.1145/3159652.3159656 55](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., Hashimoto, T.B.: Stanford alpaca: An instruction-following llama model (2023) 56. Touvron, H., Lavril, T., Izacard, G., M… [↗](https://doi.org/10.1145/3159652.3159656)
11. Hou et al. 66. Wu, L., Zheng, Z., Qiu, Z., Wang, H., Gu, H., Shen, T., Qin, C., Zhu, C., Zhu, H., Liu, Q., et al.: A survey on large language models for recommendation. arXiv preprint arXiv:2305.19860 (2023) 67. Wu, L., Zheng, Z., Qiu, Z., Wang, H., Gu, H., Shen, T., Qin, C., Zhu, C., Zhu, H., Liu,… — [arXiv:2305.19860](https://arxiv.org/abs/2305.19860) [find](https://scholar.google.com/scholar?q=Hou+et+al%2E+66%2E+Wu%2C+L%2E%2C+Zheng%2C+Z%2E%2C+Qiu%2C+Z%2E%2C+Wang%2C+H%2E%2C+Gu%2C+H%2E%2C+Shen%2C+T%2E%2C+Qin%2C+C%2E%2C+Zhu%2C+C%2E%2C+Zhu%2C+H%2E%2C+Liu%2C+Q%2E%2C+et+al%2E%3A+A+survey+on+large+language+models+for+recommendation%2E+arXiv+preprint+arXiv%3A2305%2E19860)
12. ACM (2022). https://doi.org/10.1145/3534678.3539120, https://doi. org/10.1145/3534678.3539120 69. Yuan, F., He, X., Karatzoglou, A., Zhang, L.: Parameter-efficient transfer from sequential behaviors for user modeling and recommendation. In: Huang, J.X., Chang, Y., Cheng, X., Kamps, J., Murdock, V.,… [↗](https://doi.org/10.1145/3534678.3539120)
13. ACM (2020). https://doi.org/10.1145/3397271.3401169, https://doi.org/10.1145/3397271.3401169 77. Zhao, W.X., Lin, Z., Feng, Z., Wang, P., Wen, J.R.: A revisiting study of appropriate offline evaluation for top-n recommendation algorithms. ACM Transactions on Information Systems 41(2), 1–41 (2022) 7… [↗](https://doi.org/10.1145/3397271.3401169)
14. Large Language Models are Zero-Shot Rankers for Recommender Systems 81. Zheng, B., Hou, Y., Lu, H., Chen, Y., Zhao, W.X., Wen, J.R.: Adapting large language models by integrating collaborative semantics for recommendation. arXiv preprint arXiv:2311.09049 (2023) 82. Zhou, K., Wang, H., Zhao, W.X., Zh… — [arXiv:2311.09049](https://arxiv.org/abs/2311.09049) [find](https://scholar.google.com/scholar?q=Large+Language+Models+are+Zero%2DShot+Rankers+for+Recommender+Systems+81%2E+Zheng%2C+B%2E%2C+Hou%2C+Y%2E%2C+Lu%2C+H%2E%2C+Chen%2C+Y%2E%2C+Zhao%2C+W%2EX%2E%2C+Wen%2C+J%2ER%2E%3A+Adapting+large+language+models+by+integrating+collaborative+semanti)
15. ACM (2019). https://doi.org/10.1145/3357384.3357992, https://doi.org/ 10.1145/3357384.3357992 84. Zhu, F., Wang, Y., Chen, C., Liu, G., Zheng, X.: A graphical and attentional framework for dual-target cross-domain recommendation. In: Bessiere, C. (ed.) IJCAI. pp. 3001–3008. ijcai.org (2020). https:/… — [doi:10.1145/3357384.3357992](https://doi.org/10.1145/3357384.3357992) [↗](https://doi.org/10.1145/3357384.3357992)
16. ACM (2022). https://doi.org/10.1145/3488560.3498392, https://doi.org/ 10.1145/3488560.3498392 — [doi:10.1145/3488560.3498392](https://doi.org/10.1145/3488560.3498392) [↗](https://doi.org/10.1145/3488560.3498392)

*Updated 2 November 2026*