---
title: "An Iterative Utility Judgment Framework Inspired by Philosophical"
description: "Building on this mapping, they introduce ITEM (Iterative utiliTy judgmEnt fraMework). 56 references."
authors: ["Hengran Zhang", "Keping Bi", "Jiafeng Guo", "Xueqi Cheng"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1579/"
doi: "10.18653/v1/2026.findings-acl.1579"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2026-iterative-utility-judgment-framework-inspired/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2026-iterative-utility-judgment-framework-inspired/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/22/) / An Iterative Utility Judgment Framework Inspired by Philosophical Relevance via LLMs

# An Iterative Utility Judgment Framework Inspired by Philosophical Relevance via LLMs

**Authors:** Hengran Zhang, [Keping Bi](https://md.research.elamur.ai/free/en/authors/keping-bi/), [Jiafeng Guo](https://md.research.elamur.ai/free/en/authors/jiafeng-guo/) and [Xueqi Cheng](https://md.research.elamur.ai/free/en/authors/xueqi-cheng/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1579/) · [PDF](https://aclanthology.org/2026.findings-acl.1579.pdf) · [DOI 10.18653/v1/2026.findings-acl.1579](https://doi.org/10.18653/v1/2026.findings-acl.1579) · [OpenAlex](https://openalex.org/W4399795011)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
In retrieval-augmented generation (RAG), the model's input space is limited, so it matters which retrieved passages are actually useful for answering a question, not just which ones are on-topic. The authors borrow Schutz's philosophical "system of relevancies," in which three interdependent kinds of relevance reinforce each other. They map these onto three RAG stages: relevance ranking, utility judgment, and answer generation. Building on this mapping, they introduce ITEM (Iterative utiliTy judgmEnt fraMework). In ITEM, an LLM repeatedly cycles through these stages, using draft answers to refine its choice of useful passages and the reverse. Two variants differ in whether ranking is part of the loop. The framework is tested on TREC DL, WebAP, GTI-NQ and NQ with Mistral, Llama 3 and ChatGPT. ITEM beats single-shot utility judgment, k-sampling and RankGPT baselines on utility selection, ranking and QA. It roughly matches Qwen3's long-reasoning mode at a fraction of the compute. This makes it a practical way to refine evidence for RAG.

## Key points
- **Variants:** ITEM-A loops answer generation with utility judgment. ITEM-AR also adds relevance ranking to the loop. ITEM-AR tends to help on harder settings such as non-factoid WebAP and the mixed passage lists of GTI-NQ. ITEM-A is usually better on factoid TREC DL.
- **Single iteration:** With one iteration on TREC DL, ChatGPT's listwise utility-judgment F1 rose by 8.7% over UJ-ExpA and by 3.4% over UJ-ImpA. Producing an explicit pseudo-answer first, in a separate step, worked better than doing both tasks in one inference.
- **Multiple iterations:** On WebAP, running several iterations raised listwise F1 by 6.4% for Mistral, 6.6% for Llama 3 and 7.3% for ChatGPT.
- **Ranking:** For ChatGPT on TREC DL, NDCG@5 went from 80.56 with RankGPT to 83.12 with ITEM-A r after three iterations. On GTI-NQ, ChatGPT reached 87.48 NDCG@5.
- **Efficiency and input format:** ITEM performs about as well as Qwen3 (Thinking) while using roughly 23% of its compute. Listwise input generally outperforms pointwise input. Code and benchmark are released at github.com/Trustworthy-Information-Access/ITEM.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Hengran Zhang, Keping Bi, Jiafeng Guo and Xueqi Cheng. 2026. An Iterative Utility Judgment Framework Inspired by Philosophical Relevance via LLMs. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2026iterative,
  title = {{An Iterative Utility Judgment Framework Inspired by Philosophical Relevance via LLMs}},
  author = {Hengran Zhang, Keping Bi, Jiafeng Guo and Xueqi Cheng},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1579},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1579}
}
```

- [← previous An Investigation of Linguistic Biases in LLM-Based Recommendations](https://md.research.elamur.ai/free/en/papers/venkateswaran-2026-investigation-linguistic-biases-llm-based/)
- [next → An LLM-Based Assistant for Debt Waiver Court Procedures](https://md.research.elamur.ai/free/en/papers/padro-2026-llm-based-assistant-debt-waiver/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/22/)

## References

56 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Keping Bi, Qingyao Ai, and W Bruce Croft. 2019. Iterative relevance feedback for answer passage retrieval with passage-level semantic match. In Advances in Information Retrieval: 41st European Conference on IR Research, ECIR 2019, Cologne, Germany, April 14–18, 2019, Proceedings, Part I 41, pages 55… [find](https://scholar.google.com/scholar?q=Keping+Bi%2C+Qingyao+Ai%2C+and+W+Bruce+Croft%2E+2019%2E+Iterative+relevance+feedback+for+answer+passage+retrieval+with+passage%2Dlevel+semantic+match%2E+In+Advances+in+Information+Retrieval%3A+41st+European+Confere)
2. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, and 1 others. 2022. Improving language models by retrieving from trillions of tokens. In International conference on machin… [find](https://scholar.google.com/scholar?q=Sebastian+Borgeaud%2C+Arthur+Mensch%2C+Jordan+Hoffmann%2C+Trevor+Cai%2C+Eliza+Rutherford%2C+Katie+Millican%2C+George+Bm+Van+Den+Driessche%2C+Jean%2DBaptiste+Lespiau%2C+Bogdan+Damoc%2C+Aidan+Clark%2C+and+1+others%2E+2022%2E+Imp)
3. Harry W Bruce. 1994. A cognitive view of the situational dynamism of user-centered relevance estimation. Journal of the American Society for Information Science, 45(3):142–148. [find](https://scholar.google.com/scholar?q=Harry+W+Bruce%2E+1994%2E+A+cognitive+view+of+the+situational+dynamism+of+user%2Dcentered+relevance+estimation%2E+Journal+of+the+American+Society+for+Information+Science%2C+45%283%29%3A142%E2%80%93148%2E)
4. Daniel Cohen, Liu Yang, and W Bruce Croft. 2018. Wikipassageqa: A benchmark collection for research on non-factoid answer passage retrieval. In The 41st international ACM SIGIR conference on research & development in information retrieval, pages 1165– 1168. [find](https://scholar.google.com/scholar?q=Daniel+Cohen%2C+Liu+Yang%2C+and+W+Bruce+Croft%2E+2018%2E+Wikipassageqa%3A+A+benchmark+collection+for+research+on+non%2Dfactoid+answer+passage+retrieval%2E+In+The+41st+international+ACM+SIGIR+conference+on+research+)
5. William S Cooper. 1971. A definition of relevance for information retrieval. Information storage and retrieval, 7(1):19–37. [find](https://scholar.google.com/scholar?q=William+S+Cooper%2E+1971%2E+A+definition+of+relevance+for+information+retrieval%2E+Information+storage+and+retrieval%2C+7%281%29%3A19%E2%80%9337%2E)
6. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. 2020. [Overview of the trec 2019 deep learning track](https://md.research.elamur.ai/free/en/papers/craswell-2020-overview-trec-2019-deep-learning/). arXiv preprint arXiv:2003.07820.
7. Michael Glass, Gaetano Rossiello, Md Faisal Mahbub Chowdhury, Ankita Naik, Pengshan Cai, and Alfio Gliozzo. 2022. [Re2G: Retrieve, rerank, generate](https://md.research.elamur.ai/free/en/papers/glass-2022-re2g-retrieve-rerank-generate/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages…
8. William Goffman and Vaun A Newill. 1964. Methodology for test and evaluation of information retrieval systems. Center for Documentation and Communication Research, School of Library . . . . [find](https://scholar.google.com/scholar?q=William+Goffman+and+Vaun+A+Newill%2E+1964%2E+Methodology+for+test+and+evaluation+of+information+retrieval+systems%2E+Center+for+Documentation+and+Communication+Research%2C+School+of+Library+%2E+%2E+%2E+%2E)
9. Helia Hashemi, Mohammad Aliannejadi, Hamed Zamani, and W Bruce Croft. 2020. Antique: A non-factoid question answering benchmark. In Advances in Information Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14–17, 2020, Proceedings, Part II 42, pages 166–173. Spri… [find](https://scholar.google.com/scholar?q=Helia+Hashemi%2C+Mohammad+Aliannejadi%2C+Hamed+Zamani%2C+and+W+Bruce+Croft%2E+2020%2E+Antique%3A+A+non%2Dfactoid+question+answering+benchmark%2E+In+Advances+in+Information+Retrieval%3A+42nd+European+Conference+on+IR+Re)
10. Helia Hashemi, Hamed Zamani, and W Bruce Croft. 2019. Performance prediction for non-factoid question answering. In Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval, pages 55–58. [find](https://scholar.google.com/scholar?q=Helia+Hashemi%2C+Hamed+Zamani%2C+and+W+Bruce+Croft%2E+2019%2E+Performance+prediction+for+non%2Dfactoid+question+answering%2E+In+Proceedings+of+the+2019+ACM+SIGIR+International+Conference+on+Theory+of+Information+)
11. Donald J Hillman. 1964. The notion of relevance (i). American Documentation, 15(1):26–34. [find](https://scholar.google.com/scholar?q=Donald+J+Hillman%2E+1964%2E+The+notion+of+relevance+%28i%29%2E+American+Documentation%2C+15%281%29%3A26%E2%80%9334%2E)
12. Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023. [Atlas: Few-shot learning with retrieval augmented language models](https://md.research.elamur.ai/free/en/papers/izacard-2022-atlas-few-shot-learning-retrieval/). Journal of Machine Learning Research, 24(251):1–43.
13. Kalervo Järvelin and Jaana Kekäläinen. 2017. Ir evaluation methods for retrieving highly relevant documents. In ACM SIGIR Forum, volume 51, pages 243–250. ACM New York, NY, USA. [find](https://scholar.google.com/scholar?q=Kalervo+J%C3%A4rvelin+and+Jaana+Kek%C3%A4l%C3%A4inen%2E+2017%2E+Ir+evaluation+methods+for+retrieving+highly+relevant+documents%2E+In+ACM+SIGIR+Forum%2C+volume+51%2C+pages+243%E2%80%93250%2E+ACM+New+York%2C+NY%2C+USA%2E)
14. Kaixin Ji, Danula Hettiachchi, Flora D. Salim, Falk Scholer, and Damiano Spina. 2024. Characterizing information seeking processes with multiple physiological signals. SIGIR. [find](https://scholar.google.com/scholar?q=Kaixin+Ji%2C+Danula+Hettiachchi%2C+Flora+D%2E+Salim%2C+Falk+Scholer%2C+and+Damiano+Spina%2E+2024%2E+Characterizing+information+seeking+processes+with+multiple+physiological+signals%2E+SIGIR%2E)
15. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, and 1 others. 2023a. Mistral 7b. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+and+1+others%2E+202)
16. Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023b. [Active retrieval augmented generation](https://md.research.elamur.ai/free/en/papers/jiang-2023-active-retrieval-augmented-generation/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969–7992, Singapore. Association f…
17. Mostafa Keikha, Jae Hyun Park, and W Bruce Croft. 2014a. Evaluating answer passages using summarization measures. In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval, pages 963–966. [find](https://scholar.google.com/scholar?q=Mostafa+Keikha%2C+Jae+Hyun+Park%2C+and+W+Bruce+Croft%2E+2014a%2E+Evaluating+answer+passages+using+summarization+measures%2E+In+Proceedings+of+the+37th+international+ACM+SIGIR+conference+on+Research+%26+developmen)
18. Mostafa Keikha, Jae Hyun Park, W Bruce Croft, and Mark Sanderson. 2014b. Retrieving passages and finding answers. In Proceedings of the 19th Australasian Document Computing Symposium, pages 81–84. [find](https://scholar.google.com/scholar?q=Mostafa+Keikha%2C+Jae+Hyun+Park%2C+W+Bruce+Croft%2C+and+Mark+Sanderson%2E+2014b%2E+Retrieving+passages+and+finding+answers%2E+In+Proceedings+of+the+19th+Australasian+Document+Computing+Symposium%2C+pages+81%E2%80%9384%2E)
19. DA Kemp. 1974. Relevance, pertinence and information system development. Information Storage and Retrieval, 10(2):37–47. [find](https://scholar.google.com/scholar?q=DA+Kemp%2E+1974%2E+Relevance%2C+pertinence+and+information+system+development%2E+Information+Storage+and+Retrieval%2C+10%282%29%3A37%E2%80%9347%2E)
20. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. ICLR. [find](https://scholar.google.com/scholar?q=Urvashi+Khandelwal%2C+Omer+Levy%2C+Dan+Jurafsky%2C+Luke+Zettlemoyer%2C+and+Mike+Lewis%2E+2020%2E+Generalization+through+memorization%3A+Nearest+neighbor+language+models%2E+ICLR%2E)
21. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, and 1 others. 2019. [Natural questions: a benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). Transactions of the Association for Computational Lin…
22. F Wilfrid Lancaster. 1968. Information retrieval systems: Characteristics, testing, and evaluation. (No Title). [find](https://scholar.google.com/scholar?q=F+Wilfrid+Lancaster%2E+1968%2E+Information+retrieval+systems%3A+Characteristics%2C+testing%2C+and+evaluation%2E+%28No+Title%29%2E)
23. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020. [Retrieval-augmented generation for knowledge-intensive nlp tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/). Advances in Neural Information Processing Systems, 33:9459–…
24. Xiaonan Li, Changtai Zhu, Linyang Li, Zhangyue Yin, Tianxiang Sun, and Xipeng Qiu. 2023. Llatrieval: Llm-verified retrieval for verifiable generation. arXiv preprint arXiv:2311.07838. [find](https://scholar.google.com/scholar?q=Xiaonan+Li%2C+Changtai+Zhu%2C+Linyang+Li%2C+Zhangyue+Yin%2C+Tianxiang+Sun%2C+and+Xipeng+Qiu%2E+2023%2E+Llatrieval%3A+Llm%2Dverified+retrieval+for+verifiable+generation%2E+arXiv+preprint+arXiv%3A2311%2E07838%2E)
25. Chin-Yew Lin. 2004. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out, pages 74–81.
26. Meta. 2024. Welcome llama 3 - meta’s new open llm. [find](https://scholar.google.com/scholar?q=Meta%2E+2024%2E+Welcome+llama+3+%2D+meta%E2%80%99s+new+open+llm%2E)
27. Stefano Mizzaro. 1997. Relevance: The whole history. Journal of the American society for information science, 48(9):810–832. [find](https://scholar.google.com/scholar?q=Stefano+Mizzaro%2E+1997%2E+Relevance%3A+The+whole+history%2E+Journal+of+the+American+society+for+information+science%2C+48%289%29%3A810%E2%80%93832%2E)
28. Stefano Mizzaro. 1998. How many relevances in information retrieval? Interacting with computers, 10(3):303–320. [find](https://scholar.google.com/scholar?q=Stefano+Mizzaro%2E+1998%2E+How+many+relevances+in+information+retrieval%3F+Interacting+with+computers%2C+10%283%29%3A303%E2%80%93320%2E)
29. OpenAI. 2022. Introducing chatgpt. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2022%2E+Introducing+chatgpt%2E)
30. Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. [In-context retrieval-augmented language models](https://md.research.elamur.ai/free/en/papers/ram-2023-context-retrieval-augmented-language-models/). Transactions of the Association for Computational Linguistics, 11:1316–1331.
31. Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021. [Rocketqav2: A joint training method for dense passage retrieval and passage re-ranking](https://md.research.elamur.ai/free/en/papers/ren-2021-rocketqav2-joint-training-method-dense/). EMNLP.
32. Stephen Robertson, Hugo Zaragoza, and 1 others. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389. [find](https://scholar.google.com/scholar?q=Stephen+Robertson%2C+Hugo+Zaragoza%2C+and+1+others%2E+2009%2E+The+probabilistic+relevance+framework%3A+Bm25+and+beyond%2E+Foundations+and+Trends%C2%AE+in+Information+Retrieval%2C+3%284%29%3A333%E2%80%93389%2E)
33. Tefko Saracevic. 1975. Relevance: A review of and a framework for the thinking on the notion in information science. Journal of the American Society for information science, 26(6):321–343. [find](https://scholar.google.com/scholar?q=Tefko+Saracevic%2E+1975%2E+Relevance%3A+A+review+of+and+a+framework+for+the+thinking+on+the+notion+in+information+science%2E+Journal+of+the+American+Society+for+information+science%2C+26%286%29%3A321%E2%80%93343%2E)
34. Tefko Saracevic. 1996. Relevance reconsidered. In Proceedings of the second conference on conceptions of library and information science (CoLIS 2), pages 201–218. [find](https://scholar.google.com/scholar?q=Tefko+Saracevic%2E+1996%2E+Relevance+reconsidered%2E+In+Proceedings+of+the+second+conference+on+conceptions+of+library+and+information+science+%28CoLIS+2%29%2C+pages+201%E2%80%93218%2E)
35. Tefko Saracevic, Paul Kantor, Alice Y Chamis, and Donna Trivison. 1988. A study of information seeking and retrieving. i. background and methodology. Journal of the American Society for Information science, 39(3):161–176. [find](https://scholar.google.com/scholar?q=Tefko+Saracevic%2C+Paul+Kantor%2C+Alice+Y+Chamis%2C+and+Donna+Trivison%2E+1988%2E+A+study+of+information+seeking+and+retrieving%2E+i%2E+background+and+methodology%2E+Journal+of+the+American+Society+for+Information+sc)
36. Linda Schamber and Michael Eisenberg. 1988. Relevance: The search for a definition. [find](https://scholar.google.com/scholar?q=Linda+Schamber+and+Michael+Eisenberg%2E+1988%2E+Relevance%3A+The+search+for+a+definition%2E)
37. Linda Schamber, Michael B Eisenberg, and Michael S Nilan. 1990. A re-examination of relevance: toward a dynamic, situational definition. Information processing & management, 26(6):755–776. [find](https://scholar.google.com/scholar?q=Linda+Schamber%2C+Michael+B+Eisenberg%2C+and+Michael+S+Nilan%2E+1990%2E+A+re%2Dexamination+of+relevance%3A+toward+a+dynamic%2C+situational+definition%2E+Information+processing+%26+management%2C+26%286%29%3A755%E2%80%93776%2E)
38. Alfred Schutz. 1970. Reflections on the Problem of Relevance. Greenwood Press, Westport, Conn. [find](https://scholar.google.com/scholar?q=Alfred+Schutz%2E+1970%2E+Reflections+on+the+Problem+of+Relevance%2E+Greenwood+Press%2C+Westport%2C+Conn%2E)
39. Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. 2023. [Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy](https://md.research.elamur.ai/free/en/papers/shao-2023-enhancing-retrieval-augmented-large-language/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9248–9274, Singapore. Association for…
40. Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023. [Replug: Retrieval-augmented black-box language models](https://md.research.elamur.ai/free/en/papers/shi-2023-replug-retrieval-augmented-black-box/). arXiv preprint arXiv:2301.12652.
41. Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, and 1 others. 2023. [Beyond human data: Scaling self-training for problem-solving with language models](https://md.research.elamur.ai/free/en/papers/singh-2023-human-data-scaling-self-training/). arXiv preprint arXiv:2312.06585.
42. Weihang Su, Yichen Tang, Qingyao Ai, Zhijing Wu, and Yiqun Liu. 2024. [Dragin: Dynamic retrieval augmented generation based on the real-time information needs of large language models](https://md.research.elamur.ai/free/en/papers/su-2024-dragin-dynamic-retrieval-augmented-generation/). arXiv preprint arXiv:2403.10081.
43. Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. 2023. Is chatgpt good at search? investigating large language models as re-ranking agent. EMNLP. [find](https://scholar.google.com/scholar?q=Weiwei+Sun%2C+Lingyong+Yan%2C+Xinyu+Ma%2C+Pengjie+Ren%2C+Dawei+Yin%2C+and+Zhaochun+Ren%2E+2023%2E+Is+chatgpt+good+at+search%3F+investigating+large+language+models+as+re%2Dranking+agent%2E+EMNLP%2E)
44. Don R Swanson. 1986. Subjective versus objective relevance in bibliographic retrieval systems. The library quarterly, 56(4):389–398. [find](https://scholar.google.com/scholar?q=Don+R+Swanson%2E+1986%2E+Subjective+versus+objective+relevance+in+bibliographic+retrieval+systems%2E+The+library+quarterly%2C+56%284%29%3A389%E2%80%93398%2E)
45. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. [Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions](https://md.research.elamur.ai/free/en/papers/trivedi-2022-interleaving-retrieval-chain-thought-reasoning/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A…
46. Brian C Vickery. 1959. The structure of information retrieval systems. In Proceedings of the International Conference on Scientific Information, volume 2, pages 1275–1290. [find](https://scholar.google.com/scholar?q=Brian+C+Vickery%2E+1959%2E+The+structure+of+information+retrieval+systems%2E+In+Proceedings+of+the+International+Conference+on+Scientific+Information%2C+volume+2%2C+pages+1275%E2%80%931290%2E)
47. Ellen M Voorhees and 1 others. 2003. Overview of the trec 2003 robust retrieval track. In Trec, pages 69–77. [find](https://scholar.google.com/scholar?q=Ellen+M+Voorhees+and+1+others%2E+2003%2E+Overview+of+the+trec+2003+robust+retrieval+track%2E+In+Trec%2C+pages+69%E2%80%9377%2E)
48. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). arXiv preprint arXiv:2203.11171.
49. Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. 2023. [Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts](https://md.research.elamur.ai/free/en/papers/xie-2023-adaptive-chameleon-stubborn-sloth-revealing/). In The Twelfth International Conference on Learning Representations.
50. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
51. Liu Yang, Qingyao Ai, Damiano Spina, Ruey-Cheng Chen, Liang Pang, W Bruce Croft, Jiafeng Guo, and Falk Scholer. 2016. Beyond factoid qa: effective methods for non-factoid answer sentence retrieval. In Advances in Information Retrieval: 38th European Conference on IR Research, ECIR 2016, Padua, Italy… [find](https://scholar.google.com/scholar?q=Liu+Yang%2C+Qingyao+Ai%2C+Damiano+Spina%2C+Ruey%2DCheng+Chen%2C+Liang+Pang%2C+W+Bruce+Croft%2C+Jiafeng+Guo%2C+and+Falk+Scholer%2E+2016%2E+Beyond+factoid+qa%3A+effective+methods+for+non%2Dfactoid+answer+sentence+retrieval%2E+In)
52. Hengran Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. 2023. From relevance to utility: Evidence retrieval with feedback for fact verification. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6373–6384, Singapore. Association for Comp… [find](https://scholar.google.com/scholar?q=Hengran+Zhang%2C+Ruqing+Zhang%2C+Jiafeng+Guo%2C+Maarten+de+Rijke%2C+Yixing+Fan%2C+and+Xueqi+Cheng%2E+2023%2E+From+relevance+to+utility%3A+Evidence+retrieval+with+feedback+for+fact+verification%2E+In+Findings+of+the+Ass)
53. Hengran Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. 2024. Are large language models good at utility judgments? Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’24). [find](https://scholar.google.com/scholar?q=Hengran+Zhang%2C+Ruqing+Zhang%2C+Jiafeng+Guo%2C+Maarten+de+Rijke%2C+Yixing+Fan%2C+and+Xueqi+Cheng%2E+2024%2E+Are+large+language+models+good+at+utility+judgments%3F+Proceedings+of+the+47th+International+ACM+SIGIR+Conf)
54. Xinran Zhao, Tong Chen, Sihao Chen, Hongming Zhang, and Tongshuang Wu. 2024. Beyond relevance: Evaluate and improve retrievers on perspective awareness. arXiv preprint arXiv:2405.02714. — [arXiv:2405.02714](https://arxiv.org/abs/2405.02714) [find](https://scholar.google.com/scholar?q=Xinran+Zhao%2C+Tong+Chen%2C+Sihao+Chen%2C+Hongming+Zhang%2C+and+Tongshuang+Wu%2E+2024%2E+Beyond+relevance%3A+Evaluate+and+improve+retrievers+on+perspective+awareness%2E+arXiv+preprint+arXiv%3A2405%2E02714%2E+%E2%80%94+%5BarXiv%3A2405%2E)
55. Yujia Zhou, Zheng Liu, Jiajie Jin, Jian-Yun Nie, and Zhicheng Dou. 2024. Metacognitive retrieval-augmented large language models. In Proceedings of the ACM on Web Conference 2024, WWW ’24, page 1453–1463, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Yujia+Zhou%2C+Zheng+Liu%2C+Jiajie+Jin%2C+Jian%2DYun+Nie%2C+and+Zhicheng+Dou%2E+2024%2E+Metacognitive+retrieval%2Daugmented+large+language+models%2E+In+Proceedings+of+the+ACM+on+Web+Conference+2024%2C+WWW+%E2%80%9924%2C+page+1453%E2%80%931)
56. Following the k-sampling method, we used 5 utility judgment results and aggregated them in a way similar to k-sampling to obtain the final utility judgments. The results are shown in Table 12. We can observe that our ITEM achieves the best performance, indicating the superior performance of iterativ… [find](https://scholar.google.com/scholar?q=Following+the+k%2Dsampling+method%2C+we+used+5+utility+judgment+results+and+aggregated+them+in+a+way+similar+to+k%2Dsampling+to+obtain+the+final+utility+judgments%2E+The+results+are+shown+in+Table+12%2E+We+can+)

*Updated 2 November 2026*