---
title: "Are We There Yet? Revealing the Risks of Utilizing Large"
description: "Ye et al. test this using three published LLM reviewing pipelines that were previously shown to agree… 53 references, cited by 26 papers here."
authors: ["Rui Ye", "Xianghe Pang", "Jingyi Chai", "Jiaao Chen", "Zhenfei Yin", "Zhen Xiang", "Xiaowen Dong", "Jing Shao", "Siheng Chen"]
year: 2024
venue: "arXiv:2412.01708"
source: "https://arxiv.org/abs/2412.01708"
doi: "10.48550/arxiv.2412.01708"
arxiv: "2412.01708"
licence: "arXiv non-exclusive distribution license"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ye-2024-there-yet-revealing-risks/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ye-2024-there-yet-revealing-risks/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/10/) / Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review

# Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review

**Authors:** [Rui Ye](https://md.research.elamur.ai/free/en/authors/rui-ye/), [Xianghe Pang](https://md.research.elamur.ai/free/en/authors/xianghe-pang/), [Jingyi Chai](https://md.research.elamur.ai/free/en/authors/jingyi-chai/), [Jiaao Chen](https://md.research.elamur.ai/free/en/authors/jiaao-chen/), [Zhenfei Yin](https://md.research.elamur.ai/free/en/authors/zhenfei-yin/), [Zhen Xiang](https://md.research.elamur.ai/free/en/authors/zhen-xiang/), [Xiaowen Dong](https://md.research.elamur.ai/free/en/authors/xiaowen-dong/), [Jing Shao](https://md.research.elamur.ai/free/en/authors/jing-shao/) and [Siheng Chen](https://md.research.elamur.ai/free/en/authors/siheng-chen/) · **Year:** 2024 · **Venue:** arXiv:2412.01708 · **Licence:** [arXiv non-exclusive distribution license](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2412.01708) · [PDF](https://arxiv.org/pdf/2412.01708) · [DOI 10.48550/arxiv.2412.01708](https://doi.org/10.48550/arxiv.2412.01708) · [arXiv 2412.01708](https://arxiv.org/abs/2412.01708) · [OpenAlex](https://openalex.org/W4405035007)

**Citations:** 9 citations (OpenAlex, 2026-10-14) · [26](https://md.research.elamur.ai/free/en/papers/ye-2024-there-yet-revealing-risks/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

Peer review is overloaded, and reviewers increasingly use LLMs to draft reviews, but the risks of doing so have not been measured systematically. Ye et al. test this using three published LLM reviewing pipelines that were previously shown to agree well with human reviewers, applied to ICLR 2024 submissions with their public human reviews. They study two threats. The first is author-side manipulation. Explicit manipulation hides near-invisible white text containing a favorable "review" in the PDF. Implicit manipulation has authors openly list trivial limitations. The second is inherent model flaws: hallucination on incomplete papers, a preference for longer manuscripts, and sensitivity to author identity and affiliation. Hidden injections let authors steer LLM reviews almost completely and inflate scores. LLMs also echo author-stated limitations far more than people do. They can give generous ratings to papers containing only a title. The authors call for a pause on LLM-executed reviewing, detection tools, and penalties for misuse. They argue that LLMs should serve only as a supplementary aid to human reviewers.

## Key points

- **Explicit manipulation:** hidden text was injected after the conclusion of each paper.
  - Overlap between LLM and human key points dropped from 53.29% to 15.91%.
  - Overlap with the injected content rose from about 3% to about 92%.
  - Average ratings climbed from 5.34 to 7.99.
- **Ranking simulation:** manipulating only 5% of reviews could push 12% of papers out of the top 30% of rankings.
- **Implicit manipulation:** LLM reviews matched author-disclosed limitations 4.5× more often than human reviews did.
- **Incomplete papers:** on average, title-only submissions scored at least as high as 42% of full papers.
- **Authorship bias in single-blind settings:** listing well-known authors (Hinton, Bengio, LeCun) produced more favorable reviews. The authors also probed bias toward prestigious affiliations (MIT, CMU, Stanford, Oxford, Google Research, Microsoft Research, Meta, OpenAI).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Oxford](https://md.research.elamur.ai/free/en/universities/university-of-oxford/) — University, Oxford, United Kingdom · 138 works · 161,448 citations
- [Georgia Institute of Technology](https://md.research.elamur.ai/free/en/universities/georgia-institute-of-technology/) — University, Atlanta, United States · 159 works · 28,096 citations
- [Shanghai Jiao Tong University](https://md.research.elamur.ai/free/en/universities/shanghai-jiao-tong-university/) — University, Shanghai, China · 232 works · 20,305 citations
- [Shanghai Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/shanghai-artificial-intelligence-laboratory/) — Research institute, Shanghai, China · 164 works · 13,544 citations
- [University of Georgia](https://md.research.elamur.ai/free/en/universities/university-of-georgia/) — University, Athens, United States · 18 works · 26,366 citations

📚 Shelves (4): [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 2,009 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 665 works · [🎭 History and Philosophy of Science](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/history-and-philosophy-of-science/) · 83 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works

## Cite

Cite as: Rui Ye, Xianghe Pang, Jingyi Chai, Jiaao Chen, Zhenfei Yin, Zhen Xiang, Xiaowen Dong, Jing Shao and Siheng Chen. 2024. Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review. arXiv:2412.01708.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ye2024are,
  title = {{Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review}},
  author = {Rui Ye, Xianghe Pang, Jingyi Chai, Jiaao Chen, Zhenfei Yin, Zhen Xiang, Xiaowen Dong, Jing Shao and Siheng Chen},
  year = {2024},
  doi = {10.48550/arxiv.2412.01708},
  eprint = {2412.01708},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.01708}
}
```

- [← previous Transformers Can Do Arithmetic with the Right Embeddings](https://md.research.elamur.ai/free/en/papers/mcleish-2024-transformers-arithmetic-right-embeddings/)
- [next → Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap](https://md.research.elamur.ai/free/en/papers/srivastava-2024-functional-benchmarks-robust-evaluation-reasoning/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/10/)

## References

53 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. SPJM ( Serge) Horbach and W ( Willem) Halffman. The changing forms and expectations of peer review. Research integrity and peer review, 3:1–15, 2018. [find](https://scholar.google.com/scholar?q=SPJM+%28+Serge%29+Horbach+and+W+%28+Willem%29+Halffman%2E+The+changing+forms+and+expectations+of+peer+review%2E+Research+integrity+and+peer+review%2C+3%3A1%E2%80%9315%2C+2018%2E)
2. Bruce Alberts, Brooks Hanson, and Katrina L Kelner. Reviewing peer review, 2008. [find](https://scholar.google.com/scholar?q=Bruce+Alberts%2C+Brooks+Hanson%2C+and+Katrina+L+Kelner%2E+Reviewing+peer+review%2C+2008%2E)
3. Daryl E Chubin and Edward J Hackett. Peerless science: Peer review and US science policy. State University of New York Press, 1990. [find](https://scholar.google.com/scholar?q=Daryl+E+Chubin+and+Edward+J+Hackett%2E+Peerless+science%3A+Peer+review+and+US+science+policy%2E+State+University+of+New+York+Press%2C+1990%2E)
4. Richard Smith. Peer review: a flawed process at the heart of science and journals. Journal of the royal society of medicine, 99(4):178–182, 2006. [find](https://scholar.google.com/scholar?q=Richard+Smith%2E+Peer+review%3A+a+flawed+process+at+the+heart+of+science+and+journals%2E+Journal+of+the+royal+society+of+medicine%2C+99%284%29%3A178%E2%80%93182%2C+2006%2E)
5. Lutz Bornmann and Rüdiger Mutz. Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references. Journal of the association for information science and technology, 66(11):2215–2222, 2015. [find](https://scholar.google.com/scholar?q=Lutz+Bornmann+and+R%C3%BCdiger+Mutz%2E+Growth+rates+of+modern+science%3A+A+bibliometric+analysis+based+on+the+number+of+publications+and+cited+references%2E+Journal+of+the+association+for+information+science+and)
6. Alison McCook. Is peer review broken? submissions are up, reviewers are overtaxed, and authors are lodging complaint after complaint about the process at top-tier journals. what’s wrong with peer review? The scientist, 20(2):26–35, 2006. [find](https://scholar.google.com/scholar?q=Alison+McCook%2E+Is+peer+review+broken%3F+submissions+are+up%2C+reviewers+are+overtaxed%2C+and+authors+are+lodging+complaint+after+complaint+about+the+process+at+top%2Dtier+journals%2E+what%E2%80%99s+wrong+with+peer+revi)
7. Martijn Arns. Open access is tiring out peer reviewers. Nature, 515(7528):467–467, 2014. [find](https://scholar.google.com/scholar?q=Martijn+Arns%2E+Open+access+is+tiring+out+peer+reviewers%2E+Nature%2C+515%287528%29%3A467%E2%80%93467%2C+2014%2E)
8. Carole J Lee, Cassidy R Sugimoto, Guo Zhang, and Blaise Cronin. Bias in peer review. Journal of the American Society for information Science and Technology, 64(1):2–17, 2013. [find](https://scholar.google.com/scholar?q=Carole+J+Lee%2C+Cassidy+R+Sugimoto%2C+Guo+Zhang%2C+and+Blaise+Cronin%2E+Bias+in+peer+review%2E+Journal+of+the+American+Society+for+information+Science+and+Technology%2C+64%281%29%3A2%E2%80%9317%2C+2013%2E)
9. Nihar B Shah. Challenges, experiments, and computational solutions in peer review. Communications of the ACM, 65(6):76–87, 2022. [find](https://scholar.google.com/scholar?q=Nihar+B+Shah%2E+Challenges%2C+experiments%2C+and+computational+solutions+in+peer+review%2E+Communications+of+the+ACM%2C+65%286%29%3A76%E2%80%9387%2C+2022%2E)
10. Charles W Fox, Arianne YK Albert, and Timothy H Vines. Recruitment of reviewers is becoming harder at some journals: a test of the influence of reviewer fatigue at six journals in ecology and evolution. Research Integrity and Peer Review, 2:1–6, 2017. [find](https://scholar.google.com/scholar?q=Charles+W+Fox%2C+Arianne+YK+Albert%2C+and+Timothy+H+Vines%2E+Recruitment+of+reviewers+is+becoming+harder+at+some+journals%3A+a+test+of+the+influence+of+reviewer+fatigue+at+six+journals+in+ecology+and+evolutio)
11. Jigisha Patel. Why training and specialization is needed for peer review: a case study of peer review for randomized controlled trials. BMC medicine, 12:1–7, 2014. [find](https://scholar.google.com/scholar?q=Jigisha+Patel%2E+Why+training+and+specialization+is+needed+for+peer+review%3A+a+case+study+of+peer+review+for+randomized+controlled+trials%2E+BMC+medicine%2C+12%3A1%E2%80%937%2C+2014%2E)
12. Balazs Aczel, Barnabas Szaszi, and Alex O Holcombe. A billion-dollar donation: estimating the cost of researchers’ time spent on peer review. Research Integrity and Peer Review, 6:1–8, 2021. [find](https://scholar.google.com/scholar?q=Balazs+Aczel%2C+Barnabas+Szaszi%2C+and+Alex+O+Holcombe%2E+A+billion%2Ddollar+donation%3A+estimating+the+cost+of+researchers%E2%80%99+time+spent+on+peer+review%2E+Research+Integrity+and+Peer+Review%2C+6%3A1%E2%80%938%2C+2021%2E)
13. Michail Kovanis, Raphaël Porcher, Philippe Ravaud, and Ludovic Trinquart. The global burden of journal peer review in the biomedical literature: Strong imbalance in the collective enterprise. PloS one, 11(11):e0166387, 2016. [find](https://scholar.google.com/scholar?q=Michail+Kovanis%2C+Rapha%C3%ABl+Porcher%2C+Philippe+Ravaud%2C+and+Ludovic+Trinquart%2E+The+global+burden+of+journal+peer+review+in+the+biomedical+literature%3A+Strong+imbalance+in+the+collective+enterprise%2E+PloS+one)
14. Robert Schulz, Adrian Barnett, René Bernard, Nicholas JL Brown, Jennifer A Byrne, Peter Eckmann, Małgorzata A Gazda, Halil Kilicoglu, Eric M Prager, Maia Salholz-Hillel, et al. Is the future of peer review automated? BMC research notes, 15(1):203, 2022. [find](https://scholar.google.com/scholar?q=Robert+Schulz%2C+Adrian+Barnett%2C+Ren%C3%A9+Bernard%2C+Nicholas+JL+Brown%2C+Jennifer+A+Byrne%2C+Peter+Eckmann%2C+Ma%C5%82gorzata+A+Gazda%2C+Halil+Kilicoglu%2C+Eric+M+Prager%2C+Maia+Salholz%2DHillel%2C+et+al%2E+Is+the+future+of+peer+r)
15. Tracey Weissgerber, Nico Riedel, Halil Kilicoglu, Cyril Labbé, Peter Eckmann, Gerben Ter Riet, Jennifer Byrne, Guillaume Cabanac, Amanda Capes-Davis, Bertrand Favier, et al. Automated screening of covid-19 preprints: can we help authors to improve transparency and reproducibility? Nature medicine, 2… [find](https://scholar.google.com/scholar?q=Tracey+Weissgerber%2C+Nico+Riedel%2C+Halil+Kilicoglu%2C+Cyril+Labb%C3%A9%2C+Peter+Eckmann%2C+Gerben+Ter+Riet%2C+Jennifer+Byrne%2C+Guillaume+Cabanac%2C+Amanda+Capes%2DDavis%2C+Bertrand+Favier%2C+et+al%2E+Automated+screening+of+cov)
16. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
17. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783, 2024.
18. Ryan Liu and Nihar B Shah. [Reviewergpt? an exploratory study on using large language models for paper reviewing](https://md.research.elamur.ai/free/en/papers/liu-2023-reviewergpt-exploratory-study-using/). arXiv preprint arXiv:2306.00622, 2023.
19. Mohammad Hosseini and Serge PJM Horbach. Fighting reviewer fatigue or amplifying bias? considerations and recommendations for use of chatgpt and other large language models in scholarly peer review. Research integrity and peer review, 8(1):4, 2023. [find](https://scholar.google.com/scholar?q=Mohammad+Hosseini+and+Serge+PJM+Horbach%2E+Fighting+reviewer+fatigue+or+amplifying+bias%3F+considerations+and+recommendations+for+use+of+chatgpt+and+other+large+language+models+in+scholarly+peer+review%2E+R)
20. Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, et al. [Can large language models provide useful feedback on research papers? a large-scale empirical analysis](https://md.research.elamur.ai/free/en/papers/liang-2023-large-language-models-provide/). NEJM AI, 1(8):AIoa2400196, 2024.
21. Keith Tyser, Ben Segev, Gaston Longhitano, Xin-Yu Zhang, Zachary Meeks, Jason Lee, Uday Garg, Nicholas Belsten, Avi Shporer, Madeleine Udell, et al. [Ai-driven review systems: Evaluating llms in scalable and bias-aware academic reviews](https://md.research.elamur.ai/free/en/papers/tyser-2024-ai-driven-review-systems/). arXiv preprint arXiv:2408.10365, 2024.
22. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. [The ai scientist: Towards fully automated open-ended scientific discovery](https://md.research.elamur.ai/free/en/papers/lu-2024-ai-scientist-fully-automated/). arXiv preprint arXiv:2408.06292, 2024.
23. Yiqiao Jin, Qinlin Zhao, Yiyang Wang, Hao Chen, Kaijie Zhu, Yijia Xiao, and Jindong Wang. [AgentReview: Exploring peer review dynamics with LLM agents](https://md.research.elamur.ai/free/en/papers/jin-2024-agentreview-exploring-peer-review/). In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing…
24. Jianxiang Yu, Zichen Ding, Jiaqi Tan, Kangyang Luo, Zhenmin Weng, Chenghua Gong, Long Zeng, RenJing Cui, Chengcheng Han, Qiushi Sun, et al. Automated peer reviewing in paper sea: Standardization, evaluation, and analysis. In Findings of the Association for Computational Linguistics: EMNLP 2024, page… [find](https://scholar.google.com/scholar?q=Jianxiang+Yu%2C+Zichen+Ding%2C+Jiaqi+Tan%2C+Kangyang+Luo%2C+Zhenmin+Weng%2C+Chenghua+Gong%2C+Long+Zeng%2C+RenJing+Cui%2C+Chengcheng+Han%2C+Qiushi+Sun%2C+et+al%2E+Automated+peer+reviewing+in+paper+sea%3A+Standardization%2C+eval)
25. Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, et al. [Monitoring ai-modified content at scale: A case study on the impact of chatgpt on ai conference peer reviews](https://md.research.elamur.ai/free/en/papers/liang-2024-monitoring-ai-modified-content/). In Forty-first International Conference on Machine…
26. Giuseppe Russo Latona, Manoel Horta Ribeiro, Tim R Davidson, Veniamin Veselovsky, and Robert West. [The ai review lottery: Widespread ai-assisted peer reviews boost paper scores and acceptance rates](https://md.research.elamur.ai/free/en/papers/latona-2024-ai-review-lottery-widespread/). arXiv preprint arXiv:2405.02150, 2024.
27. Sungduk Yu, Man Luo, Avinash Madasu, Vasudev Lal, and Phillip Howard. [Is your paper being reviewed by an llm? investigating ai text detectability in peer review](https://md.research.elamur.ai/free/en/papers/yu-2024-paper-being-reviewed-llm/). arXiv preprint arXiv:2410.03019, 2024.
28. Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. [How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms](https://md.research.elamur.ai/free/en/papers/zeng-2024-johnny-persuade-llms-jailbreak-them/). arXiv preprint arXiv:2401.06373, 2024.
29. Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. [Toxicity in chatgpt: Analyzing persona-assigned language models](https://md.research.elamur.ai/free/en/papers/deshpande-2023-toxicity-chatgpt-analyzing-persona-assigned/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1236–1270, 2023.
30. Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. Defending chatgpt against jailbreak attack via self-reminders. Nature Machine Intelligence, 5(12):1486–1496, 2023. [find](https://scholar.google.com/scholar?q=Yueqi+Xie%2C+Jingwei+Yi%2C+Jiawei+Shao%2C+Justin+Curl%2C+Lingjuan+Lyu%2C+Qifeng+Chen%2C+Xing+Xie%2C+and+Fangzhao+Wu%2E+Defending+chatgpt+against+jailbreak+attack+via+self%2Dreminders%2E+Nature+Machine+Intelligence%2C+5%2812%29)
31. Patrick Schramowski, Cigdem Turan, Nico Andersen, Constantin A Rothkopf, and Kristian Kersting. Large pre-trained language models contain human-like biases of what is right and wrong to do. Nature Machine Intelligence, 4(3):258–268, 2022. [find](https://scholar.google.com/scholar?q=Patrick+Schramowski%2C+Cigdem+Turan%2C+Nico+Andersen%2C+Constantin+A+Rothkopf%2C+and+Kristian+Kersting%2E+Large+pre%2Dtrained+language+models+contain+human%2Dlike+biases+of+what+is+right+and+wrong+to+do%2E+Nature+Mac)
32. Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. [Bias and fairness in large language models: A survey](https://md.research.elamur.ai/free/en/papers/gallegos-2023-bias-fairness-large-language-models/). Computational Linguistics, pages 1–79, 2024.
33. Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. Benchmarking cognitive biases in large language models as evaluators. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 517–545, Ba… [find](https://scholar.google.com/scholar?q=Ryan+Koo%2C+Minhwa+Lee%2C+Vipul+Raheja%2C+Jong+Inn+Park%2C+Zae+Myung+Kim%2C+and+Dongyeop+Kang%2E+Benchmarking+cognitive+biases+in+large+language+models+as+evaluators%2E+In+Lun%2DWei+Ku%2C+Andre+Martins%2C+and+Vivek+Sriku)
34. Jessica Maria Echterhoff, Yao Liu, Abeer Alessa, Julian McAuley, and Zexue He. Cognitive bias in decision-making with LLMs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 12640– 12653, Miami, Florida, USA, N… [find](https://scholar.google.com/scholar?q=Jessica+Maria+Echterhoff%2C+Yao+Liu%2C+Abeer+Alessa%2C+Julian+McAuley%2C+and+Zexue+He%2E+Cognitive+bias+in+decision%2Dmaking+with+LLMs%2E+In+Yaser+Al%2DOnaizan%2C+Mohit+Bansal%2C+and+Yun%2DNung+Chen%2C+editors%2C+Findings+of+t)
35. NeurIPS. Paper checklist. [https://neurips.cc/public/guides/PaperChecklist](https://neurips.cc/public/guides/PaperChecklist), 2024. Accessed: 2024-11-04. [↗](https://neurips.cc/public/guides/PaperChecklist)
36. Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren’s song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.[01219, 2023](https://md.research.elamur.ai/free/en/papers/zhang-2023-siren-s-song-ai-ocean/).
37. Potsawee Manakul, Adian Liusie, and Mark Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9004–9017, 2023. [find](https://scholar.google.com/scholar?q=Potsawee+Manakul%2C+Adian+Liusie%2C+and+Mark+Gales%2E+Selfcheckgpt%3A+Zero%2Dresource+black%2Dbox+hallucination+detection+for+generative+large+language+models%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+M)
38. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024. [find](https://scholar.google.com/scholar?q=Yann+Dubois%2C+Bal%C3%A1zs+Galambosi%2C+Percy+Liang%2C+and+Tatsunori+B+Hashimoto%2E+Length%2Dcontrolled+alpacaeval%3A+A+simple+way+to+debias+automatic+evaluators%2E+arXiv+preprint+arXiv%3A2404%2E04475%2C+2024%2E)
39. DeepSeek-AI. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024. [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2E+Deepseek%2Dv2%3A+A+strong%2C+economical%2C+and+efficient+mixture%2Dof%2Dexperts+language+model%2C+2024%2E)
40. Qwen Team. Qwen2.5: A party of foundation models, September 2024. [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+Qwen2%2E5%3A+A+party+of+foundation+models%2C+September+2024%2E)
41. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. [Chatbot arena: An open platform for evaluating llms by human preference](https://md.research.elamur.ai/free/en/papers/chiang-2024-chatbot-arena-open-platform-evaluating/), 2024.
42. D Ecer and G Maciocci. Sciencebeam—using computer vision to extract pdf data. Elife [Internet], 4, 2017. [find](https://scholar.google.com/scholar?q=D+Ecer+and+G+Maciocci%2E+Sciencebeam%E2%80%94using+computer+vision+to+extract+pdf+data%2E+Elife+%5BInternet%5D%2C+4%2C+2017%2E)
43. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926, 2023. [find](https://scholar.google.com/scholar?q=Peiyi+Wang%2C+Lei+Li%2C+Liang+Chen%2C+Zefan+Cai%2C+Dawei+Zhu%2C+Binghuai+Lin%2C+Yunbo+Cao%2C+Qi+Liu%2C+Tianyu+Liu%2C+and+Zhifang+Sui%2E+Large+language+models+are+not+fair+evaluators%2E+arXiv+preprint+arXiv%3A2305%2E17926%2C+2023)
44. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=Chujie+Zheng%2C+Hao+Zhou%2C+Fandong+Meng%2C+Jie+Zhou%2C+and+Minlie+Huang%2E+Large+language+models+are+not+robust+multiple+choice+selectors%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2C+2)
45. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Advances in Neural Information Processing Systems, 36:46595–46623, 2023.
46. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. [A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/). arXiv preprint arXiv:2311.05232, 2023.
47. Minghao Wu and Alham Fikri Aji. [Style over substance: Evaluation biases for large language models](https://md.research.elamur.ai/free/en/papers/wu-2025-style-over-substance-evaluation-biases/). arXiv preprint arXiv:2307.03025, 2023.
48. QS Top Universities. QS World University Rankings by Subject 2024: Computer Science & Information Systems, 2024. Accessed: 2024-11-19. [find](https://scholar.google.com/scholar?q=QS+Top+Universities%2E+QS+World+University+Rankings+by+Subject+2024%3A+Computer+Science+%26+Information+Systems%2C+2024%2E+Accessed%3A+2024%2D11%2D19%2E)
49. ICLR. Assisting iclr 2025 reviewers with feedback. [https://blog.iclr.cc/2024/10/09/iclr2025-assisting-reviewers/](https://blog.iclr.cc/2024/10/09/iclr2025-assisting-reviewers/), 2025. Accessed: 2024-11-30. [↗](https://blog.iclr.cc/2024/10/09/iclr2025-assisting-reviewers/)
50. Miroslav Kubat, Stan Matwin, et al. Addressing the curse of imbalanced training sets: one-sided selection. In Icml, volume 97, page 179. Citeseer, 1997. [find](https://scholar.google.com/scholar?q=Miroslav+Kubat%2C+Stan+Matwin%2C+et+al%2E+Addressing+the+curse+of+imbalanced+training+sets%3A+one%2Dsided+selection%2E+In+Icml%2C+volume+97%2C+page+179%2E+Citeseer%2C+1997%2E)
51. Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions on knowledge and data engineering, 21(9):1263–1284, 2009. [find](https://scholar.google.com/scholar?q=Haibo+He+and+Edwardo+A+Garcia%2E+Learning+from+imbalanced+data%2E+IEEE+Transactions+on+knowledge+and+data+engineering%2C+21%289%29%3A1263%E2%80%931284%2C+2009%2E)
52. D MCFADDEN. Conditional logit analysis of qualitative choice behavior. Frontiers in Econometrics, 1974. [find](https://scholar.google.com/scholar?q=D+MCFADDEN%2E+Conditional+logit+analysis+of+qualitative+choice+behavior%2E+Frontiers+in+Econometrics%2C+1974%2E)
53. Tom Fawcett. An introduction to roc analysis. Pattern recognition letters, 27(8):861–874, 2006. [find](https://scholar.google.com/scholar?q=Tom+Fawcett%2E+An+introduction+to+roc+analysis%2E+Pattern+recognition+letters%2C+27%288%29%3A861%E2%80%93874%2C+2006%2E)

*Updated 3 November 2026*