---
title: "Mind the Blind Spots — paper by Shin et al. (2025)"
description: "The authors propose a focus-level evaluation framework. 34 references, cited by 9 papers here."
authors: ["Hyungyu Shin", "Jingyu Tang", "Yoonjoo Lee", "Nayoung Kim", "Hyunseung Lim", "Ji Yong Cho", "Hwajung Hong", "Moontae Lee", "Juho Kim"]
year: 2025
venue: "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2025.emnlp-main.1805/"
doi: "10.18653/v1/2025.emnlp-main.1805"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shin-2025-mind-blind-spots-focus-level/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shin-2025-mind-blind-spots-focus-level/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/6/) / Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviews

# Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviews

**Authors:** Hyungyu Shin, Jingyu Tang, Yoonjoo Lee, Nayoung Kim, Hyunseung Lim, Ji Yong Cho, Hwajung Hong, [Moontae Lee](https://md.research.elamur.ai/free/en/authors/moontae-lee/) and [Juho Kim](https://md.research.elamur.ai/free/en/authors/juho-kim/) · **Year:** 2025 · **Venue:** [Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2025](https://md.research.elamur.ai/free/en/venues/emnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.emnlp-main.1805/) · [PDF](https://aclanthology.org/2025.emnlp-main.1805.pdf) · [DOI 10.18653/v1/2025.emnlp-main.1805](https://doi.org/10.18653/v1/2025.emnlp-main.1805) · [OpenAlex](https://openalex.org/W4416035115)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · [9](https://md.research.elamur.ai/free/en/papers/shin-2025-mind-blind-spots-focus-level/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLMs can now draft peer reviews, but existing evaluations rely on surface overlap (BLEU, ROUGE) or content qualities such as specificity. These checks do not reveal whether a model praises and criticizes the same things human experts do. The authors propose a focus-level evaluation framework. It represents a review's focus as a normalized distribution of strength and weakness points over predefined facets, along two dimensions: target (what is discussed, e.g., problem, method, experiment) and aspect (which criterion applies, e.g., validity, clarity, novelty). They derived the facets from AI conference guidelines and earlier review-analysis work. An automatic LLM annotator labels each review point, and the resulting distributions are compared with those from ICLR expert reviews on OpenReview. They tested eight off-the-shelf LLMs, the MARG system and a fine-tuned gpt-4o. The untuned models overemphasize technical validity and largely neglect novelty when listing weaknesses, and their agreement with experts on individual points is low. Fine-tuning brings the focus distributions closest to human ones. The work offers an interpretable diagnostic of LLM review blind spots and a basis for training more balanced reviewers.

## Key points
- **Benchmark data:** 676 ICLR papers (2021–2024) were drawn from 18,407 collected OpenReview submissions. From these, 3,657 expert strengths and weaknesses were extracted, mainly from meta-reviews enriched with details from individual reviews.
- **Facet scheme and annotator:** The scheme has 7 target facets and 5 aspect facets. The automatic annotator agreed well with humans (Cohen's kappa 0.81 for target, 0.79 for aspect).
- **Models evaluated:** 8 LLMs (4 GPT, 2 Llama, 2 DeepSeek), plus MARG and a gpt-4o fine-tuned on the dataset. Even the best model matched human target/aspect labels per point with an F1 of only 0.373.
- **Different strengths across models:** The fine-tuned gpt-4o had the most human-like focus distribution, while Llama-405B scored highest on text similarity. This argues for evaluating reviews from several angles.
- **Released resource:** The authors released papers, expert reviews, annotated expert points, reviews from 8 LLMs, and 43,042 LLM-generated strengths and weaknesses with target/aspect labels (hosted on figshare).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Hyungyu Shin, Jingyu Tang, Yoonjoo Lee, Nayoung Kim, Hyunseung Lim, Ji Yong Cho, Hwajung Hong, Moontae Lee and Juho Kim. 2025. Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviews. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shin2025mind,
  title = {{Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviews}},
  author = {Hyungyu Shin, Jingyu Tang, Yoonjoo Lee, Nayoung Kim, Hyunseung Lim, Ji Yong Cho, Hwajung Hong, Moontae Lee and Juho Kim},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1805},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.1805}
}
```

- [← previous MA-RAG: Multi-Agent Retrieval-Augmented Generation via Collaborative Chain-of-Thought Reasoning](https://md.research.elamur.ai/free/en/papers/nguyen-2025-ma-rag-multi-agent-retrieval/)
- [next → MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teaming](https://md.research.elamur.ai/free/en/papers/guo-2025-mtsa-multi-turn-safety-alignment/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/6/)

## References

34 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Akhil Agnihotri, Rahul Jain, Deepak Ramachandran, and Zheng Wen. 2025. Multi-objective preference optimization: Improving human alignment of generative models. Preprint, arXiv:2505.10892. — [arXiv:2505.10892](https://arxiv.org/abs/2505.10892) [find](https://scholar.google.com/scholar?q=Akhil+Agnihotri%2C+Rahul+Jain%2C+Deepak+Ramachandran%2C+and+Zheng+Wen%2E+2025%2E+Multi%2Dobjective+preference+optimization%3A+Improving+human+alignment+of+generative+models%2E+Preprint%2C+arXiv%3A2505%2E10892%2E+%E2%80%94+%5BarXiv%3A250)
2. Nicolas Bougie and Narimasa Watanabe. 2024. Generative adversarial reviews: When llms become the critic. ArXiv, abs/2412.10415. — [arXiv:2412.10415](https://arxiv.org/abs/2412.10415) [find](https://scholar.google.com/scholar?q=Nicolas+Bougie+and+Narimasa+Watanabe%2E+2024%2E+Generative+adversarial+reviews%3A+When+llms+become+the+critic%2E+ArXiv%2C+abs%2F2412%2E10415%2E+%E2%80%94+%5BarXiv%3A2412%2E10415%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2412%2E10415%29)
3. Souvic Chakraborty, Pawan Goyal, and Animesh Mukherjee. 2020. Aspect-based sentiment analysis of scientific reviews. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, pages 207–216. [find](https://scholar.google.com/scholar?q=Souvic+Chakraborty%2C+Pawan+Goyal%2C+and+Animesh+Mukherjee%2E+2020%2E+Aspect%2Dbased+sentiment+analysis+of+scientific+reviews%2E+In+Proceedings+of+the+ACM%2FIEEE+Joint+Conference+on+Digital+Libraries+in+2020%2C+pages)
4. Eric Chamoun, Michael Schlichktrull, and Andreas Vlachos. 2024. Automated focused feedback generation for scientific writing assistance. arXiv preprint arXiv:2405.20477. [find](https://scholar.google.com/scholar?q=Eric+Chamoun%2C+Michael+Schlichktrull%2C+and+Andreas+Vlachos%2E+2024%2E+Automated+focused+feedback+generation+for+scientific+writing+assistance%2E+arXiv+preprint+arXiv%3A2405%2E20477%2E)
5. G. Choudhary, Natwar Modani, and Nitish Maurya. 2022. React: A review comment dataset for actionability (and more). ArXiv, abs/2210.00443. — [arXiv:2210.00443](https://arxiv.org/abs/2210.00443) [find](https://scholar.google.com/scholar?q=G%2E+Choudhary%2C+Natwar+Modani%2C+and+Nitish+Maurya%2E+2022%2E+React%3A+A+review+comment+dataset+for+actionability+%28and+more%29%2E+ArXiv%2C+abs%2F2210%2E00443%2E+%E2%80%94+%5BarXiv%3A2210%2E00443%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2210%2E00443%29)
6. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. [find](https://scholar.google.com/scholar?q=Jacob+Cohen%2E+1960%2E+A+coefficient+of+agreement+for+nominal+scales%2E+Educational+and+psychological+measurement%2C+20%281%29%3A37%E2%80%9346%2E)
7. Mike D’Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. 2024. Marg: Multi-agent review generation for scientific papers. arXiv preprint arXiv:2401.04259. [find](https://scholar.google.com/scholar?q=Mike+D%E2%80%99Arcy%2C+Tom+Hope%2C+Larry+Birnbaum%2C+and+Doug+Downey%2E+2024%2E+Marg%3A+Multi%2Dagent+review+generation+for+scientific+papers%2E+arXiv+preprint+arXiv%3A2401%2E04259%2E)
8. Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, et al. 2024. [Llms assist nlp researchers: Critique paper (meta-) reviewing](https://md.research.elamur.ai/free/en/papers/du-2024-llms-assist-nlp-researchers-critique/). arXiv preprint arXiv:2406.16253.
9. Xian Gao, Jiacheng Ruan, Jingsheng Gao, Ting Liu, and Yuzhuo Fu. 2025. Reviewagents: Bridging the gap between human and ai-generated paper reviews. ArXiv, abs/2503.08506. — [arXiv:2503.08506](https://arxiv.org/abs/2503.08506) [find](https://scholar.google.com/scholar?q=Xian+Gao%2C+Jiacheng+Ruan%2C+Jingsheng+Gao%2C+Ting+Liu%2C+and+Yuzhuo+Fu%2E+2025%2E+Reviewagents%3A+Bridging+the+gap+between+human+and+ai%2Dgenerated+paper+reviews%2E+ArXiv%2C+abs%2F2503%2E08506%2E+%E2%80%94+%5BarXiv%3A2503%2E08506%5D%28https%3A%2F%2F)
10. Zhaolin Gao, Kianté Brantley, and Thorsten Joachims. 2024. [Reviewer2: Optimizing review generation through prompt generation](https://md.research.elamur.ai/free/en/papers/gao-2024-reviewer2-optimizing-review-generation-through/). Preprint, arXiv:2402.10886.
11. Tirthankar Ghosal, Sandeep Kumar, Prabhat Kumar Bharti, and Asif Ekbal. 2022. Peer review analyze: A novel benchmark resource for computational analysis of peer reviews. Plos one, 17(1):e0259238. [find](https://scholar.google.com/scholar?q=Tirthankar+Ghosal%2C+Sandeep+Kumar%2C+Prabhat+Kumar+Bharti%2C+and+Asif+Ekbal%2E+2022%2E+Peer+review+analyze%3A+A+novel+benchmark+resource+for+computational+analysis+of+peer+reviews%2E+Plos+one%2C+17%281%29%3Ae0259238%2E)
12. Hugo Horta and Jisun Jung. 2024. The crisis of peer review: Part of the evolution of science. Higher Education Quarterly, page e12511. [find](https://scholar.google.com/scholar?q=Hugo+Horta+and+Jisun+Jung%2E+2024%2E+The+crisis+of+peer+review%3A+Part+of+the+evolution+of+science%2E+Higher+Education+Quarterly%2C+page+e12511%2E)
13. Eftekhar Hossain, Sanjeev Kumar Sinha, Naman Bansal, Alex Knipper, Souvika Sarkar, John Salvador, Yash Mahajan, Sri Guttikonda, Mousumi Akter, Md. Mahadi Hassan, Matthew Freestone, Matthew C. Williams Jr., Dongji Feng, and Santu Karmaker. 2025. Llms as meta-reviewers’ assistants: A case study. Prepr… [find](https://scholar.google.com/scholar?q=Eftekhar+Hossain%2C+Sanjeev+Kumar+Sinha%2C+Naman+Bansal%2C+Alex+Knipper%2C+Souvika+Sarkar%2C+John+Salvador%2C+Yash+Mahajan%2C+Sri+Guttikonda%2C+Mousumi+Akter%2C+Md%2E+Mahadi+Hassan%2C+Matthew+Freestone%2C+Matthew+C%2E+Williams)
14. Mohammad Hosseini and Serge P.J.M. Horbach. 2023. Fighting reviewer fatigue or amplifying bias? considerations and recommendations for use of chatgpt and other large language models in scholarly peer review. Research Integrity and Peer Review, 8. [find](https://scholar.google.com/scholar?q=Mohammad+Hosseini+and+Serge+P%2EJ%2EM%2E+Horbach%2E+2023%2E+Fighting+reviewer+fatigue+or+amplifying+bias%3F+considerations+and+recommendations+for+use+of+chatgpt+and+other+large+language+models+in+scholarly+peer+)
15. Johann Füller Julian Just, Thomas Ströhle and Katja Hutter. 2024. Ai-based novelty detection in crowdsourced idea spaces. Innovation, 26(3):359–386. [find](https://scholar.google.com/scholar?q=Johann+F%C3%BCller+Julian+Just%2C+Thomas+Str%C3%B6hle+and+Katja+Hutter%2E+2024%2E+Ai%2Dbased+novelty+detection+in+crowdsourced+idea+spaces%2E+Innovation%2C+26%283%29%3A359%E2%80%93386%2E)
16. Giuseppe Russo Latona, Manoel Horta Ribeiro, Tim R Davidson, Veniamin Veselovsky, and Robert West. 2024. The ai review lottery: Widespread ai-assisted peer reviews boost paper scores and acceptance rates. arXiv preprint arXiv:2405.02150. [find](https://scholar.google.com/scholar?q=Giuseppe+Russo+Latona%2C+Manoel+Horta+Ribeiro%2C+Tim+R+Davidson%2C+Veniamin+Veselovsky%2C+and+Robert+West%2E+2024%2E+The+ai+review+lottery%3A+Widespread+ai%2Dassisted+peer+reviews+boost+paper+scores+and+acceptance+ra)
17. Miao Li, Eduard H. Hovy, and Jey Han Lau. 2023. Summarizing multiple documents with conversational structure for meta-review generation. In Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Miao+Li%2C+Eduard+H%2E+Hovy%2C+and+Jey+Han+Lau%2E+2023%2E+Summarizing+multiple+documents+with+conversational+structure+for+meta%2Dreview+generation%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processi)
18. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110. [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+et+al%2E+2022%2E+Holistic+evaluation+of+language+models%2E+)
19. Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, et al. 2024. Can large language models provide useful feedback on research papers? a large-scale empirical analysis. NEJM AI, 1(8):AIoa2400196. [find](https://scholar.google.com/scholar?q=Weixin+Liang%2C+Yuhui+Zhang%2C+Hancheng+Cao%2C+Binglu+Wang%2C+Daisy+Yi+Ding%2C+Xinyu+Yang%2C+Kailas+Vodrahalli%2C+Siyu+He%2C+Daniel+Scott+Smith%2C+Yian+Yin%2C+et+al%2E+2024%2E+Can+large+language+models+provide+useful+feedbac)
20. Ethan Lin, Zhiyuan Peng, and Yi Fang. 2024. Evaluating and enhancing large language models for novelty assessment in scholarly publications. ArXiv, abs/2409.16605. [find](https://scholar.google.com/scholar?q=Ethan+Lin%2C+Zhiyuan+Peng%2C+and+Yi+Fang%2E+2024%2E+Evaluating+and+enhancing+large+language+models+for+novelty+assessment+in+scholarly+publications%2E+ArXiv%2C+abs%2F2409%2E16605%2E)
21. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob N. Foerster, Jeff Clune, and David Ha. 2024. [The ai scientist: Towards fully automated open-ended scientific discovery](https://md.research.elamur.ai/free/en/papers/lu-2024-ai-scientist-fully-automated/). ArXiv, abs/2408.06292.
22. Sheng Lu, Ilia Kuznetsov, and Iryna Gurevych. 2025. Identifying aspects in peer reviews. arXiv preprint arXiv:2504.06910. [find](https://scholar.google.com/scholar?q=Sheng+Lu%2C+Ilia+Kuznetsov%2C+and+Iryna+Gurevych%2E+2025%2E+Identifying+aspects+in+peer+reviews%2E+arXiv+preprint+arXiv%3A2504%2E06910%2E)
23. Mehrnaz Mostafapour, Jacqueline H Fortier, Karen Pacheco, Heather Murray, and Gary Garber. 2024. Evaluating literature reviews conducted by humans versus chatgpt: Comparative study. Jmir ai, 3:e56537. [find](https://scholar.google.com/scholar?q=Mehrnaz+Mostafapour%2C+Jacqueline+H+Fortier%2C+Karen+Pacheco%2C+Heather+Murray%2C+and+Gary+Garber%2E+2024%2E+Evaluating+literature+reviews+conducted+by+humans+versus+chatgpt%3A+Comparative+study%2E+Jmir+ai%2C+3%3Ae56537%2E)
24. Lakshmi Ramachandran, Edward F Gehringer, and Ravi K Yadav. 2017. Automated assessment of the quality of peer reviews using natural language processing techniques. International Journal of Artificial Intelligence in Education, 27(3):534–581. [find](https://scholar.google.com/scholar?q=Lakshmi+Ramachandran%2C+Edward+F+Gehringer%2C+and+Ravi+K+Yadav%2E+2017%2E+Automated+assessment+of+the+quality+of+peer+reviews+using+natural+language+processing+techniques%2E+International+Journal+of+Artificial+)
25. Zachary Robertson. 2023. Gpt4 is slightly helpful for peer-review assistance: A pilot study. ArXiv, abs/2307.05492. — [arXiv:2307.05492](https://arxiv.org/abs/2307.05492) [find](https://scholar.google.com/scholar?q=Zachary+Robertson%2E+2023%2E+Gpt4+is+slightly+helpful+for+peer%2Dreview+assistance%3A+A+pilot+study%2E+ArXiv%2C+abs%2F2307%2E05492%2E+%E2%80%94+%5BarXiv%3A2307%2E05492%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2307%2E05492%29)
26. Ahmed Saad, Nathan Jenko, Sisith Ariyaratne, Nick Birch, Karthikeyan P Iyengar, Arthur Mark Davies, Raju Vaishya, and Rajesh Botchu. 2024. Exploring the potential of chatgpt in the peer review process: an observational study. Diabetes & Metabolic Syndrome: Clinical Research & Reviews, 18(2):102946. [find](https://scholar.google.com/scholar?q=Ahmed+Saad%2C+Nathan+Jenko%2C+Sisith+Ariyaratne%2C+Nick+Birch%2C+Karthikeyan+P+Iyengar%2C+Arthur+Mark+Davies%2C+Raju+Vaishya%2C+and+Rajesh+Botchu%2E+2024%2E+Exploring+the+potential+of+chatgpt+in+the+peer+review+process)
27. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). arXiv preprint arXiv:2206.04615.
28. Lu Sun, Stone Tao, Junjie Hu, and Steven P. Dow. 2024. Metawriter: Exploring the potential and perils of ai writing support in scientific peer review. Proceedings of the ACM on Human-Computer Interaction, 8:1 – 32. [find](https://scholar.google.com/scholar?q=Lu+Sun%2C+Stone+Tao%2C+Junjie+Hu%2C+and+Steven+P%2E+Dow%2E+2024%2E+Metawriter%3A+Exploring+the+potential+and+perils+of+ai+writing+support+in+scientific+peer+review%2E+Proceedings+of+the+ACM+on+Human%2DComputer+Interact)
29. Carolina Tropini, B Brett Finlay, Mark Nichter, Melissa K Melby, Jessica L Metcalf, Maria Gloria Dominguez-Bello, Liping Zhao, Margaret J McFall-Ngai, Naama Geva-Zatorsky, Katherine R Amato, et al. 2023. Time to rethink academic publishing: the peer reviewer crisis. [find](https://scholar.google.com/scholar?q=Carolina+Tropini%2C+B+Brett+Finlay%2C+Mark+Nichter%2C+Melissa+K+Melby%2C+Jessica+L+Metcalf%2C+Maria+Gloria+Dominguez%2DBello%2C+Liping+Zhao%2C+Margaret+J+McFall%2DNgai%2C+Naama+Geva%2DZatorsky%2C+Katherine+R+Amato%2C+et+al%2E+20)
30. Keith Tyser, Ben Segev, Gaston Longhitano, Xin-Yu Zhang, Zachary Meeks, Jason Lee, Uday Garg, Nicholas Belsten, Avi Shporer, Madeleine Udell, et al. 2024. Ai-driven review systems: evaluating llms in scalable and bias-aware academic reviews. arXiv preprint arXiv:2408.10365. [find](https://scholar.google.com/scholar?q=Keith+Tyser%2C+Ben+Segev%2C+Gaston+Longhitano%2C+Xin%2DYu+Zhang%2C+Zachary+Meeks%2C+Jason+Lee%2C+Uday+Garg%2C+Nicholas+Belsten%2C+Avi+Shporer%2C+Madeleine+Udell%2C+et+al%2E+2024%2E+Ai%2Ddriven+review+systems%3A+evaluating+llms+in+)
31. Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. 2024. Rewards-in-context: Multi-objective alignment of foundation models with dynamic preference adjustment. arXiv preprint arXiv:2402.10207. [find](https://scholar.google.com/scholar?q=Rui+Yang%2C+Xiaoman+Pan%2C+Feng+Luo%2C+Shuang+Qiu%2C+Han+Zhong%2C+Dong+Yu%2C+and+Jianshu+Chen%2E+2024%2E+Rewards%2Din%2Dcontext%3A+Multi%2Dobjective+alignment+of+foundation+models+with+dynamic+preference+adjustment%2E+arXiv+pr)
32. Rui Ye, Xianghe Pang, Jingyi Chai, Jiaao Chen, Zhen fei Yin, Zhen Xiang, Xiaowen Dong, Jing Shao, and Siheng Chen. 2024. Are we there yet? revealing the risks of utilizing large language models in scholarly peer review. ArXiv, abs/2412.01708. [find](https://scholar.google.com/scholar?q=Rui+Ye%2C+Xianghe+Pang%2C+Jingyi+Chai%2C+Jiaao+Chen%2C+Zhen+fei+Yin%2C+Zhen+Xiang%2C+Xiaowen+Dong%2C+Jing+Shao%2C+and+Siheng+Chen%2E+2024%2E+Are+we+there+yet%3F+revealing+the+risks+of+utilizing+large+language+models+in+sch)
33. Weizhe Yuan, Pengfei Liu, and Graham Neubig. 2022. Can we automate scientific reviewing? Journal of Artificial Intelligence Research, 75:171–212. [find](https://scholar.google.com/scholar?q=Weizhe+Yuan%2C+Pengfei+Liu%2C+and+Graham+Neubig%2E+2022%2E+Can+we+automate+scientific+reviewing%3F+Journal+of+Artificial+Intelligence+Research%2C+75%3A171%E2%80%93212%2E)
34. Ruiyang Zhou, Lu Chen, and Kai Yu. 2024. [Is llm a reliable reviewer? a comprehensive evaluation of llm on automatic paper reviewing tasks](https://md.research.elamur.ai/free/en/papers/zhou-2024-is-llm-reliable-reviewer-comprehensive/). In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 9340–9351.

*Updated 30 October 2026*