---
title: "LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing"
description: "This EMNLP 2024 paper asks whether large language models can take on part of the growing… 39 references, cited by 48 papers here."
authors: ["Jiangshu Du", "Yibo Wang", "Wenting Zhao", "Zhongfen Deng", "Shuaiqi Liu", "Renze Lou", "Henry Peng Zou", "Pranav Narayanan Venkit", "Nan Zhang", "Mukund Srinath", "Haoran Ranran Zhang", "Vipul Gupta", "Yinghui Li", "Tao Li", "Fei Wang", "Qin Liu", "Tianlin Liu", "Pengzhi Gao", "Congying Xia", "Chen Xing", "Cheng Jiayang", "Zhaowei Wang", "Ying Su", "Raj Sanjay Shah", "Ruohao Guo", "Jing Gu", "Haoran Li", "Kangda Wei", "Zihao Wang", "Lu Cheng", "Surangika Ranathunga", "Meng Fang", "Jie Fu", "Fei Liu", "Ruihong Huang", "Eduardo Blanco", "Yixin Cao", "Rui Zhang", "Philip S. Yu", "Wenpeng Yin"]
year: 2024
venue: "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2024.emnlp-main.292/"
doi: "10.18653/v1/2024.emnlp-main.292"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/du-2024-llms-assist-nlp-researchers-critique/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/du-2024-llms-assist-nlp-researchers-critique/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing

# LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing

**Authors:** Jiangshu Du, Yibo Wang, [Wenting Zhao](https://md.research.elamur.ai/free/en/authors/wenting-zhao/), Zhongfen Deng, Shuaiqi Liu, [Renze Lou](https://md.research.elamur.ai/free/en/authors/renze-lou/), Henry Peng Zou, Pranav Narayanan Venkit, [Nan Zhang](https://md.research.elamur.ai/free/en/authors/nan-zhang/), Mukund Srinath, Haoran Ranran Zhang, Vipul Gupta, Yinghui Li, Tao Li, [Fei Wang](https://md.research.elamur.ai/free/en/authors/fei-wang/), [Qin Liu](https://md.research.elamur.ai/free/en/authors/qin-liu/), Tianlin Liu, Pengzhi Gao, Congying Xia, Chen Xing, [Cheng Jiayang](https://md.research.elamur.ai/free/en/authors/cheng-jiayang/), Zhaowei Wang, Ying Su, Raj Sanjay Shah, Ruohao Guo, Jing Gu, [Haoran Li](https://md.research.elamur.ai/free/en/authors/haoran-li/), Kangda Wei, Zihao Wang, Lu Cheng, Surangika Ranathunga, Meng Fang, [Jie Fu](https://md.research.elamur.ai/free/en/authors/jie-fu/), [Fei Liu](https://md.research.elamur.ai/free/en/authors/fei-liu/), [Ruihong Huang](https://md.research.elamur.ai/free/en/authors/ruihong-huang/), [Eduardo Blanco](https://md.research.elamur.ai/free/en/authors/eduardo-blanco/), [Yixin Cao](https://md.research.elamur.ai/free/en/authors/yixin-cao/), [Rui Zhang](https://md.research.elamur.ai/free/en/authors/rui-zhang/), [Philip S. Yu](https://md.research.elamur.ai/free/en/authors/philip-s-yu/) and [Wenpeng Yin](https://md.research.elamur.ai/free/en/authors/wenpeng-yin/) · **Year:** 2024 · **Venue:** [Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2024](https://md.research.elamur.ai/free/en/venues/emnlp/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.emnlp-main.292/) · [PDF](https://aclanthology.org/2024.emnlp-main.292.pdf) · [DOI 10.18653/v1/2024.emnlp-main.292](https://doi.org/10.18653/v1/2024.emnlp-main.292) · [OpenAlex](https://openalex.org/W4404782918)

**Citations:** 23 citations (OpenAlex, 2026-10-14) · [48](https://md.research.elamur.ai/free/en/papers/du-2024-llms-assist-nlp-researchers-critique/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary

This EMNLP 2024 paper asks whether large language models can take on part of the growing peer-review burden in NLP, both as reviewers and as meta-reviewers who audit individual reviews. The authors are explicit that they are not endorsing LLM reviewing. Their goals are to help detect hidden LLM use and to show the limits of current models.

They built ReviewCritique from NLP submissions on OpenReview, using the pre-deadline versions together with their human reviews, rebuttals and meta-reviews. For a balanced subset, they added reviews generated by three commercial LLMs. Experienced NLP researchers then labeled every review sentence for deficiency, such as factual errors, unsupported criticism, non-constructive or unprofessional remarks, and explained each label.

LLM-written reviews contained more deficient segments than human ones. They also tended to be generic and repetitive across papers, which the authors measured with a new diversity metric. When asked to spot flawed segments in human reviews, all tested models scored low F1, and their explanations matched experts only weakly. The resource gives a fine-grained, expert-annotated benchmark for AI-assisted peer review.

## Key points

- **Data source:** 100 NLP papers submitted to ICLR and NeurIPS (2020–2023), split equally between accepted and rejected. Each paper comes with 3–5 human reviews, author rebuttals and meta-reviews.
- **LLM-generated reviews:** For a 20-paper subset, GPT-4, Gemini-1.5 and Claude Opus each produced three reviews. The prompts used ICLR guidelines, example reviews and an ICLR 2024 template.
- **Annotation:** 40 NLP researchers with multiple first-author top-venue papers labeled reviews at the sentence level, with explanations. Many annotators hold PhDs or have area-chair experience.
- **Meta-reviewing results:** The best F1 for detecting deficient segments was only about 22, reached by Claude Opus with an ensemble prompt. Llama3-8B, Llama3-70B and Qwen2-72B scored lower. For explanations, Claude Opus again led, with ROUGE-1 of 20.18 and BERTScore of 57.28.
- **Hardest deficiency types for LLMs:** LLMs had the most trouble with six categories: experiment-related problems, writing-related problems, contradictions, shallow (superficial) reviewing, unstated statements, and summaries that misrepresent the paper. The dataset is released at github.com/jiangshdd/ReviewCritique.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Southern California](https://md.research.elamur.ai/free/en/universities/university-of-southern-california/) — University, Los Angeles, United States · 104 works · 33,876 citations
- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 137 works · 83,262 citations
- [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) — University, Shanghai, China · 65 works · 19,402 citations
- [Georgia Institute of Technology](https://md.research.elamur.ai/free/en/universities/georgia-institute-of-technology/) — University, Atlanta, United States · 69 works · 22,788 citations
- [Salesforce](https://md.research.elamur.ai/free/en/companies/salesforce/) — Company, San Francisco, United States · 37 works · 15,019 citations
- [Singapore University of Technology and Design](https://md.research.elamur.ai/free/en/universities/singapore-university-of-technology-and-design/) — University, Singapore, Singapore · 33 works · 10,760 citations
- [University of Illinois Chicago](https://md.research.elamur.ai/free/en/universities/university-of-illinois-chicago/) — University, Chicago, United States · 38 works · 19,844 citations
- [Hong Kong Polytechnic University](https://md.research.elamur.ai/free/en/universities/hong-kong-polytechnic-university/) — University, Hong Kong, China · 34 works · 4,476 citations
- [Pennsylvania State University](https://md.research.elamur.ai/free/en/universities/pennsylvania-state-university/) — University, State College, United States · 22 works · 4,878 citations
- [University of California, Santa Cruz](https://md.research.elamur.ai/free/en/universities/university-of-california-santa-cruz/) — University, Santa Cruz, United States · 24 works · 3,112 citations
- [Emory University](https://md.research.elamur.ai/free/en/universities/emory-university/) — University, Atlanta, United States · 16 works · 39,535 citations
- [University of Arizona](https://md.research.elamur.ai/free/en/universities/university-of-arizona/) — University, Tucson, United States · 19 works · 10,269 citations
- [Texas A&M University](https://md.research.elamur.ai/free/en/universities/texas-a-m-university/) — University, College Station, United States · 13 works · 1,683 citations
- [University of California, Davis](https://md.research.elamur.ai/free/en/universities/university-of-california-davis/) — University, Davis, United States · 13 works · 41,713 citations
- [University of Liverpool](https://md.research.elamur.ai/free/en/universities/university-of-liverpool/) — University, Liverpool, United Kingdom · 7 works · 539 citations
- [Xiaomi](https://md.research.elamur.ai/free/en/companies/xiaomi/) — Company, Beijing, China · 3 works · 301 citations

Also on the byline: [University of Basel](https://ror.org/02s6k3f65) · [Massey University](https://ror.org/052czxv31).

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 744 works · [🤝 General Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/general-social-sciences/) · 87 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works

## Cite

Cite as: Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, Haoran Ranran Zhang, Vipul Gupta, Yinghui Li, Tao Li, Fei Wang, Qin Liu, Tianlin Liu, Pengzhi Gao, Congying Xia, Chen Xing, Cheng Jiayang, Zhaowei Wang, Ying Su, Raj Sanjay Shah, Ruohao Guo, Jing Gu, Haoran Li, Kangda Wei, Zihao Wang, Lu Cheng, Surangika Ranathunga, Meng Fang, Jie Fu, Fei Liu, Ruihong Huang, Eduardo Blanco, Yixin Cao, Rui Zhang, Philip S. Yu and Wenpeng Yin. 2024. LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{du2024llms,
  title = {{LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing}},
  author = {Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, Haoran Ranran Zhang, Vipul Gupta, Yinghui Li, Tao Li, Fei Wang, Qin Liu, Tianlin Liu, Pengzhi Gao, Congying Xia, Chen Xing, Cheng Jiayang, Zhaowei Wang, Ying Su, Raj Sanjay Shah, Ruohao Guo, Jing Gu, Haoran Li, Kangda Wei, Zihao Wang, Lu Cheng, Surangika Ranathunga, Meng Fang, Jie Fu, Fei Liu, Ruihong Huang, Eduardo Blanco, Yixin Cao, Rui Zhang, Philip S. Yu and Wenpeng Yin},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.292},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.292}
}
```

- [← previous Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA](https://md.research.elamur.ai/free/en/papers/wang-2024-leave-no-document-behind-benchmarking/)
- [next → Towards Safer Large Language Models through Machine Unlearning](https://md.research.elamur.ai/free/en/papers/liu-2024-safer-large-language-models-through/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

39 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. AI@Meta. 2024. Llama 3 model card. [find](https://scholar.google.com/scholar?q=AI%40Meta%2E+2024%2E+Llama+3+model+card%2E)
2. Signe Altmäe, Alberto Sola-Leyva, and Andres Salumets. 2023. Artificial intelligence in scientific writing: a friend or a foe? Reproductive BioMedicine Online, 47(1):3–9. [find](https://scholar.google.com/scholar?q=Signe+Altm%C3%A4e%2C+Alberto+Sola%2DLeyva%2C+and+Andres+Salumets%2E+2023%2E+Artificial+intelligence+in+scientific+writing%3A+a+friend+or+a+foe%3F+Reproductive+BioMedicine+Online%2C+47%281%29%3A3%E2%80%939%2E)
3. Anthropic. 2024. Introducing the next generation of claude. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2024%2E+Introducing+the+next+generation+of+claude%2E)
4. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, P… — [Qwen Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/)
5. Zhongfen Deng, Hao Peng, Congying Xia, Jianxin Li, Lifang He, and S Yu Philip. 2020. Hierarchical bidirectional self-attention networks for paper review rating recommendation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6302–6314. [find](https://scholar.google.com/scholar?q=Zhongfen+Deng%2C+Hao+Peng%2C+Congying+Xia%2C+Jianxin+Li%2C+Lifang+He%2C+and+S+Yu+Philip%2E+2020%2E+Hierarchical+bidirectional+self%2Dattention+networks+for+paper+review+rating+recommendation%2E+In+Proceedings+of+the+28)
6. Nils Dycke, Ilia Kuznetsov, and Iryna Gurevych. 2023. NLPeer: A unified resource for the computational study of peer review. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pages 5049–5073. [find](https://scholar.google.com/scholar?q=Nils+Dycke%2C+Ilia+Kuznetsov%2C+and+Iryna+Gurevych%2E+2023%2E+NLPeer%3A+A+unified+resource+for+the+computational+study+of+peer+review%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computatio)
7. Zhaolin Gao, Kianté Brantley, and Thorsten Joachims. 2024. Reviewer2: Optimizing review generation through prompt generation. arXiv preprint arXiv:2402.10886. [find](https://scholar.google.com/scholar?q=Zhaolin+Gao%2C+Kiant%C3%A9+Brantley%2C+and+Thorsten+Joachims%2E+2024%2E+Reviewer2%3A+Optimizing+review+generation+through+prompt+generation%2E+arXiv+preprint+arXiv%3A2402%2E10886%2E)
8. Tirthankar Ghosal, Sandeep Kumar, Prabhat Kumar Bharti, and Asif Ekbal. 2022a. Peer review analyze: A novel benchmark resource for computational analysis of peer reviews. Plos one, 17(1):e0259238. [find](https://scholar.google.com/scholar?q=Tirthankar+Ghosal%2C+Sandeep+Kumar%2C+Prabhat+Kumar+Bharti%2C+and+Asif+Ekbal%2E+2022a%2E+Peer+review+analyze%3A+A+novel+benchmark+resource+for+computational+analysis+of+peer+reviews%2E+Plos+one%2C+17%281%29%3Ae0259238%2E)
9. Tirthankar Ghosal, Kamal Kaushik Varanasi, and Valia Kordoni. 2022b. Hedgepeer: A dataset for uncertainty detection in peer reviews. In Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries, pages 1–5. [find](https://scholar.google.com/scholar?q=Tirthankar+Ghosal%2C+Kamal+Kaushik+Varanasi%2C+and+Valia+Kordoni%2E+2022b%2E+Hedgepeer%3A+A+dataset+for+uncertainty+detection+in+peer+reviews%2E+In+Proceedings+of+the+22nd+ACM%2FIEEE+Joint+Conference+on+Digital+Lib)
10. Gemini Team Google. 2023. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805.
11. Yanzhu Guo, Guokan Shang, Virgile Rennard, Michalis Vazirgiannis, and Chloé Clavel. 2023. Automatic analysis of substantiation in scientific peer reviews. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10198–10216. [find](https://scholar.google.com/scholar?q=Yanzhu+Guo%2C+Guokan+Shang%2C+Virgile+Rennard%2C+Michalis+Vazirgiannis%2C+and+Chlo%C3%A9+Clavel%2E+2023%2E+Automatic+analysis+of+substantiation+in+scientific+peer+reviews%2E+In+Findings+of+the+Association+for+Computatio)
12. Xinyu Hua, Mitko Nikolov, Nikhil Badugu, and Lu Wang. 2019. Argument mining for understanding peer reviews. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2131–2137. [find](https://scholar.google.com/scholar?q=Xinyu+Hua%2C+Mitko+Nikolov%2C+Nikhil+Badugu%2C+and+Lu+Wang%2E+2019%2E+Argument+mining+for+understanding+peer+reviews%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+Association+for+C)
13. Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. 2023. [Challenges and applications of large language models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/). arXiv preprint arXiv:2307.10169.
14. Prabhat Kumar Bharti, Meith Navlakha, Mayank Agarwal, and Asif Ekbal. 2023. Politepeer: does peer review hurt? a dataset to gauge politeness intensity in the peer reviews. Language Resources and Evaluation, pages 1–23. [find](https://scholar.google.com/scholar?q=Prabhat+Kumar+Bharti%2C+Meith+Navlakha%2C+Mayank+Agarwal%2C+and+Asif+Ekbal%2E+2023%2E+Politepeer%3A+does+peer+review+hurt%3F+a+dataset+to+gauge+politeness+intensity+in+the+peer+reviews%2E+Language+Resources+and+Evalu)
15. Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018. [A dataset of peer reviews (PeerRead): Collection, insights and NLP applications](https://md.research.elamur.ai/free/en/papers/kang-2018-dataset-peer-reviews-peerread-collection/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Comp…
16. Liying Cheng, Lidong Bing, Qian Yu, Wei Lu, and Luo Si. 2020. Ape: Argument pair extraction from peer review and rebuttal via multi-task learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7000–7011. [find](https://scholar.google.com/scholar?q=Liying+Cheng%2C+Lidong+Bing%2C+Qian+Yu%2C+Wei+Lu%2C+and+Luo+Si%2E+2020%2E+Ape%3A+Argument+pair+extraction+from+peer+review+and+rebuttal+via+multi%2Dtask+learning%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Me)
17. Neha Nayak Kennard, Tim O’Gorman, Rajarshi Das, Akshay Sharma, Chhandak Bagchi, Matthew Clinton, Pranay Kumar Yelugam, Hamed Zamani, and Andrew McCallum. 2022. DISAPERE: A dataset for discourse structure in peer review discussions. In Proceedings of the 2022 Conference of the North American Chapter… [find](https://scholar.google.com/scholar?q=Neha+Nayak+Kennard%2C+Tim+O%E2%80%99Gorman%2C+Rajarshi+Das%2C+Akshay+Sharma%2C+Chhandak+Bagchi%2C+Matthew+Clinton%2C+Pranay+Kumar+Yelugam%2C+Hamed+Zamani%2C+and+Andrew+McCallum%2E+2022%2E+DISAPERE%3A+A+dataset+for+discourse+struct)
18. Sandeep Kumar, Tirthankar Ghosal, and Asif Ekbal. 2023. When reviewers lock horns: Finding disagreements in scientific peer reviews. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 16693–16704. [find](https://scholar.google.com/scholar?q=Sandeep+Kumar%2C+Tirthankar+Ghosal%2C+and+Asif+Ekbal%2E+2023%2E+When+reviewers+lock+horns%3A+Finding+disagreements+in+scientific+peer+reviews%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+Natur)
19. Jiyi Li, Ayaka Sato, Kazuya Shimura, and Fumiyo Fukumoto. 2020. Multi-task peer-review score prediction. In Proceedings of the First Workshop on Scholarly Document Processing, pages 121–126. [find](https://scholar.google.com/scholar?q=Jiyi+Li%2C+Ayaka+Sato%2C+Kazuya+Shimura%2C+and+Fumiyo+Fukumoto%2E+2020%2E+Multi%2Dtask+peer%2Dreview+score+prediction%2E+In+Proceedings+of+the+First+Workshop+on+Scholarly+Document+Processing%2C+pages+121%E2%80%93126%2E)
20. Miao Li, Eduard Hovy, and Jey Lau. 2023. Summarizing multiple documents with conversational structure for meta-review generation. Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7089–7112. [find](https://scholar.google.com/scholar?q=Miao+Li%2C+Eduard+Hovy%2C+and+Jey+Lau%2E+2023%2E+Summarizing+multiple+documents+with+conversational+structure+for+meta%2Dreview+generation%2E+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2023%2C)
21. Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Smith, Yian Yin, et al. 2023. Can large language models provide useful feedback on research papers? a large-scale empirical analysis. arXiv preprint arXiv:2310.01783. [find](https://scholar.google.com/scholar?q=Weixin+Liang%2C+Yuhui+Zhang%2C+Hancheng+Cao%2C+Binglu+Wang%2C+Daisy+Ding%2C+Xinyu+Yang%2C+Kailas+Vodrahalli%2C+Siyu+He%2C+Daniel+Smith%2C+Yian+Yin%2C+et+al%2E+2023%2E+Can+large+language+models+provide+useful+feedback+on+rese)
22. Chin-Yew Lin. 2004. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out, pages 74–81.
23. Jialiang Lin, Jiaxin Song, Zhangping Zhou, Yidong Chen, and Xiaodong Shi. 2023a. Automated scholarly paper review: Concepts, technologies, and challenges. Information Fusion, 98:101830. [find](https://scholar.google.com/scholar?q=Jialiang+Lin%2C+Jiaxin+Song%2C+Zhangping+Zhou%2C+Yidong+Chen%2C+and+Xiaodong+Shi%2E+2023a%2E+Automated+scholarly+paper+review%3A+Concepts%2C+technologies%2C+and+challenges%2E+Information+Fusion%2C+98%3A101830%2E)
24. Jialiang Lin, Jiaxin Song, Zhangping Zhou, Yidong Chen, and Xiaodong Shi. 2023b. Moprd: A multidisciplinary open peer review dataset. Neural Computing and Applications, 35(34):24191–24206. [find](https://scholar.google.com/scholar?q=Jialiang+Lin%2C+Jiaxin+Song%2C+Zhangping+Zhou%2C+Yidong+Chen%2C+and+Xiaodong+Shi%2E+2023b%2E+Moprd%3A+A+multidisciplinary+open+peer+review+dataset%2E+Neural+Computing+and+Applications%2C+35%2834%29%3A24191%E2%80%9324206%2E)
25. Ryan Liu and Nihar B Shah. 2023. Reviewergpt? an exploratory study on using large language models for paper reviewing. arXiv preprint arXiv:2306.00622. [find](https://scholar.google.com/scholar?q=Ryan+Liu+and+Nihar+B+Shah%2E+2023%2E+Reviewergpt%3F+an+exploratory+study+on+using+large+language+models+for+paper+reviewing%2E+arXiv+preprint+arXiv%3A2306%2E00622%2E)
26. Jesse G Meyer, Ryan J Urbanowicz, Patrick CN Martin, Karen O’Connor, Ruowang Li, Pei-Chen Peng, Tiffani J Bright, Nicholas Tatonetti, Kyoung Jae Won, Graciela Gonzalez-Hernandez, et al. 2023. Chatgpt and large language models in academia: opportunities and challenges. BioData Mining, 16(1):20. [find](https://scholar.google.com/scholar?q=Jesse+G+Meyer%2C+Ryan+J+Urbanowicz%2C+Patrick+CN+Martin%2C+Karen+O%E2%80%99Connor%2C+Ruowang+Li%2C+Pei%2DChen+Peng%2C+Tiffani+J+Bright%2C+Nicholas+Tatonetti%2C+Kyoung+Jae+Won%2C+Graciela+Gonzalez%2DHernandez%2C+et+al%2E+2023%2E+Chatgpt+)
27. Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Nick Barnes, and Ajmal Mian. 2023. [A comprehensive overview of large language models](https://md.research.elamur.ai/free/en/papers/naveed-2023-comprehensive-overview-large-language-models/). arXiv preprint arXiv:2307.06435.
28. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
29. Tribikram Pradhan, Chaitanya Bhatia, Prashant Kumar, and Sukomal Pal. 2021. A deep neural architecture based meta-review generation and final decision prediction of a scholarly article. Neurocomputing, 428:218–238. [find](https://scholar.google.com/scholar?q=Tribikram+Pradhan%2C+Chaitanya+Bhatia%2C+Prashant+Kumar%2C+and+Sukomal+Pal%2E+2021%2E+A+deep+neural+architecture+based+meta%2Dreview+generation+and+final+decision+prediction+of+a+scholarly+article%2E+Neurocomputing)
30. Zafaryab Rasool, Stefanus Kurniawan, Sherwin Balugo, Scott Barnett, Rajesh Vasa, Courtney Chesser, Benjamin M Hampstead, Sylvie Belleville, Kon Mouzakis, and Alex Bahar-Fuchs. 2024. Evaluating llms on document-based qa: Exact answer selection and numerical extraction using cogtale dataset. Natural L… [find](https://scholar.google.com/scholar?q=Zafaryab+Rasool%2C+Stefanus+Kurniawan%2C+Sherwin+Balugo%2C+Scott+Barnett%2C+Rajesh+Vasa%2C+Courtney+Chesser%2C+Benjamin+M+Hampstead%2C+Sylvie+Belleville%2C+Kon+Mouzakis%2C+and+Alex+Bahar%2DFuchs%2E+2024%2E+Evaluating+llms+on)
31. Nils Reimers and Iryna Gurevych. 2019. [Sentence-bert: Sentence embeddings using siamese bert-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3980–3990.
32. Zachary Robertson. 2023. Gpt4 is slightly helpful for peer-review assistance: A pilot study. arXiv preprint arXiv:2307.05492. — [arXiv:2307.05492](https://arxiv.org/abs/2307.05492) [find](https://scholar.google.com/scholar?q=Zachary+Robertson%2E+2023%2E+Gpt4+is+slightly+helpful+for+peer%2Dreview+assistance%3A+A+pilot+study%2E+arXiv+preprint+arXiv%3A2307%2E05492%2E+%E2%80%94+%5BarXiv%3A2307%2E05492%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2307%2E05492%29)
33. Federico Ruggeri, Mohsen Mesgar, and Iryna Gurevych. 2023. A dataset of argumentative dialogues on scientific papers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pages 7684–7699. [find](https://scholar.google.com/scholar?q=Federico+Ruggeri%2C+Mohsen+Mesgar%2C+and+Iryna+Gurevych%2E+2023%2E+A+dataset+of+argumentative+dialogues+on+scientific+papers%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Lin)
34. Chenhui Shen, Liying Cheng, Ran Zhou, Lidong Bing, Yang You, and Luo Si. 2022. MReD: A meta-review dataset for structure-controllable text generation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2521–2535. [find](https://scholar.google.com/scholar?q=Chenhui+Shen%2C+Liying+Cheng%2C+Ran+Zhou%2C+Lidong+Bing%2C+Yang+You%2C+and+Luo+Si%2E+2022%2E+MReD%3A+A+meta%2Dreview+dataset+for+structure%2Dcontrollable+text+generation%2E+In+Findings+of+the+Association+for+Computational+)
35. Qingyun Wang, Qi Zeng, Lifu Huang, Kevin Knight, Heng Ji, and Nazneen Fatema Rajani. 2020. ReviewRobot: Explainable paper review generation based on knowledge synthesis. In Proceedings of the 13th International Conference on Natural Language Generation, pages 384–397. [find](https://scholar.google.com/scholar?q=Qingyun+Wang%2C+Qi+Zeng%2C+Lifu+Huang%2C+Kevin+Knight%2C+Heng+Ji%2C+and+Nazneen+Fatema+Rajani%2E+2020%2E+ReviewRobot%3A+Explainable+paper+review+generation+based+on+knowledge+synthesis%2E+In+Proceedings+of+the+13th+Int)
36. Wenting Xiong and Diane Litman. 2011. Automatically predicting peer-review helpfulness. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 502–507. [find](https://scholar.google.com/scholar?q=Wenting+Xiong+and+Diane+Litman%2E+2011%2E+Automatically+predicting+peer%2Dreview+helpfulness%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+Human+Language+Techn)
37. Weizhe Yuan, Pengfei Liu, and Graham Neubig. 2022. Can we automate scientific reviewing? Journal of Artificial Intelligence Research, 75:171–212. [find](https://scholar.google.com/scholar?q=Weizhe+Yuan%2C+Pengfei+Liu%2C+and+Graham+Neubig%2E+2022%2E+Can+we+automate+scientific+reviewing%3F+Journal+of+Artificial+Intelligence+Research%2C+75%3A171%E2%80%93212%2E)
38. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+In+International+Conference+on+Learning+Representations%2E)
39. Ruiyang Zhou, Lu Chen, and Kai Yu. 2024. [Is llm a reliable reviewer? a comprehensive evaluation of llm on automatic paper reviewing tasks](https://md.research.elamur.ai/free/en/papers/zhou-2024-is-llm-reliable-reviewer-comprehensive/). In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, pages 9340–9351.

*Updated 24 October 2026*