---
title: "Automatic Reviewers Fail to Detect Faulty Reasoning in"
description: "They then build a fully automatic counterfactual pipeline with three steps. 69 references, cited by 4 papers here."
authors: ["Nils Dycke", "Iryna Gurevych"]
year: 2026
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/2026.tacl-1.22/"
doi: "10.1162/tacl.a.642"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/dycke-2026-automatic-reviewers-fail-detect-faulty/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/dycke-2026-automatic-reviewers-fail-detect-faulty/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/3/) / Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework

# Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework

**Authors:** [Nils Dycke](https://md.research.elamur.ai/free/en/authors/nils-dycke/) and [Iryna Gurevych](https://md.research.elamur.ai/free/en/authors/iryna-gurevych/) · **Year:** 2026 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2026](https://md.research.elamur.ai/free/en/venues/tacl/2026/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.tacl-1.22/) · [PDF](https://aclanthology.org/2026.tacl-1.22.pdf) · [DOI 10.1162/tacl.a.642](https://doi.org/10.1162/tacl.a.642) · [OpenAlex](https://openalex.org/W4415990061)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · 4 in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLMs are increasingly used as automatic review generators (ARGs), but studies of their quality disagree. One reason is that these evaluations mix many reviewing skills and often ignore publication context. Dycke and Gurevych isolate one skill that depends only on the paper itself: checking whether results, conclusions and claimed findings actually support each other. They formalize soundness as a hierarchical "research logic" graph that links method, results, conclusions and findings. They then build a fully automatic counterfactual pipeline with three steps. First, an LLM extracts this structure from accepted papers. Second, it makes targeted edits that break a support relation, for example by over-claiming. Third, reviews of the original and edited versions are compared statistically. Applied to recent AI/NLP papers, the framework shows that the tested ARGs' reviews are not significantly changed by the injected flaws. Instead, the ARGs react to surface-level features, which points to shallow heuristics rather than real reasoning. This raises serious doubts about relying on ARGs in peer review without supervision.

## Key points
- Dataset: counterfactual versions of nearly 150 full papers from ACL, EMNLP, NeurIPS and ICLR. The authors contrast this with only 13 short essays in Liu and Shah (2023).
- Edits target only soundness. Figures are not altered; captions and surrounding text are edited instead, so the study covers text-only (unimodal) ARGs.
- Effects are measured as relative differences between original and counterfactual reviews, tested at α = 0.05. This avoids needing an absolute notion of review quality, and a validation check confirmed that the pipeline can detect such effects.
- The authors argue that earlier reports of about 40% error discovery for GPT-4o-mini mostly reflect knowledge recall, not reasoning.
- Three recommendations follow: test distinct reviewing skills separately, foster human–LLM collaboration, and account for sensitivity to surface features. Code and data are at github.com/UKPLab/tacl2026-counter-review-logic. Closed commercial models were used, so all model outputs are released for replicability.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Technische Universität Darmstadt](https://md.research.elamur.ai/free/en/universities/technische-universitat-darmstadt/) — University, Darmstadt, Germany · 109 works · 35,920 citations
- [National Research Center for Applied Cybersecurity ATHENE](https://md.research.elamur.ai/free/en/labs/national-research-center-for-applied-cybersecurity-athene/) — Research institute, Darmstadt, Germany · 3 works · 17 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🎭 History and Philosophy of Science](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/history-and-philosophy-of-science/) · 53 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Nils Dycke and Iryna Gurevych. 2026. Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{dycke2026automatic,
  title = {{Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework}},
  author = {Nils Dycke and Iryna Gurevych},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.1162/tacl.a.642},
  url = {https://doi.org/10.1162/tacl.a.642}
}
```

- [← previous Attention Sinks in Diffusion Language Models](https://md.research.elamur.ai/free/en/papers/rulli-2026-attention-sinks-diffusion-language-models/)
- [next → AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discovery](https://md.research.elamur.ai/free/en/papers/li-2026-autosota-end-end-automated-research/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/3/)

## References

69 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Sal… — [Phi-4 Technical Report](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-4-technical-report/)
2. Lutz Bornmann. 2011. Scientific peer review. Annual Review of Information Science and Technology, 45(1):197–245. https://doi.org/10.1002/aris.2011.1440450112 — [doi:10.1002/aris.2011.1440450112](https://doi.org/10.1002/aris.2011.1440450112) [↗](https://doi.org/10.1002/aris.2011.1440450112)
3. Eric Chamoun, Michael Schlichtkrull, and Andreas Vlachos. 2024. Automated focused feedback generation for scientific writing assistance. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9742–9763, Bangkok, Thailand. Association for Computational Linguistics. https://doi… [find](https://scholar.google.com/scholar?q=Eric+Chamoun%2C+Michael+Schlichtkrull%2C+and+Andreas+Vlachos%2E+2024%2E+Automated+focused+feedback+generation+for+scientific+writing+assistance%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+A)
4. Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab, and Marjan Ghazvininejad. 2022. A review on language models as knowledge bases. arXiv preprint arXiv:2204.06031v1. [find](https://scholar.google.com/scholar?q=Badr+AlKhamissi%2C+Millicent+Li%2C+Asli+Celikyilmaz%2C+Mona+Diab%2C+and+Marjan+Ghazvininejad%2E+2022%2E+A+review+on+language+models+as+knowledge+bases%2E+arXiv+preprint+arXiv%3A2204%2E06031v1%2E)
5. Anthropic. 2024. Claude 3.5 sonnet. [https://www.anthropic.com/news/claude-3-5](https://www.anthropic.com/news/claude-3-5) -sonnet. Accessed: 2025-11-05. [↗](https://www.anthropic.com/news/claude-3-5)
6. Yuan Chang, Ziyue Li, Hengyuan Zhang, Yuanbo Kong, Yanru Wu, Hayden Kwok-Hay So, Zhijiang Guo, Liya Zhu, and Ngai Wong. 2025. [TreeReview: A dynamic tree of questions framework for deep and efficient LLM-based scientific peer review](https://md.research.elamur.ai/free/en/papers/chang-2025-treereview-dynamic-tree-questions-framework/). In Proceedings of the 2025 Conference on Empirical Methods in Natur…
7. John Scott Armstrong and Kesten C. Green. 2022. The Scientific Method: A Guide to Finding Useful Knowledge. Cambridge University Press. https://doi.org/10.1017/9781009092265 [↗](https://doi.org/10.1017/9781009092265)
8. Association for the Advancement of Artificial Intelligence. 2025. AAAI launches AI-powered peer review assessment system. [https://aaai.org/aaai-launches-ai](https://aaai.org/aaai-launches-ai) -powered-peer-review-assessment -system/. Accessed: 2025-07-18. [↗](https://aaai.org/aaai-launches-ai)
9. Francis Bacon. 1878. Clarendon press. [find](https://scholar.google.com/scholar?q=Francis+Bacon%2E+1878%2E+Clarendon+press%2E)
10. Yoav Benjamini and Yosef Hochberg. 1995. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1):289–300. https://doi.org/10.1111/j.2517-6161.1995 .tb02031.x Edward M. Bennett, Renee Alpe… [↗](https://doi.org/10.1111/j.2517-6161.1995)
11. Public Opinion Changyu Chen, Xiting Wang, Ting-En Lin, Ang Lv, Yuchuan Wu, Xin Gao, Ji-Rong Wen, Rui Yan, and Yongbin Li. 2024. Masked thought: Simply masking partial reasoning steps can improve mathematical reasoning learning of language models. In Proceedings of the 62nd Annual Meeting of the Asso… [find](https://scholar.google.com/scholar?q=Public+Opinion+Changyu+Chen%2C+Xiting+Wang%2C+Ting%2DEn+Lin%2C+Ang+Lv%2C+Yuchuan+Wu%2C+Xin+Gao%2C+Ji%2DRong+Wen%2C+Rui+Yan%2C+and+Yongbin+Li%2E+2024%2E+Masked+thought%3A+Simply+masking+partial+reasoning+steps+can+improve+mathe)
12. Mike D’Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. 2024. MARG: Multi-agent review generation for scientific papers. arXiv preprint arXiv:2401.04259v1. [find](https://scholar.google.com/scholar?q=Mike+D%E2%80%99Arcy%2C+Tom+Hope%2C+Larry+Birnbaum%2C+and+Doug+Downey%2E+2024%2E+MARG%3A+Multi%2Dagent+review+generation+for+scientific+papers%2E+arXiv+preprint+arXiv%3A2401%2E04259v1%2E)
13. Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui… [find](https://scholar.google.com/scholar?q=Ma%2C+Yuting+Yan%2C+Yuxiang+Luo%2C+Yuxiang+You%2C+Yuxuan+Liu%2C+Yuyang+Zhou%2C+Z%2E+F%2E+Wu%2C+Z%2E+Z%2E+Ren%2C+Zehui+Ren%2C+Zhangli+Sha%2C+Zhe+Fu%2C+Zhean+Xu%2C+Zhen+Huang%2C+Zhen+Zhang%2C+Zhenda+Xie%2C+Zhengyan+Zhang%2C+Zhewen+Hao%2C+Zhibin)
14. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, H… — [LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing](https://md.research.elamur.ai/free/en/papers/du-2024-llms-assist-nlp-researchers-critique/)
15. Subhabrata Dutta, Timo Kaufmann, Goran Glavaš, Ivan Habernal, Kristian Kersting, Frauke Kreuter, Mira Mezini, Iryna Gurevych, Eyke Hüllermeier, and Hinrich Schuetze. 2025. [Problem solving through human–AI preference-based cooperation](https://md.research.elamur.ai/free/en/papers/dutta-2025-problem-solving-through-human-ai/). Computational Linguistics, pages 1–36. [https://doi.org](https://doi.org) /10.1162… [↗](https://doi.org)
16. Nils Dycke, Ilia Kuznetsov, and Iryna Gurevych. 2023a. [NLPeer: A unified resource for the computational study of peer review](https://md.research.elamur.ai/free/en/papers/dycke-2023-nlpeer-unified-resource-computational-study/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5049–5073, Toronto, Canada. Association for Computat…
17. Nils Dycke, Ilia Kuznetsov, and Iryna Gurevych. 2023b. [Overview of PragTag-2023: Low-resource multi-domain pragmatic tagging of peer reviews](https://md.research.elamur.ai/free/en/papers/dycke-2023-overview-pragtag-2023-low-resource/). In Proceedings of the 10th Workshop on Argument Mining, pages 187–196, Singapore. Association for Computational Linguistics. https://doi .org/10.18653/v1/2023…
18. Yash Goyal, Amir Feder, Uri Shalit, and Been Kim. 2019. Explaining classifiers with causal concept effect (CaCE). arXiv preprint arXiv: 1907.07165v2. — [arXiv:1907.07165](https://arxiv.org/abs/1907.07165) [find](https://scholar.google.com/scholar?q=Yash+Goyal%2C+Amir+Feder%2C+Uri+Shalit%2C+and+Been+Kim%2E+2019%2E+Explaining+classifiers+with+causal+concept+effect+%28CaCE%29%2E+arXiv+preprint+arXiv%3A+1907%2E07165v2%2E+%E2%80%94+%5BarXiv%3A1907%2E07165%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1907%2E07)
19. Nils Dycke, Matej Zečević, Ilia Kuznetsov, Beatrix Suess, Kristian Kersting, and Iryna Gurevych. 2025. STRICTA: Structured reasoning in critical text assessment for peer review and beyond. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa… [find](https://scholar.google.com/scholar?q=Nils+Dycke%2C+Matej+Zec%CC%8Cevic%CC%81%2C+Ilia+Kuznetsov%2C+Beatrix+Suess%2C+Kristian+Kersting%2C+and+Iryna+Gurevych%2E+2025%2E+STRICTA%3A+Structured+reasoning+in+critical+text+assessment+for+peer+review+and+beyond%2E+In+Procee)
20. Xi Fang, Weijie Xu, Fiona Anting Tan, Ziqing Hu, Jiani Zhang, Yanjun Qi, Srinivasan H. Sengamedu, and Christos Faloutsos. 2024. Large language models (LLMs) on tabular data: Prediction, generation, and understanding - a survey. Transactions on Machine Learning Research. [find](https://scholar.google.com/scholar?q=Xi+Fang%2C+Weijie+Xu%2C+Fiona+Anting+Tan%2C+Ziqing+Hu%2C+Jiani+Zhang%2C+Yanjun+Qi%2C+Srinivasan+H%2E+Sengamedu%2C+and+Christos+Faloutsos%2E+2024%2E+Large+language+models+%28LLMs%29+on+tabular+data%3A+Prediction%2C+generation%2C+an)
21. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024. [RULER: What’s the real context size of your long-context language models?](https://md.research.elamur.ai/free/en/papers/hsieh-2024-ruler-s-real-context-size/) In First Conference on Language Modeling.
22. Kung-Hsiang Huang, Hou Pong Chan, May Fung, Haoyi Qiu, Mingyang Zhou, Shafiq Joty, Shih-Fu Chang, and Heng Ji. 2025. From pixels to insights: A survey on automatic chart understanding in the era of large foundation models. IEEE Transactions on Knowledge and Data Engineering, 37(5):2550–2568. https:/… — [doi:10.1109/tkde.2024.3513320](https://doi.org/10.1109/tkde.2024.3513320) [find](https://scholar.google.com/scholar?q=Kung%2DHsiang+Huang%2C+Hou+Pong+Chan%2C+May+Fung%2C+Haoyi+Qiu%2C+Mingyang+Zhou%2C+Shafiq+Joty%2C+Shih%2DFu+Chang%2C+and+Heng+Ji%2E+2025%2E+From+pixels+to+insights%3A+A+survey+on+automatic+chart+understanding+in+the+era+of+la)
23. Maximilian Idahl and Zahra Ahmadi. 2025. OpenReviewer: A specialized large language model for generating critical scientific paper reviews. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Syst… [find](https://scholar.google.com/scholar?q=Maximilian+Idahl+and+Zahra+Ahmadi%2E+2025%2E+OpenReviewer%3A+A+specialized+large+language+model+for+generating+critical+scientific+paper+reviews%2E+In+Proceedings+of+the+2025+Conference+of+the+Nations+of+the+)
24. Alvan R. Feinstein and Domenic V. Cicchetti. 1990. High agreement but low kappa: I. the problems of two paradoxes. Journal of Clinical Epidemiology, 43(6):543–549. https://doi.org/10.1016/0895-4356(90)90158-L, PubMed: 2348207 — [doi:10.1016/0895-4356(90)90158-l](https://doi.org/10.1016/0895-4356%2890%2990158-l) [↗](https://doi.org/10.1016/0895-4356(90)90158-L)
25. Tom Jefferson, Elizabeth Wager, and Frank Davidoff. 2002. Measuring the quality of editorial peer review. JAMA, 287(21):2786–2790. https://doi.org/10.1001/jama.287 .21.2786, PubMed: 12038912 — [doi:10.1001/jama.287](https://doi.org/10.1001/jama.287) [↗](https://doi.org/10.1001/jama.287)
26. Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. 2024. [Bias and fairness in large language models: A survey](https://md.research.elamur.ai/free/en/papers/gallegos-2023-bias-fairness-large-language-models/). Computational Linguistics, 50(3):1097–1179. https://doi.org/10.1162/coli a 00524 [↗](https://doi.org/10.1162/coli)
27. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. [Survey of hallucination in natural language generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/). ACM Computing Surveys, 55(12). https://doi.org/10.1145/3571730 [↗](https://doi.org/10.1145/3571730)
28. Zhaolin Gao, Kianté Brantley, and Thorsten Joachims. 2024. [Reviewer2: Optimizing review generation through prompt generation](https://md.research.elamur.ai/free/en/papers/gao-2024-reviewer2-optimizing-review-generation-through/). arXiv preprint arXiv:2402.10886v2.
29. Chhavi Kirtani, Madhav Krishan Garg, Tejash Prasad, Tanmay Singhal, Murari Mandal, and Dhruv Kumar. 2025. ReviewEval: An evaluation framework for AI-generated reviews. arXiv preprint arXiv:2502.11736v3. [find](https://scholar.google.com/scholar?q=Chhavi+Kirtani%2C+Madhav+Krishan+Garg%2C+Tejash+Prasad%2C+Tanmay+Singhal%2C+Murari+Mandal%2C+and+Dhruv+Kumar%2E+2025%2E+ReviewEval%3A+An+evaluation+framework+for+AI%2Dgenerated+reviews%2E+arXiv+preprint+arXiv%3A2502%2E11736v)
30. Yair Ori Gat, Nitay Calderon, Amir Feder, Alexander Chapanin, Amit Sharma, and Roi Reichart. 2024. Faithful explanations of black-box NLP models using LLM-generated counterfactuals. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7–11, 2024. OpenR… [find](https://scholar.google.com/scholar?q=Yair+Ori+Gat%2C+Nitay+Calderon%2C+Amir+Feder%2C+Alexander+Chapanin%2C+Amit+Sharma%2C+and+Roi+Reichart%2E+2024%2E+Faithful+explanations+of+black%2Dbox+NLP+models+using+LLM%2Dgenerated+counterfactuals%2E+In+The+Twelfth+Int)
31. Ilia Kuznetsov, Osama Mohammed Afzal, Koen Dercksen, Nils Dycke, Alexander Goldberg, Tom Hope, Dirk Hovy, Jonathan K. Kummerfeld, Anne Lauscher, Kevin Leyton-Brown, Sheng Lu, Mausam, Margot Mieskes, Aurélie Névéol, Danish Pruthi, Lizhen Qu, Roy Schwartz, Noah A. Smith, Thamar Solorio, Jingyan Wan… [find](https://scholar.google.com/scholar?q=Ilia+Kuznetsov%2C+Osama+Mohammed+Afzal%2C+Koen+Dercksen%2C+Nils+Dycke%2C+Alexander+Goldberg%2C+Tom+Hope%2C+Dirk+Hovy%2C+Jonathan+K%2E+Kummerfeld%2C+Anne+Lauscher%2C+Kevin+Leyton%2DBrown%2C+Sheng+Lu%2C+Mausam%2C+Margot+Mieskes%2C+A)
32. Bangkok, Thailand. Association for Computational Linguistics. https://doi.org/10 .18653/v1/2024.acl-long.358 [find](https://scholar.google.com/scholar?q=Bangkok%2C+Thailand%2E+Association+for+Computational+Linguistics%2E+https%3A%2F%2Fdoi%2Eorg%2F10+%2E18653%2Fv1%2F2024%2Eacl%2Dlong%2E358)
33. Mary J. Lindstrom and Douglas M. Bates. 1988. Newton—Raphson and EM algorithms for linear mixed-effects models for repeated-measures data. Journal of the American Statistical Association, 83(404):1014–1022. https://doi.org/10.1080/01621459.1988.10478693 — [doi:10.1080/01621459.1988.10478693](https://doi.org/10.1080/01621459.1988.10478693) [↗](https://doi.org/10.1080/01621459.1988.10478693)
34. Jiatao Li, Yanheng Li, Xinyu Hu, Mingqi Gao, and Xiaojun Wan. 2025. Aspect-guided multi-level perturbation analysis of large language models in automated peer review. arXiv preprint arXiv:2502.12510v1. — [arXiv:2502.12510](https://arxiv.org/abs/2502.12510) [find](https://scholar.google.com/scholar?q=Jiatao+Li%2C+Yanheng+Li%2C+Xinyu+Hu%2C+Mingqi+Gao%2C+and+Xiaojun+Wan%2E+2025%2E+Aspect%2Dguided+multi%2Dlevel+perturbation+analysis+of+large+language+models+in+automated+peer+review%2E+arXiv+preprint+arXiv%3A2502%2E12510v1)
35. Ryan Liu and Nihar B. Shah. 2023. ReviewerGPT? An exploratory study on using large language models for paper reviewing. arXiv preprint arXiv:2306.00622v1. [find](https://scholar.google.com/scholar?q=Ryan+Liu+and+Nihar+B%2E+Shah%2E+2023%2E+ReviewerGPT%3F+An+exploratory+study+on+using+large+language+models+for+paper+reviewing%2E+arXiv+preprint+arXiv%3A2306%2E00622v1%2E)
36. Yongqi Li, Mayi Xu, Xin Miao, Shen Zhou, and Tieyun Qian. 2024. Prompting large language models for counterfactual generation: An empirical study. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20–25 May, 2… [find](https://scholar.google.com/scholar?q=Yongqi+Li%2C+Mayi+Xu%2C+Xin+Miao%2C+Shen+Zhou%2C+and+Tieyun+Qian%2E+2024%2E+Prompting+large+language+models+for+counterfactual+generation%3A+An+empirical+study%2E+In+Proceedings+of+the+2024+Joint+International+Confer)
37. Sheng Lu, Ilia Kuznetsov, and Iryna Gurevych. 2025. Identifying aspects in peer reviews. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 6145–6167, Suzhou, China. Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.findings-emnlp.326 [↗](https://doi.org/10.18653/v1/2025.findings-emnlp.326)
38. Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, Daniel Mcfarland, and James Y. Zou. 2024a. Monitoring AI-modified content at scale: A case study on the impact of ChatGPT on AI conference peer reviews. In Proceedings… [find](https://scholar.google.com/scholar?q=Weixin+Liang%2C+Zachary+Izzo%2C+Yaohui+Zhang%2C+Haley+Lepp%2C+Hancheng+Cao%2C+Xuandong+Zhao%2C+Lingjiao+Chen%2C+Haotian+Ye%2C+Sheng+Liu%2C+Zhi+Huang%2C+Daniel+Mcfarland%2C+and+James+Y%2E+Zou%2E+2024a%2E+Monitoring+AI%2Dmodified+co)
39. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with… [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+Shashank+Gupta%2C+Bodhisattwa+Prasad+Majumder%2C+Katherine+Her)
40. Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, Daniel A. McFarland, and James Zou. 2024b. Can large language models provide useful feedback on research papers? A large-scale empirical analysis. NEJM AI, 1(8):… [find](https://scholar.google.com/scholar?q=Weixin+Liang%2C+Yuhui+Zhang%2C+Hancheng+Cao%2C+Binglu+Wang%2C+Daisy+Yi+Ding%2C+Xinyu+Yang%2C+Kailas+Vodrahalli%2C+Siyu+He%2C+Daniel+Scott+Smith%2C+Yian+Yin%2C+Daniel+A%2E+McFarland%2C+and+James+Zou%2E+2024b%2E+Can+large+language)
41. Alison McCook. 2006. Is peer review broken? Submissions are up, reviewers are overtaxed, and authors are lodging complaint after complaint about the process at top-tier journals. What’s wrong with peer review? The Scientist, 20(2):26–35. [find](https://scholar.google.com/scholar?q=Alison+McCook%2E+2006%2E+Is+peer+review+broken%3F+Submissions+are+up%2C+reviewers+are+overtaxed%2C+and+authors+are+lodging+complaint+after+complaint+about+the+process+at+top%2Dtier+journals%2E+What%E2%80%99s+wrong+with+pee)
42. Guan-Ting Lin, Cheng-Han Chiang, and Hung-yi Lee. 2024. Advancing large language models to capture varied speaking styles and respond properly in spoken conversations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6626–6642,… [find](https://scholar.google.com/scholar?q=Guan%2DTing+Lin%2C+Cheng%2DHan+Chiang%2C+and+Hung%2Dyi+Lee%2E+2024%2E+Advancing+large+language+models+to+capture+varied+speaking+styles+and+respond+properly+in+spoken+conversations%2E+In+Proceedings+of+the+62nd+Annua)
43. Feiteng Mu and Wenjie Li. 2024. A causal approach for counterfactual reasoning in narratives. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6556–6569, Bangkok, Thailand. Association for Computational Linguistics. https://do… [find](https://scholar.google.com/scholar?q=Feiteng+Mu+and+Wenjie+Li%2E+2024%2E+A+causal+approach+for+counterfactual+reasoning+in+narratives%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long)
44. Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Mal… — [Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviews](https://md.research.elamur.ai/free/en/papers/shin-2025-mind-blind-spots-focus-level/)
45. Guijin Son, Jiwoo Hong, Honglu Fan, Heejeong Nam, Hyunwoo Ko, Seungwon Lim, Jinyeop Song, Jinha Choi, Gonçalo Paulo, Youngjae Yu, and Stella Biderman. 2025. When AI co-scientists fail: SPOT-a benchmark for automated verification of scientific research. arXiv preprint arXiv:2505.11855v1. [find](https://scholar.google.com/scholar?q=Guijin+Son%2C+Jiwoo+Hong%2C+Honglu+Fan%2C+Heejeong+Nam%2C+Hyunwoo+Ko%2C+Seungwon+Lim%2C+Jinyeop+Song%2C+Jinha+Choi%2C+Gonc%CC%A7alo+Paulo%2C+Youngjae+Yu%2C+and+Stella+Biderman%2E+2025%2E+When+AI+co%2Dscientists+fail%3A+SPOT%2Da+benchma)
46. Judea Pearl. 2000. Causality: Models, Reasoning and Inference, 2 edition, volume 7. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Judea+Pearl%2E+2000%2E+Causality%3A+Models%2C+Reasoning+and+Inference%2C+2+edition%2C+volume+7%2E+Cambridge+University+Press%2E)
47. James Phoenix and Mike Taylor. 2024. Prompt Engineering for Generative AI. O’Reilly Media, Inc. [find](https://scholar.google.com/scholar?q=James+Phoenix+and+Mike+Taylor%2E+2024%2E+Prompt+Engineering+for+Generative+AI%2E+O%E2%80%99Reilly+Media%2C+Inc%2E)
48. Simone Teufel, Advaith Siddharthan, and Colin Batchelor. 2009. Towards domain-independent argumentative zoning: Evidence from chemistry and computational linguistics. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 1493–1502, Singapore. Association fo… [find](https://scholar.google.com/scholar?q=Simone+Teufel%2C+Advaith+Siddharthan%2C+and+Colin+Batchelor%2E+2009%2E+Towards+domain%2Dindependent+argumentative+zoning%3A+Evidence+from+chemistry+and+computational+linguistics%2E+In+Proceedings+of+the+2009+Confer)
49. Haocong Rao, Cyril Leung, and Chunyan Miao. 2023. Can ChatGPT assess human personalities? A general evaluation framework. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1184–1194, Singapore. Association for Computational Linguistics. https://doi.org/10.18653/v1/2023… [↗](https://doi.org/10.18653/v1/2023…)
50. D. Sculley, Jasper Snoek, and Alex Wiltschko. 2018. Avoiding a tragedy of the commons in the peer review process. arXiv preprint arXiv:1901.06246v1. — [arXiv:1901.06246](https://arxiv.org/abs/1901.06246) [find](https://scholar.google.com/scholar?q=D%2E+Sculley%2C+Jasper+Snoek%2C+and+Alex+Wiltschko%2E+2018%2E+Avoiding+a+tragedy+of+the+commons+in+the+peer+review+process%2E+arXiv+preprint+arXiv%3A1901%2E06246v1%2E+%E2%80%94+%5BarXiv%3A1901%2E06246%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1901%2E062)
51. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
52. Hyungyu Shin, Jingyu Tang, Yoonjoo Lee, Nayoung Kim, Hyunseung Lim, Ji Yong Cho, Hwajung Hong, Moontae Lee, and Juho Kim. Keith Tyser, Ben Segev, Gaston Longhitano, Xin-Yu Zhang, Zachary Meeks, Jason Lee, Uday Garg, Nicholas Belsten, Avi Shporer, Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinij… — [A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications](https://md.research.elamur.ai/free/en/papers/sahoo-2024-systematic-survey-prompt-engineering-large/)
53. Oscar Sainz, Jon Campos, Iker Garcı́a-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. [NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark](https://md.research.elamur.ai/free/en/papers/sainz-2023-nlp-evaluation-trouble-need-measure/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787, Singa…
54. Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Melanie+Sclar%2C+Yejin+Choi%2C+Yulia+Tsvetkov%2C+and+Alane+Suhr%2E+2024%2E+Quantifying+language+models%E2%80%99+sensitivity+to+spurious+features+in+prompt+design+or%3A+How+I+learned+to+start+worrying+about+prompt+formatt)
55. Madeleine Udell, Dov Te’eni, and Iddo Drori. 2024. AI-driven review systems: Evaluating LLMs in scalable and bias-aware academic reviews. arXiv preprint arXiv:2408.10365v1. doi.org/10.18653/v1/2024.findings-acl .369 [find](https://scholar.google.com/scholar?q=Madeleine+Udell%2C+Dov+Te%E2%80%99eni%2C+and+Iddo+Drori%2E+2024%2E+AI%2Ddriven+review+systems%3A+Evaluating+LLMs+in+scalable+and+bias%2Daware+academic+reviews%2E+arXiv+preprint+arXiv%3A2408%2E10365v1%2E+doi%2Eorg%2F10%2E18653%2Fv1%2F2024%2Efi)
56. Shengwei Xu, Yuxuan Lu, Grant Schoenebeck, and Yuqing Kong. 2025. Benchmarking LLMs’ judgments with no gold standard. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24–28, 2025. OpenReview.net. [find](https://scholar.google.com/scholar?q=Shengwei+Xu%2C+Yuxuan+Lu%2C+Grant+Schoenebeck%2C+and+Yuqing+Kong%2E+2025%2E+Benchmarking+LLMs%E2%80%99+judgments+with+no+gold+standard%2E+In+The+Thirteenth+International+Conference+on+Learning+Representations%2C+ICLR+2025%2C)
57. Ludo Waltman, Wolfgang Kaltenbrunner, Stephen Pinfield, and Helen Buckley Woods. 2023. How to improve scientific peer review: Four schools of thought. Learned Publishing, 36(3):334–347. https://doi.org/10 .1002/leap.1544, PubMed: 38504796 [find](https://scholar.google.com/scholar?q=Ludo+Waltman%2C+Wolfgang+Kaltenbrunner%2C+Stephen+Pinfield%2C+and+Helen+Buckley+Woods%2E+2023%2E+How+to+improve+scientific+peer+review%3A+Four+schools+of+thought%2E+Learned+Publishing%2C+36%283%29%3A334%E2%80%93347%2E+https%3A%2F%2Fdoi%2Eor)
58. Yongjie Wang, Xiaoqi Qiu, Yu Yue, Xu Guo, Zhiwei Zeng, Yuhong Feng, and Zhiqi Shen. 2024a. A survey on natural language counterfactual generation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4798–4818, Miami, Florida, USA. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Yongjie+Wang%2C+Xiaoqi+Qiu%2C+Yu+Yue%2C+Xu+Guo%2C+Zhiwei+Zeng%2C+Yuhong+Feng%2C+and+Zhiqi+Shen%2E+2024a%2E+A+survey+on+natural+language+counterfactual+generation%2E+In+Findings+of+the+Association+for+Computational+Ling)
59. Jianxiang Yu, Zichen Ding, Jiaqi Tan, Kangyang Luo, Zhenmin Weng, Chenghua Gong, Long Zeng, RenJing Cui, Chengcheng Han, Qiushi Sun, Zhiyong Wu, Yunshi Lan, and Xiang Li. 2024. Automated peer reviewing in paper SEA: Standardization, evaluation, and analysis. In Findings of the Association for Comput… [find](https://scholar.google.com/scholar?q=Jianxiang+Yu%2C+Zichen+Ding%2C+Jiaqi+Tan%2C+Kangyang+Luo%2C+Zhenmin+Weng%2C+Chenghua+Gong%2C+Long+Zeng%2C+RenJing+Cui%2C+Chengcheng+Han%2C+Qiushi+Sun%2C+Zhiyong+Wu%2C+Yunshi+Lan%2C+and+Xiang+Li%2E+2024%2E+Automated+peer+reviewin)
60. Weizhe Yuan, Pengfei Liu, and Graham Neubig. 2022. Can we automate scientific reviewing? Journal of Artificial Intelligence Research, 75:171–212. https://doi.org/10.1613/jair.1.12862 [↗](https://doi.org/10.1613/jair.1.12862)
61. Ziao Wang, Xiaofeng Zhang, and Hongwei Du. 2024b. Beyond what if: Advancing counterfactual text generation with structural causal modeling. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, pages 6522–6530. [find](https://scholar.google.com/scholar?q=Ziao+Wang%2C+Xiaofeng+Zhang%2C+and+Hongwei+Du%2E+2024b%2E+Beyond+what+if%3A+Advancing+counterfactual+text+generation+with+structural+causal+modeling%2E+In+Proceedings+of+the+Thirty%2DThird+International+Joint+Confe)
62. Tianmai M. Zhang and Neil F. Abernethy. 2025. [Reviewing scientific papers for critical problems with reasoning LLMs: Baseline approaches and automatic evaluation](https://md.research.elamur.ai/free/en/papers/zhang-2025-reviewing-scientific-papers-critical-problems/). arXiv preprint arXiv:2505.23824v2.
63. Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang. 2025. CycleResearcher: Improving automated research via automated review. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Yixuan+Weng%2C+Minjun+Zhu%2C+Guangsheng+Bao%2C+Hongbo+Zhang%2C+Jindong+Wang%2C+Yue+Zhang%2C+and+Linyi+Yang%2E+2025%2E+CycleResearcher%3A+Improving+automated+research+via+automated+review%2E+In+The+Thirteenth+Internationa)
64. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. [Judging LLM-as-a-judge with MT-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Advances in Neural Information Processing Systems, volume 3…
65. Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. 2021. [Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models](https://md.research.elamur.ai/free/en/papers/wu-2021-polyjuice-generating-counterfactuals-explaining-evaluating/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on N…
66. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. [Large language models are human-level prompt engineers](https://md.research.elamur.ai/free/en/papers/zhou-2022-large-language-models-human-level/). In The Eleventh International Conference on Learning Representations.
67. Amelie Wuehrl, Dustin Wright, Roman Klinger, and Isabelle Augenstein. 2024. Understanding fine-grained distortions in reports of scientific findings. In Findings of the Association for Computational Linguistics: ACL 2024, pages 6175–6191, Bangkok, Thailand. Association for Computational Linguistics.… [find](https://scholar.google.com/scholar?q=Amelie+Wuehrl%2C+Dustin+Wright%2C+Roman+Klinger%2C+and+Isabelle+Augenstein%2E+2024%2E+Understanding+fine%2Dgrained+distortions+in+reports+of+scientific+findings%2E+In+Findings+of+the+Association+for+Computational+L)
68. Minjun Zhu, Yixuan Weng, Linyi Yang, and Yue Zhang. 2025. Deepreview: Improving LLM-based paper review with human-like deep thinking process. arXiv preprint arXiv: 2503.08569v1. [find](https://scholar.google.com/scholar?q=Minjun+Zhu%2C+Yixuan+Weng%2C+Linyi+Yang%2C+and+Yue+Zhang%2E+2025%2E+Deepreview%3A+Improving+LLM%2Dbased+paper+review+with+human%2Dlike+deep+thinking+process%2E+arXiv+preprint+arXiv%3A+2503%2E08569v1%2E)
69. Algorithm 1 Pseudo-code for research logic extraction. Each function starting with ‘llm’ uses LLMs in zero-shot mode with a single self-refinement step. Require: Input = title, abstract, text Ensure: Output = F, C, R, m, supports, anchors, coreferences RG ← llm identify research goal(title, abstract… [find](https://scholar.google.com/scholar?q=Algorithm+1+Pseudo%2Dcode+for+research+logic+extraction%2E+Each+function+starting+with+%E2%80%98llm%E2%80%99+uses+LLMs+in+zero%2Dshot+mode+with+a+single+self%2Drefinement+step%2E+Require%3A+Input+%3D+title%2C+abstract%2C+text+Ensure%3A+)

*Updated 30 October 2026*