---
title: "Can large language models provide useful feedback on research"
description: "Rising submission volumes make it harder to get timely, high-quality critique of manuscripts. 61 references, cited by 89 papers here."
authors: ["Weixin Liang", "Yuhui Zhang", "Hancheng Cao", "Binglu Wang", "Daisy Ding", "Xinyu Yang", "Kailas Vodrahalli", "Siyu He", "Daniel Smith", "Yian Yin", "Daniel McFarland", "James Zou"]
year: 2023
venue: "arXiv:2310.01783"
source: "https://arxiv.org/abs/2310.01783"
doi: "10.48550/arxiv.2310.01783"
arxiv: "2310.01783"
licence: "arXiv non-exclusive distribution license"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liang-2023-large-language-models-provide/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liang-2023-large-language-models-provide/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/) / Can large language models provide useful feedback on research papers? A large-scale empirical analysis

# Can large language models provide useful feedback on research papers? A large-scale empirical analysis

**Authors:** [Weixin Liang](https://md.research.elamur.ai/free/en/authors/weixin-liang/), [Yuhui Zhang](https://md.research.elamur.ai/free/en/authors/yuhui-zhang/), [Hancheng Cao](https://md.research.elamur.ai/free/en/authors/hancheng-cao/), Binglu Wang, Daisy Ding, [Xinyu Yang](https://md.research.elamur.ai/free/en/authors/xinyu-yang/), Kailas Vodrahalli, [Siyu He](https://md.research.elamur.ai/free/en/authors/siyu-he/), Daniel Smith, Yian Yin, [Daniel McFarland](https://md.research.elamur.ai/free/en/authors/daniel-a-mcfarland/) and [James Zou](https://md.research.elamur.ai/free/en/authors/james-zou/) · **Year:** 2023 · **Venue:** arXiv:2310.01783 · **Licence:** [arXiv non-exclusive distribution license](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2310.01783) · [PDF](https://arxiv.org/pdf/2310.01783) · [DOI 10.48550/arxiv.2310.01783](https://doi.org/10.48550/arxiv.2310.01783) · [arXiv 2310.01783](https://arxiv.org/abs/2310.01783) · [OpenAlex](https://openalex.org/W4400734098)

**Citations:** 243 citations (OpenAlex, 2026-10-14) · [89](https://md.research.elamur.ai/free/en/papers/liang-2023-large-language-models-provide/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

Rising submission volumes make it harder to get timely, high-quality critique of manuscripts. ICLR submissions, for example, grew from 960 in 2018 to 4,966 in 2023. Junior and under-resourced researchers are hit hardest. The authors test whether GPT-4 can supply useful reviewer-style comments. Their pipeline parses a paper's full PDF and packs the title, abstract, captions and body text into a structured prompt. In a single pass, GPT-4 then comments on significance and novelty, reasons to accept, reasons to reject, and possible improvements.

Quality was assessed in two ways. Retrospectively, comments from GPT-4 and from real reviewers (15 Nature family journals and ICLR) were automatically extracted and semantically matched. The share of points GPT-4 had in common with a reviewer was about the same as the share two human reviewers had in common. Prospectively, researchers rated GPT-4 feedback on their own papers, and most found it useful. However, the model favours generic requests and seldom critiques method design in depth. LLM feedback therefore looks like a complement to expert review, most valuable for early drafts.

## Key points

- **Data:** 3,096 accepted Nature family papers (8,745 reviewer comments) and 1,709 ICLR 2022–2023 papers, both accepted and rejected (6,505 comments). Only post-2022 papers were used, to avoid overlap with GPT-4's training data (cutoff Sep 2021).
- **Matching pipeline:** Human checks gave F1 scores of 96.8% for comment extraction and 82.4% for matching.
- **Overlap with individual reviewers:** GPT-4 vs. one reviewer was 30.85% on Nature data (human–human: 28.58%) and 39.23% on ICLR (human–human: 35.25%). For rejected ICLR papers it rose to 43.80%. On Nature data, 57.55% of GPT-4's points were raised by at least one reviewer, and a paper-shuffling control showed the feedback is not merely generic.
- **User study:** 308 researchers from 110 US institutions in AI and computational biology took part. 57.4% rated the feedback helpful or very helpful, and 82.4% preferred it to the input of at least some human reviewers.
- **Code:** github.com/Weixin-Liang/LLM-scientific-feedback

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 506 works · 330,053 citations
- [Cornell University](https://md.research.elamur.ai/free/en/universities/cornell-university/) — University, Ithaca, United States · 134 works · 65,159 citations
- [Northwestern University](https://md.research.elamur.ai/free/en/universities/northwestern-university/) — University, Evanston, United States · 46 works · 2,850 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 249 works · [🎯 Statistics, Probability and Uncertainty](https://md.research.elamur.ai/free/en/shelves/decision-sciences/statistics-probability-and-uncertainty/) · 65 works · [🎭 History and Philosophy of Science](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/history-and-philosophy-of-science/) · 62 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Smith, Yian Yin, Daniel McFarland and James Zou. 2023. Can large language models provide useful feedback on research papers? A large-scale empirical analysis. arXiv:2310.01783.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{liang2023can,
  title = {{Can large language models provide useful feedback on research papers? A large-scale empirical analysis}},
  author = {Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Smith, Yian Yin, Daniel McFarland and James Zou},
  year = {2023},
  doi = {10.48550/arxiv.2310.01783},
  eprint = {2310.01783},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.01783}
}
```

- [← previous LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://md.research.elamur.ai/free/en/papers/zhang-2023-llama-adapter-efficient-fine-tuning/)
- [next → Language Is Not All You Need: Aligning Perception with Language Models](https://md.research.elamur.ai/free/en/papers/huang-2023-language-all-need-aligning-perception/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/)

## References

61 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Shannon, C. E. A mathematical theory of communication. The Bell system technical journal 27, 379–423 (1948). [find](https://scholar.google.com/scholar?q=Shannon%2C+C%2E+E%2E+A+mathematical+theory+of+communication%2E+The+Bell+system+technical+journal+27%2C+379%E2%80%93423+%281948%29%2E)
2. Tribus, M. & McIrvine, E. C. Energy and information. Sci. Am. 225, 179–190 (1971). [find](https://scholar.google.com/scholar?q=Tribus%2C+M%2E+%26+McIrvine%2C+E%2E+C%2E+Energy+and+information%2E+Sci%2E+Am%2E+225%2C+179%E2%80%93190+%281971%29%2E)
3. Kuhn, T. S. The structure of scientifi revolutions. The Un Chic. Press. 2, 90 (1962). [find](https://scholar.google.com/scholar?q=Kuhn%2C+T%2E+S%2E+The+structure+of+scientifi+revolutions%2E+The+Un+Chic%2E+Press%2E+2%2C+90+%281962%29%2E)
4. Horbach, S. P. & Halffman, W. The changing forms and expectations of peer review. Res. integrity peer review 3, 1–15 (2018). [find](https://scholar.google.com/scholar?q=Horbach%2C+S%2E+P%2E+%26+Halffman%2C+W%2E+The+changing+forms+and+expectations+of+peer+review%2E+Res%2E+integrity+peer+review+3%2C+1%E2%80%9315+%282018%29%2E)
5. Price, D. J. D. S. Little science, big science (Columbia University Press, 1963). [find](https://scholar.google.com/scholar?q=Price%2C+D%2E+J%2E+D%2E+S%2E+Little+science%2C+big+science+%28Columbia+University+Press%2C+1963%29%2E)
6. Jones, B. F. The burden of knowledge and the “death of the renaissance man”: Is innovation getting harder? The Rev. Econ. Stud. 76, 283–317 (2009). [find](https://scholar.google.com/scholar?q=Jones%2C+B%2E+F%2E+The+burden+of+knowledge+and+the+%E2%80%9Cdeath+of+the+renaissance+man%E2%80%9D%3A+Is+innovation+getting+harder%3F+The+Rev%2E+Econ%2E+Stud%2E+76%2C+283%E2%80%93317+%282009%29%2E)
7. Aczel, B., Szaszi, B. & Holcombe, A. O. A billion-dollar donation: estimating the cost of researchers’ time spent on peer review. Res. Integr. Peer Rev. 6, 1–8 (2021). [find](https://scholar.google.com/scholar?q=Aczel%2C+B%2E%2C+Szaszi%2C+B%2E+%26+Holcombe%2C+A%2E+O%2E+A+billion%2Ddollar+donation%3A+estimating+the+cost+of+researchers%E2%80%99+time+spent+on+peer+review%2E+Res%2E+Integr%2E+Peer+Rev%2E+6%2C+1%E2%80%938+%282021%29%2E)
8. Alberts, B., Hanson, B. & Kelner, K. L. Reviewing peer review (2008). [find](https://scholar.google.com/scholar?q=Alberts%2C+B%2E%2C+Hanson%2C+B%2E+%26+Kelner%2C+K%2E+L%2E+Reviewing+peer+review+%282008%29%2E)
9. Björk, B.-C. & Solomon, D. The publishing delay in scholarly peer-reviewed journals. J. informetrics 7, 914–923 (2013). [find](https://scholar.google.com/scholar?q=Bj%C3%B6rk%2C+B%2E%2DC%2E+%26+Solomon%2C+D%2E+The+publishing+delay+in+scholarly+peer%2Dreviewed+journals%2E+J%2E+informetrics+7%2C+914%E2%80%93923+%282013%29%2E)
10. Lee, C. J., Sugimoto, C. R., Zhang, G. & Cronin, B. Bias in peer review. J. Am. Soc. for information Sci. Technol. 64, 2–17 (2013). [find](https://scholar.google.com/scholar?q=Lee%2C+C%2E+J%2E%2C+Sugimoto%2C+C%2E+R%2E%2C+Zhang%2C+G%2E+%26+Cronin%2C+B%2E+Bias+in+peer+review%2E+J%2E+Am%2E+Soc%2E+for+information+Sci%2E+Technol%2E+64%2C+2%E2%80%9317+%282013%29%2E)
11. Kovanis, M., Porcher, R., Ravaud, P. & Trinquart, L. The global burden of journal peer review in the biomedical literature: Strong imbalance in the collective enterprise. PloS one 11, e0166387 (2016). [find](https://scholar.google.com/scholar?q=Kovanis%2C+M%2E%2C+Porcher%2C+R%2E%2C+Ravaud%2C+P%2E+%26+Trinquart%2C+L%2E+The+global+burden+of+journal+peer+review+in+the+biomedical+literature%3A+Strong+imbalance+in+the+collective+enterprise%2E+PloS+one+11%2C+e0166387+%282016%29%2E)
12. Shah, N. B. Challenges, experiments, and computational solutions in peer review. Commun. ACM 65, 76–87 (2022). [find](https://scholar.google.com/scholar?q=Shah%2C+N%2E+B%2E+Challenges%2C+experiments%2C+and+computational+solutions+in+peer+review%2E+Commun%2E+ACM+65%2C+76%E2%80%9387+%282022%29%2E)
13. Bourdieu, P. Cultural reproduction and social reproduction. In Knowledge, education, and cultural change, 71–112 (Routledge, 2018). [find](https://scholar.google.com/scholar?q=Bourdieu%2C+P%2E+Cultural+reproduction+and+social+reproduction%2E+In+Knowledge%2C+education%2C+and+cultural+change%2C+71%E2%80%93112+%28Routledge%2C+2018%29%2E)
14. Merton, R. K. The matthew effect in science: The reward and communication systems of science are considered. Science 159, 56–63 (1968). [find](https://scholar.google.com/scholar?q=Merton%2C+R%2E+K%2E+The+matthew+effect+in+science%3A+The+reward+and+communication+systems+of+science+are+considered%2E+Science+159%2C+56%E2%80%9363+%281968%29%2E)
15. Chu, J. S. & Evans, J. A. Slowed canonical progress in large fields of science. Proc. Natl. Acad. Sci. 118, e2021636118 (2021). [find](https://scholar.google.com/scholar?q=Chu%2C+J%2E+S%2E+%26+Evans%2C+J%2E+A%2E+Slowed+canonical+progress+in+large+fields+of+science%2E+Proc%2E+Natl%2E+Acad%2E+Sci%2E+118%2C+e2021636118+%282021%29%2E)
16. Bloom, N., Jones, C. I., Van Reenen, J. & Webb, M. Are ideas getting harder to find? Am. Econ. Rev. 110, 1104–1144 (2020). [find](https://scholar.google.com/scholar?q=Bloom%2C+N%2E%2C+Jones%2C+C%2E+I%2E%2C+Van+Reenen%2C+J%2E+%26+Webb%2C+M%2E+Are+ideas+getting+harder+to+find%3F+Am%2E+Econ%2E+Rev%2E+110%2C+1104%E2%80%931144+%282020%29%2E)
17. Brown, T. et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Adv. Neural Inf. Process. Syst. 33, 1877–1901 (2020).
18. Ouyang, L. et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Adv. Neural Inf. Process. Syst. 35, 27730–27744 (2022).
19. OpenAI. [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). ArXiv abs/2303.08774 (2023).
20. Ayers, J. W. et al. Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA internal medicine (2023). [find](https://scholar.google.com/scholar?q=Ayers%2C+J%2E+W%2E+et+al%2E+Comparing+physician+and+artificial+intelligence+chatbot+responses+to+patient+questions+posted+to+a+public+social+media+forum%2E+JAMA+internal+medicine+%282023%29%2E)
21. Lee, M. et al. Evaluating human-language model interaction. arXiv preprint arXiv:2212.09746 (2022). [find](https://scholar.google.com/scholar?q=Lee%2C+M%2E+et+al%2E+Evaluating+human%2Dlanguage+model+interaction%2E+arXiv+preprint+arXiv%3A2212%2E09746+%282022%29%2E)
22. Kung, T. H. et al. [Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models](https://md.research.elamur.ai/free/en/papers/kung-2023-performance-chatgpt-usmle-potential-ai/). PLoS digital health 2, e0000198 (2023).
23. Terwiesch, C. Would chat gpt3 get a wharton mba? a prediction based on its performance in the operations management course. Mack Inst. for Innov. Manag. at Whart. Sch. Univ. Pennsylvania (2023). [find](https://scholar.google.com/scholar?q=Terwiesch%2C+C%2E+Would+chat+gpt3+get+a+wharton+mba%3F+a+prediction+based+on+its+performance+in+the+operations+management+course%2E+Mack+Inst%2E+for+Innov%2E+Manag%2E+at+Whart%2E+Sch%2E+Univ%2E+Pennsylvania+%282023%29%2E)
24. Schulz, R. et al. Is the future of peer review automated? BMC Res. Notes 15, 1–5 (2022). [find](https://scholar.google.com/scholar?q=Schulz%2C+R%2E+et+al%2E+Is+the+future+of+peer+review+automated%3F+BMC+Res%2E+Notes+15%2C+1%E2%80%935+%282022%29%2E)
25. Liu, R. & Shah, N. B. [Reviewergpt? an exploratory study on using large language models for paper reviewing](https://md.research.elamur.ai/free/en/papers/liu-2023-reviewergpt-exploratory-study-using/). arXiv preprint arXiv:2306.00622 (2023).
26. Robertson, Z. Gpt4 is slightly helpful for peer-review assistance: A pilot study. arXiv preprint arXiv:2307.05492 (2023). — [arXiv:2307.05492](https://arxiv.org/abs/2307.05492) [find](https://scholar.google.com/scholar?q=Robertson%2C+Z%2E+Gpt4+is+slightly+helpful+for+peer%2Dreview+assistance%3A+A+pilot+study%2E+arXiv+preprint+arXiv%3A2307%2E05492+%282023%29%2E+%E2%80%94+%5BarXiv%3A2307%2E05492%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2307%2E05492%29)
27. Nature. How to Write a Report. [https://www.nature.com/nature/for-referees/how-to-write-a-report](https://www.nature.com/nature/for-referees/how-to-write-a-report). Accessed: 21 September 2023. [↗](https://www.nature.com/nature/for-referees/how-to-write-a-report)
28. Nature Communications. Writing Your Report. writing-your-report. Accessed: 21 September 2023. [find](https://scholar.google.com/scholar?q=Nature+Communications%2E+Writing+Your+Report%2E+writing%2Dyour%2Dreport%2E+Accessed%3A+21+September+2023%2E)
29. Rogers, A. & Augenstein, I. How to review for ACL Rolling Review. [https://aclrollingreview.org/reviewertutorial](https://aclrollingreview.org/reviewertutorial) (2021). [↗](https://aclrollingreview.org/reviewertutorial)
30. Association for Computational Linguistics. ACL’23 Peer Review Policies. [https://2023.aclweb.org/blog/review-acl23/](https://2023.aclweb.org/blog/review-acl23/) (2023). [↗](https://2023.aclweb.org/blog/review-acl23/)
31. Association for Computational Linguistics. ACL-IJCNLP 2021 Instructions for Reviewers. [https://2021.aclweb.org/blog/instructions-for-reviewers/](https://2021.aclweb.org/blog/instructions-for-reviewers/) (2021). [↗](https://2021.aclweb.org/blog/instructions-for-reviewers/)
32. International Conference on Machine Learning. ICML 2023 Reviewer Tutorial. [https://icml.cc/Conferences/2023/ReviewerTutorial](https://icml.cc/Conferences/2023/ReviewerTutorial) (2023). [↗](https://icml.cc/Conferences/2023/ReviewerTutorial)
33. Nicholas, K. A. & Gordon, W. S. A quick guide to writing a solid peer review. Eos, Transactions Am. Geophys. Union 92, 233–234 (2011). [find](https://scholar.google.com/scholar?q=Nicholas%2C+K%2E+A%2E+%26+Gordon%2C+W%2E+S%2E+A+quick+guide+to+writing+a+solid+peer+review%2E+Eos%2C+Transactions+Am%2E+Geophys%2E+Union+92%2C+233%E2%80%93234+%282011%29%2E)
34. Luhn, H. P. The automatic creation of literature abstracts. IBM J. research development 2, 159–165 (1958). [find](https://scholar.google.com/scholar?q=Luhn%2C+H%2E+P%2E+The+automatic+creation+of+literature+abstracts%2E+IBM+J%2E+research+development+2%2C+159%E2%80%93165+%281958%29%2E)
35. Edmundson, H. P. New methods in automatic extracting. J. ACM (JACM) 16, 264–285 (1969). [find](https://scholar.google.com/scholar?q=Edmundson%2C+H%2E+P%2E+New+methods+in+automatic+extracting%2E+J%2E+ACM+%28JACM%29+16%2C+264%E2%80%93285+%281969%29%2E)
36. Mihalcea, R. & Tarau, P. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing, 404–411 (2004). [find](https://scholar.google.com/scholar?q=Mihalcea%2C+R%2E+%26+Tarau%2C+P%2E+Textrank%3A+Bringing+order+into+text%2E+In+Proceedings+of+the+2004+conference+on+empirical+methods+in+natural+language+processing%2C+404%E2%80%93411+%282004%29%2E)
37. Erkan, G. & Radev, D. R. Lexrank: Graph-based lexical centrality as salience in text summarization. J. artificial intelligence research 22, 457–479 (2004). [find](https://scholar.google.com/scholar?q=Erkan%2C+G%2E+%26+Radev%2C+D%2E+R%2E+Lexrank%3A+Graph%2Dbased+lexical+centrality+as+salience+in+text+summarization%2E+J%2E+artificial+intelligence+research+22%2C+457%E2%80%93479+%282004%29%2E)
38. Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K. & Harshman, R. Indexing by latent semantic analysis. J. Am. society for information science 41, 391–407 (1990). [https://www.nature.com/ncomms/for-reviewers/](https://www.nature.com/ncomms/for-reviewers/) [↗](https://www.nature.com/ncomms/for-reviewers/)
39. Socher, R., Huang, E., Pennin, J., Manning, C. D. & Ng, A. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. Adv. neural information processing systems 24 (2011). [find](https://scholar.google.com/scholar?q=Socher%2C+R%2E%2C+Huang%2C+E%2E%2C+Pennin%2C+J%2E%2C+Manning%2C+C%2E+D%2E+%26+Ng%2C+A%2E+Dynamic+pooling+and+unfolding+recursive+autoencoders+for+paraphrase+detection%2E+Adv%2E+neural+information+processing+systems+24+%282011%29%2E)
40. Bowman, S. R., Angeli, G., Potts, C. & Manning, C. D. [A large annotated corpus for learning natural language inference](https://md.research.elamur.ai/free/en/papers/bowman-2015-large-annotated-corpus-learning-natural/). arXiv preprint arXiv:1508.05326 (2015).
41. Birhane, A. et al. The values encoded in machine learning research. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’22, 173–184, DOI: 10.1145/3531146.3533083 (Association for Computing Machinery, New York, NY, USA, 2022). — [doi:10.1145/3531146.3533083](https://doi.org/10.1145/3531146.3533083) [find](https://scholar.google.com/scholar?q=Birhane%2C+A%2E+et+al%2E+The+values+encoded+in+machine+learning+research%2E+In+Proceedings+of+the+2022+ACM+Conference+on+Fairness%2C+Accountability%2C+and+Transparency%2C+FAccT+%E2%80%9922%2C+173%E2%80%93184%2C+DOI%3A+10%2E1145%2F3531146%2E35)
42. Smith, J. J., Amershi, S., Barocas, S., Wallach, H. & Wortman Vaughan, J. Real ml: Recognizing, exploring, and articulating limitations of machine learning research. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, 587–597 (2022). [find](https://scholar.google.com/scholar?q=Smith%2C+J%2E+J%2E%2C+Amershi%2C+S%2E%2C+Barocas%2C+S%2E%2C+Wallach%2C+H%2E+%26+Wortman+Vaughan%2C+J%2E+Real+ml%3A+Recognizing%2C+exploring%2C+and+articulating+limitations+of+machine+learning+research%2E+In+Proceedings+of+the+2022+ACM+Con)
43. Koch, B., Denton, E., Hanna, A. & Foster, J. G. Reduced, reused and recycled: The life of a dataset in machine learning research. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) (2021). [find](https://scholar.google.com/scholar?q=Koch%2C+B%2E%2C+Denton%2C+E%2E%2C+Hanna%2C+A%2E+%26+Foster%2C+J%2E+G%2E+Reduced%2C+reused+and+recycled%3A+The+life+of+a+dataset+in+machine+learning+research%2E+In+Thirty%2Dfifth+Conference+on+Neural+Information+Processing+Systems+Da)
44. Scheuerman, M. K., Hanna, A. & Denton, E. Do datasets have politics? disciplinary values in computer vision dataset development. Proc. ACM on Human-Computer Interact. 5, 1–37 (2021). [find](https://scholar.google.com/scholar?q=Scheuerman%2C+M%2E+K%2E%2C+Hanna%2C+A%2E+%26+Denton%2C+E%2E+Do+datasets+have+politics%3F+disciplinary+values+in+computer+vision+dataset+development%2E+Proc%2E+ACM+on+Human%2DComputer+Interact%2E+5%2C+1%E2%80%9337+%282021%29%2E)
45. Meyer, B. D., Mok, W. K. & Sullivan, J. X. Household surveys in crisis. J. Econ. Perspectives 29, 199–226 (2015). [find](https://scholar.google.com/scholar?q=Meyer%2C+B%2E+D%2E%2C+Mok%2C+W%2E+K%2E+%26+Sullivan%2C+J%2E+X%2E+Household+surveys+in+crisis%2E+J%2E+Econ%2E+Perspectives+29%2C+199%E2%80%93226+%282015%29%2E)
46. Ross, M. B. et al. Women are credited less in science than men. Nature 608, 135–145 (2022). [find](https://scholar.google.com/scholar?q=Ross%2C+M%2E+B%2E+et+al%2E+Women+are+credited+less+in+science+than+men%2E+Nature+608%2C+135%E2%80%93145+%282022%29%2E)
47. Noy, S. & Zhang, W. Experimental evidence on the productivity effects of generative artificial intelligence. Available at SSRN 4375283 (2023). [find](https://scholar.google.com/scholar?q=Noy%2C+S%2E+%26+Zhang%2C+W%2E+Experimental+evidence+on+the+productivity+effects+of+generative+artificial+intelligence%2E+Available+at+SSRN+4375283+%282023%29%2E)
48. Peng, S., Kalliamvakou, E., Cihon, P. & Demirer, M. The impact of ai on developer productivity: Evidence from github copilot. arXiv preprint arXiv:2302.06590 (2023). — [arXiv:2302.06590](https://arxiv.org/abs/2302.06590) [find](https://scholar.google.com/scholar?q=Peng%2C+S%2E%2C+Kalliamvakou%2C+E%2E%2C+Cihon%2C+P%2E+%26+Demirer%2C+M%2E+The+impact+of+ai+on+developer+productivity%3A+Evidence+from+github+copilot%2E+arXiv+preprint+arXiv%3A2302%2E06590+%282023%29%2E+%E2%80%94+%5BarXiv%3A2302%2E06590%5D%28https%3A%2F%2Farxiv)
49. Epstein, Z. et al. Art and the science of generative ai. Science 380, 1110–1111 (2023). [find](https://scholar.google.com/scholar?q=Epstein%2C+Z%2E+et+al%2E+Art+and+the+science+of+generative+ai%2E+Science+380%2C+1110%E2%80%931111+%282023%29%2E)
50. Wang, H. et al. Scientific discovery in the age of artificial intelligence. Nature 620, 47–60 (2023). [find](https://scholar.google.com/scholar?q=Wang%2C+H%2E+et+al%2E+Scientific+discovery+in+the+age+of+artificial+intelligence%2E+Nature+620%2C+47%E2%80%9360+%282023%29%2E)
51. Nature will publish peer review reports as a trial. Nature 578, 8, DOI: 10.1038/d41586-020-00309-9 (2020). — [doi:10.1038/d41586-020-00309-9](https://doi.org/10.1038/d41586-020-00309-9) [find](https://scholar.google.com/scholar?q=Nature+will+publish+peer+review+reports+as+a+trial%2E+Nature+578%2C+8%2C+DOI%3A+10%2E1038%2Fd41586%2D020%2D00309%2D9+%282020%29%2E+%E2%80%94+%5Bdoi%3A10%2E1038%2Fd41586%2D020%2D00309%2D9%5D%28https%3A%2F%2Fdoi%2Eorg%2F10%2E1038%2Fd41586%2D020%2D00309%2D9%29)
52. Nature is trialling transparent peer review - the early results are encouraging. Nature 603, 8, DOI: 10.1038/d41586-022-00493-w (2022). — [doi:10.1038/d41586-022-00493-w](https://doi.org/10.1038/d41586-022-00493-w) [find](https://scholar.google.com/scholar?q=Nature+is+trialling+transparent+peer+review+%2D+the+early+results+are+encouraging%2E+Nature+603%2C+8%2C+DOI%3A+10%2E1038%2Fd41586%2D022%2D00493%2Dw+%282022%29%2E+%E2%80%94+%5Bdoi%3A10%2E1038%2Fd41586%2D022%2D00493%2Dw%5D%28https%3A%2F%2Fdoi%2Eorg%2F10%2E1038%2Fd4158)
53. Ecer, D. & Maciocci, G. Sciencebeam—using computer vision to extract pdf data. Elife Blog Post (2017). [Online; accessed 2023-Sep-8]. [find](https://scholar.google.com/scholar?q=Ecer%2C+D%2E+%26+Maciocci%2C+G%2E+Sciencebeam%E2%80%94using+computer+vision+to+extract+pdf+data%2E+Elife+Blog+Post+%282017%29%2E+%5BOnline%3B+accessed+2023%2DSep%2D8%5D%2E)
54. Goodman, S. N., Berlin, J., Fletcher, S. W. & Fletcher, R. H. Manuscript quality before and after peer review and editing at annals of internal medicine. Annals internal medicine 121, 11–21 (1994). [find](https://scholar.google.com/scholar?q=Goodman%2C+S%2E+N%2E%2C+Berlin%2C+J%2E%2C+Fletcher%2C+S%2E+W%2E+%26+Fletcher%2C+R%2E+H%2E+Manuscript+quality+before+and+after+peer+review+and+editing+at+annals+of+internal+medicine%2E+Annals+internal+medicine+121%2C+11%E2%80%9321+%281994%29%2E)
55. Abid, A. et al. An online platform for interactive feedback in biomedical machine learning. Nat. Mach. Intell. 2, 86–88 (2020). [find](https://scholar.google.com/scholar?q=Abid%2C+A%2E+et+al%2E+An+online+platform+for+interactive+feedback+in+biomedical+machine+learning%2E+Nat%2E+Mach%2E+Intell%2E+2%2C+86%E2%80%9388+%282020%29%2E)
56. Collins, E., Augenstein, I. & Riedel, S. A supervised approach to extractive summarisation of scientific papers. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), 195–205 (2017). [find](https://scholar.google.com/scholar?q=Collins%2C+E%2E%2C+Augenstein%2C+I%2E+%26+Riedel%2C+S%2E+A+supervised+approach+to+extractive+summarisation+of+scientific+papers%2E+In+Proceedings+of+the+21st+Conference+on+Computational+Natural+Language+Learning+%28CoNLL)
57. Nuijten, M. B., Hartgerink, C. H., Van Assen, M. A., Epskamp, S. & Wicherts, J. M. The prevalence of statistical reporting errors in psychology (1985–2013). Behav. research methods 48, 1205–1226 (2016). [find](https://scholar.google.com/scholar?q=Nuijten%2C+M%2E+B%2E%2C+Hartgerink%2C+C%2E+H%2E%2C+Van+Assen%2C+M%2E+A%2E%2C+Epskamp%2C+S%2E+%26+Wicherts%2C+J%2E+M%2E+The+prevalence+of+statistical+reporting+errors+in+psychology+%281985%E2%80%932013%29%2E+Behav%2E+research+methods+48%2C+1205%E2%80%931226+%282016)
58. Zhou, Y., Beltagy, I., Bethard, S., Cotterell, R. & Chakraborty, T. ACL pubcheck. [https://github.com/acl-org/aclpubcheck](https://github.com/acl-org/aclpubcheck). [↗](https://github.com/acl-org/aclpubcheck)
59. Zhang, J., Zhang, H., Deng, Z. & Roth, D. Investigating fairness disparities in peer review: A language model enhanced approach. arXiv preprint arXiv:2211.06398 (2022). [find](https://scholar.google.com/scholar?q=Zhang%2C+J%2E%2C+Zhang%2C+H%2E%2C+Deng%2C+Z%2E+%26+Roth%2C+D%2E+Investigating+fairness+disparities+in+peer+review%3A+A+language+model+enhanced+approach%2E+arXiv+preprint+arXiv%3A2211%2E06398+%282022%29%2E)
60. Hosseini, M. & Horbach, S. P. Fighting reviewer fatigue or amplifying bias? considerations and recommendations for use of chatgpt and other large language models in scholarly peer review. Res. Integr. Peer Rev. 8, 4 (2023). [find](https://scholar.google.com/scholar?q=Hosseini%2C+M%2E+%26+Horbach%2C+S%2E+P%2E+Fighting+reviewer+fatigue+or+amplifying+bias%3F+considerations+and+recommendations+for+use+of+chatgpt+and+other+large+language+models+in+scholarly+peer+review%2E+Res%2E+Integr%2E)
61. Verharen, J. P. Chatgpt identifies gender disparities in scientific peer review. bioRxiv 2023–07 (2023). Parsed PDF Raw PDF Title Mind the Gap: Understanding Modality Gap... Abstract We present modality gap, an intriguing geometric… Introduction Multi-modal models map inputs from different data moda… [find](https://scholar.google.com/scholar?q=Verharen%2C+J%2E+P%2E+Chatgpt+identifies+gender+disparities+in+scientific+peer+review%2E+bioRxiv+2023%E2%80%9307+%282023%29%2E+Parsed+PDF+Raw+PDF+Title+Mind+the+Gap%3A+Understanding+Modality+Gap%2E%2E%2E+Abstract+We+present+modali)

*Updated 2 November 2026*