---
title: "Peer Reviews of Peer Reviews — paper by Goldberg et al. (2023)"
description: "This paper tests whether such scores can be trusted. 0.5 points. 53 references, cited by 14 papers here."
authors: ["Alexander Goldberg", "Ivan Stelmakh", "Kyunghyun Cho", "Alice Oh", "Alekh Agarwal", "Danielle Belgrave", "Nihar B. Shah"]
year: 2023
venue: "arXiv:2311.09497"
source: "https://arxiv.org/abs/2311.09497"
doi: "10.48550/arxiv.2311.09497"
arxiv: "2311.09497"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/goldberg-2023-peer-reviews-peer-reviews/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/goldberg-2023-peer-reviews-peer-reviews/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/12/) / Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Experiments

# Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Experiments

**Authors:** [Alexander Goldberg](https://md.research.elamur.ai/free/en/authors/alexander-goldberg/), [Ivan Stelmakh](https://md.research.elamur.ai/free/en/authors/ivan-stelmakh/), [Kyunghyun Cho](https://md.research.elamur.ai/free/en/authors/kyunghyun-cho/), [Alice Oh](https://md.research.elamur.ai/free/en/authors/alice-oh/), [Alekh Agarwal](https://md.research.elamur.ai/free/en/authors/alekh-agarwal/), Danielle Belgrave and [Nihar B. Shah](https://md.research.elamur.ai/free/en/authors/nihar-b-shah/) · **Year:** 2023 · **Venue:** [arXiv:2311.09497](https://md.research.elamur.ai/free/en/venues/plos-one/) ([PLoS ONE 2023](https://md.research.elamur.ai/free/en/venues/plos-one/2023/)) · **Publisher:** [Public Library of Science](https://md.research.elamur.ai/free/en/publishers/public-library-of-science/) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2311.09497) · [PDF](https://arxiv.org/pdf/2311.09497) · [DOI 10.48550/arxiv.2311.09497](https://doi.org/10.48550/arxiv.2311.09497) · [arXiv 2311.09497](https://arxiv.org/abs/2311.09497) · [OpenAlex](https://openalex.org/W4409103579)

**Citations:** 18 citations (OpenAlex, 2026-10-14) · [14](https://md.research.elamur.ai/free/en/papers/goldberg-2023-peer-reviews-peer-reviews/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Conferences, journals and funding bodies often score individual peer reviews. They use these scores to reward strong reviewers or as the outcome measure in experiments on the review process. This paper tests whether such scores can be trusted. At NeurIPS 2022, the authors recruited paper authors, reviewers and meta-reviewers to rate existing reviews on a 7-point scale. Raters gave an overall score plus scores on criteria such as constructiveness, coverage, understanding and substantiation.

The study combined two kinds of evidence. A randomized controlled trial showed some evaluators reviews padded with uninformative text. Observational analyses examined author ratings, agreement between evaluators, miscalibration, and how people turn criteria scores into an overall judgment.

Padding alone made reviews look better. Authors favored reviews that recommended accepting their own papers. Noise levels matched those in ordinary paper reviewing. So "reviewing reviews" carries the same weaknesses as peer review itself. Incentive schemes and experiments, including studies comparing LLM-written and human reviews, should therefore not treat these ratings as ground truth.

## Key points
- **Length bias (RCT):** Reviews artificially lengthened with non-informative content scored about 0.5 points higher on the 7-point scale (Mann-Whitney U, τ = 0.64, p \< 0.0001). The effect also appeared in individual criteria scores. Evaluators in this trial were people who had not reviewed the paper in question.
- **Author-outcome bias:** Authors rated "reject" reviews of their own papers roughly 1.4 points below "accept" reviews (τ = 0.82, p \< 0.0001). This held after controlling for review length, quality (estimated from non-author evaluations) and how many accepted or rejected papers each author had.
- **Inconsistency:** Two evaluators of the same review disagreed 28–32% of the time, a rate close to the disagreement among NeurIPS reviewers scoring papers.
- **Miscalibration and subjectivity:** Miscalibration was estimated with a linear model of scores. Subjectivity was gauged by fitting a function that converts criteria scores into overall scores and taking that function's remaining loss as the measure. Both were comparable to NeurIPS paper review.
- **Limitations:** Dishonest behavior, such as collusion rings, could not be studied because the ratings had no consequences. The 7-point scale also differs from the 10-point scale used for NeurIPS paper reviews.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 533 works · 443,885 citations
- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 633 works · 195,218 citations
- [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) — University, New York, United States · 233 works · 104,003 citations
- [Korea Advanced Institute of Science and Technology](https://md.research.elamur.ai/free/en/universities/korea-advanced-institute-of-science-and-technology/) — University, Daejeon, South Korea · 59 works · 3,320 citations

Also on the byline: [New Economic School](https://ror.org/033904843) · [GlaxoSmithKline Research & Development Limited](https://ror.org/05mayyh83).

📚 Shelves (6): [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,673 works · [💭 Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) · 950 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 306 works · [🎭 History and Philosophy of Science](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/history-and-philosophy-of-science/) · 83 works · [🎯 Statistics, Probability and Uncertainty](https://md.research.elamur.ai/free/en/shelves/decision-sciences/statistics-probability-and-uncertainty/) · 67 works

## Cite

Cite as: Alexander Goldberg, Ivan Stelmakh, Kyunghyun Cho, Alice Oh, Alekh Agarwal, Danielle Belgrave and Nihar B. Shah. 2023. Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Experiments. arXiv:2311.09497.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{goldberg2023peer,
  title = {{Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Experiments}},
  author = {Alexander Goldberg, Ivan Stelmakh, Kyunghyun Cho, Alice Oh, Alekh Agarwal, Danielle Belgrave and Nihar B. Shah},
  year = {2023},
  doi = {10.48550/arxiv.2311.09497},
  eprint = {2311.09497},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.09497}
}
```

- [← previous Nash Learning from Human Feedback](https://md.research.elamur.ai/free/en/papers/munos-2023-nash-learning-human-feedback/)
- [next → PEIT: Bridging the Modality Gap with Pre-trained Models for End-to-End Image Translation](https://md.research.elamur.ai/free/en/papers/zhu-2023-peit-bridging-modality-gap-pre/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/12/)

## References

53 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Clare Ardern, Nadia Martino, Sammy Nag, Robyn Tamblyn, David Moher, Adrian Mota, and Karim Khan. Three years of quality assurance data assessing the performance of over 4000 grant peer review contributions to the Canadian institutes of health research project grant competition. FACETS, 2023. [find](https://scholar.google.com/scholar?q=Clare+Ardern%2C+Nadia+Martino%2C+Sammy+Nag%2C+Robyn+Tamblyn%2C+David+Moher%2C+Adrian+Mota%2C+and+Karim+Khan%2E+Three+years+of+quality+assurance+data+assessing+the+performance+of+over+4000+grant+peer+review+contribu)
2. Stefano Balietti, Robert Goldstone, and Dirk Helbing. Peer review and competition in the art exhibition game. Proceedings of the National Academy of Sciences, 2016. [find](https://scholar.google.com/scholar?q=Stefano+Balietti%2C+Robert+Goldstone%2C+and+Dirk+Helbing%2E+Peer+review+and+competition+in+the+art+exhibition+game%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+2016%2E)
3. Michael L Callaham, William G Baxt, Joseph F Waeckerle, and Robert L Wears. Reliability of editors’ subjective quality ratings of peer reviews of manuscripts. Jama, 280(3):229–231, 1998. [find](https://scholar.google.com/scholar?q=Michael+L+Callaham%2C+William+G+Baxt%2C+Joseph+F+Waeckerle%2C+and+Robert+L+Wears%2E+Reliability+of+editors%E2%80%99+subjective+quality+ratings+of+peer+reviews+of+manuscripts%2E+Jama%2C+280%283%29%3A229%E2%80%93231%2C+1998%2E)
4. Michael L. Callaham, Robert K. Knopp, and E. John Gallagher. Effect of Written Feedback by Editors on Quality of Reviews: Two Randomized Trials. JAMA, 287(21):2781–2783, 06 2002. [find](https://scholar.google.com/scholar?q=Michael+L%2E+Callaham%2C+Robert+K%2E+Knopp%2C+and+E%2E+John+Gallagher%2E+Effect+of+Written+Feedback+by+Editors+on+Quality+of+Reviews%3A+Two+Randomized+Trials%2E+JAMA%2C+287%2821%29%3A2781%E2%80%932783%2C+06+2002%2E)
5. Corinna Cortes and Neil D. Lawrence. Inconsistency in conference peer review: Revisiting the 2014 NeurIPS experiment, 2021. [find](https://scholar.google.com/scholar?q=Corinna+Cortes+and+Neil+D%2E+Lawrence%2E+Inconsistency+in+conference+peer+review%3A+Revisiting+the+2014+NeurIPS+experiment%2C+2021%2E)
6. Michael Callaham and Charles McCulloch. Longitudinal trends in the performance of scientific peer reviewers. Annals of emergency medicine, 57(2):141–148, 2011. [find](https://scholar.google.com/scholar?q=Michael+Callaham+and+Charles+McCulloch%2E+Longitudinal+trends+in+the+performance+of+scientific+peer+reviewers%2E+Annals+of+emergency+medicine%2C+57%282%29%3A141%E2%80%93148%2C+2011%2E)
7. Kevin C Chung, Melissa J Shauver, Sunitha Malay, Lin Zhong, Aaron Weinstein, and Rod J Rohrich. Is double-blinded peer review necessary? the effect of blinding on review quality. Plastic and reconstructive surgery, 136(6):1369–1377, 2015. [find](https://scholar.google.com/scholar?q=Kevin+C+Chung%2C+Melissa+J+Shauver%2C+Sunitha+Malay%2C+Lin+Zhong%2C+Aaron+Weinstein%2C+and+Rod+J+Rohrich%2E+Is+double%2Dblinded+peer+review+necessary%3F+the+effect+of+blinding+on+review+quality%2E+Plastic+and+reconstru)
8. Michael L Callaham and John Tercier. The relationship of previous training and experience of journal peer reviewers to subsequent review quality. PLoS medicine, 4(1):e40, 2007. [find](https://scholar.google.com/scholar?q=Michael+L+Callaham+and+John+Tercier%2E+The+relationship+of+previous+training+and+experience+of+journal+peer+reviewers+to+subsequent+review+quality%2E+PLoS+medicine%2C+4%281%29%3Ae40%2C+2007%2E)
9. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024. [find](https://scholar.google.com/scholar?q=Yann+Dubois%2C+Bala%CC%81zs+Galambosi%2C+Percy+Liang%2C+and+Tatsunori+B+Hashimoto%2E+Length%2Dcontrolled+alpacaeval%3A+A+simple+way+to+debias+automatic+evaluators%2E+arXiv+preprint+arXiv%3A2404%2E04475%2C+2024%2E)
10. Mike D’Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. [MARG: multi-agent review generation for scientific papers](https://md.research.elamur.ai/free/en/papers/arcy-2024-marg-multi-agent-review/). arXiv preprint arXiv:2401.04259, 2024.
11. Komal Dhull, Steven Jecmen, Pravesh Kothari, and Nihar B Shah. Strategyproofing peer assessment via partitioning: The price in terms of evaluators’ expertise. In HCOMP, 2022. [find](https://scholar.google.com/scholar?q=Komal+Dhull%2C+Steven+Jecmen%2C+Pravesh+Kothari%2C+and+Nihar+B+Shah%2E+Strategyproofing+peer+assessment+via+partitioning%3A+The+price+in+terms+of+evaluators%E2%80%99+expertise%2E+In+HCOMP%2C+2022%2E)
12. Esin Durmus, Faisal Ladhak, and Tatsunori Hashimoto. Spurious correlations in reference-free evaluation of text generation. arXiv preprint arXiv:2204.09890, 2022. [find](https://scholar.google.com/scholar?q=Esin+Durmus%2C+Faisal+Ladhak%2C+and+Tatsunori+Hashimoto%2E+Spurious+correlations+in+reference%2Dfree+evaluation+of+text+generation%2E+arXiv+preprint+arXiv%3A2204%2E09890%2C+2022%2E)
13. Arthur T Evans, Robert A McNutt, Suzanne W Fletcher, and Robert H Fletcher. The characteristics of peer reviewers who produce good-quality reviews. Journal of general internal medicine, 8:422–428, 1993. [find](https://scholar.google.com/scholar?q=Arthur+T+Evans%2C+Robert+A+McNutt%2C+Suzanne+W+Fletcher%2C+and+Robert+H+Fletcher%2E+The+characteristics+of+peer+reviewers+who+produce+good%2Dquality+reviews%2E+Journal+of+general+internal+medicine%2C+8%3A422%E2%80%93428%2C+199)
14. Irene D Feurer, Gary J Becker, Daniel Picus, Estella Ramirez, Michael D Darcy, and Marshall E Hicks. Evaluating peer reviews: pilot testing of a grading instrument. JAMA, 272(2):98–100, 1994. [find](https://scholar.google.com/scholar?q=Irene+D+Feurer%2C+Gary+J+Becker%2C+Daniel+Picus%2C+Estella+Ramirez%2C+Michael+D+Darcy%2C+and+Marshall+E+Hicks%2E+Evaluating+peer+reviews%3A+pilot+testing+of+a+grading+instrument%2E+JAMA%2C+272%282%29%3A98%E2%80%93100%2C+1994%2E)
15. European Science Foundation. ESF survey analysis report on peer review practices, 2011. Available online [https://www.esf.org/fileadmin/user_upload/esf/PeerReview-Practices_Survey2011.pdf](https://www.esf.org/fileadmin/user_upload/esf/PeerReview-Practices_Survey2011.pdf). [↗](https://www.esf.org/fileadmin/user_upload/esf/PeerReview-Practices_Survey2011.pdf)
16. H. Ge, M. Welling, and Z. Ghahramani. A Bayesian model for calibrating conference review scores, 2013. Available online [http://mlg.eng.cam.ac.uk/hong/unpublished/nips-review-model.pdf](http://mlg.eng.cam.ac.uk/hong/unpublished/nips-review-model.pdf) Last accessed: April 4, 2021. [↗](http://mlg.eng.cam.ac.uk/hong/unpublished/nips-review-model.pdf)
17. Steven Jecmen, Hanrui Zhang, Ryan Liu, Nihar B. Shah, Vincent Conitzer, and Fei Fang. Mitigating manipulation in peer review via randomized reviewer assignments. In NeurIPS, 2020. [find](https://scholar.google.com/scholar?q=Steven+Jecmen%2C+Hanrui+Zhang%2C+Ryan+Liu%2C+Nihar+B%2E+Shah%2C+Vincent+Conitzer%2C+and+Fei+Fang%2E+Mitigating+manipulation+in+peer+review+via+randomized+reviewer+assignments%2E+In+NeurIPS%2C+2020%2E)
18. Conny Kühne, Klemens Böhm, and Jing Zhi Yue. Reviewing the reviewers: A study of author perception on peer reviews in computer science. In 6th International Conference on Collaborative Computing: Networking, Applications and Worksharing (CollaborateCom 2010), pages 1–8, 2010. [find](https://scholar.google.com/scholar?q=Conny+Ku%CC%88hne%2C+Klemens+Bo%CC%88hm%2C+and+Jing+Zhi+Yue%2E+Reviewing+the+reviewers%3A+A+study+of+author+perception+on+peer+reviews+in+computer+science%2E+In+6th+International+Conference+on+Collaborative+Computing%3A+Ne)
19. Maria K Kowalczuk, Frank Dudbridge, Shreeya Nanda, Stephanie L Harriman, Jigisha Patel, and Elizabeth C Moylan. Retrospective analysis of the quality of reports by author-suggested and non-author-suggested reviewers in journals operating on open or single-blind peer review models. Bmj Open, 5(9):e00… [find](https://scholar.google.com/scholar?q=Maria+K+Kowalczuk%2C+Frank+Dudbridge%2C+Shreeya+Nanda%2C+Stephanie+L+Harriman%2C+Jigisha+Patel%2C+and+Elizabeth+C+Moylan%2E+Retrospective+analysis+of+the+quality+of+reports+by+author%2Dsuggested+and+non%2Dauthor%2Dsugg)
20. Aditya Khosla, Derek Hoiem, and Serge Belongie. Analysis of reviews for cvpr 2012, 2013. [find](https://scholar.google.com/scholar?q=Aditya+Khosla%2C+Derek+Hoiem%2C+and+Serge+Belongie%2E+Analysis+of+reviews+for+cvpr+2012%2C+2013%2E)
21. Hsuan-Tien Lin, Maria Florina Balcan, Raia Hadsell, and Marc’Aurelio Ranzato. What we learned from NeurIPS 2020 reviewing process, Oct 2020. [find](https://scholar.google.com/scholar?q=Hsuan%2DTien+Lin%2C+Maria+Florina+Balcan%2C+Raia+Hadsell%2C+and+Marc%E2%80%99Aurelio+Ranzato%2E+What+we+learned+from+NeurIPS+2020+reviewing+process%2C+Oct+2020%2E)
22. Carole J Lee. Commensuration bias in peer review. Philosophy of Science, 82(5):1272–1283, 2015. [find](https://scholar.google.com/scholar?q=Carole+J+Lee%2E+Commensuration+bias+in+peer+review%2E+Philosophy+of+Science%2C+82%285%29%3A1272%E2%80%931283%2C+2015%2E)
23. Minhyeok Lee. Game-theoretical analysis of reviewer rewards in peer-review journal systems: Analysis and experimental evaluation using deep reinforcement learning. arXiv preprint arXiv:2305.12088, 2023. — [arXiv:2305.12088](https://arxiv.org/abs/2305.12088) [find](https://scholar.google.com/scholar?q=Minhyeok+Lee%2E+Game%2Dtheoretical+analysis+of+reviewer+rewards+in+peer%2Dreview+journal+systems%3A+Analysis+and+experimental+evaluation+using+deep+reinforcement+learning%2E+arXiv+preprint+arXiv%3A2305%2E12088%2C+202)
24. Michael L Littman. Collusion rings threaten the integrity of computer science research. Communications of the ACM, 64(6):43–44, 2021. [find](https://scholar.google.com/scholar?q=Michael+L+Littman%2E+Collusion+rings+threaten+the+integrity+of+computer+science+research%2E+Communications+of+the+ACM%2C+64%286%29%3A43%E2%80%9344%2C+2021%2E)
25. Ryan Liu and Nihar B Shah. [Reviewergpt? an exploratory study on using large language models for paper reviewing](https://md.research.elamur.ai/free/en/papers/liu-2023-reviewergpt-exploratory-study-using/). arXiv preprint arXiv:2306.00622, 2023.
26. Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Smith, Yian Yin, Daniel McFarland, and James Zou. [Can large language models provide useful feedback on research papers? a large-scale empirical analysis](https://md.research.elamur.ai/free/en/papers/liang-2023-large-language-models-provide/), 2023.
27. Abdelghani Maddi and Egidio Luis Miotti. On the peer review reports: It’s not the size that matters... really? arXiv preprint arXiv:2309.02000, 2023. — [arXiv:2309.02000](https://arxiv.org/abs/2309.02000) [find](https://scholar.google.com/scholar?q=Abdelghani+Maddi+and+Egidio+Luis+Miotti%2E+On+the+peer+review+reports%3A+It%E2%80%99s+not+the+size+that+matters%2E%2E%2E+really%3F+arXiv+preprint+arXiv%3A2309%2E02000%2C+2023%2E+%E2%80%94+%5BarXiv%3A2309%2E02000%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2309%2E02)
28. Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734, 2024. [find](https://scholar.google.com/scholar?q=Yu+Meng%2C+Mengzhou+Xia%2C+and+Danqi+Chen%2E+Simpo%3A+Simple+preference+optimization+with+a+reference%2Dfree+reward%2E+arXiv+preprint+arXiv%3A2405%2E14734%2C+2024%2E)
29. Ritesh Noothigattu, Nihar Shah, and Ariel Procaccia. Loss functions, axioms, and peer review. J. Artif. Int. Res., 70:1481–1515, may 2021. [find](https://scholar.google.com/scholar?q=Ritesh+Noothigattu%2C+Nihar+Shah%2C+and+Ariel+Procaccia%2E+Loss+functions%2C+axioms%2C+and+peer+review%2E+J%2E+Artif%2E+Int%2E+Res%2E%2C+70%3A1481%E2%80%931515%2C+may+2021%2E)
30. Konstantina Papagiannaki. Author feedback experiment at pam 2007. ACM SIGCOMM Computer Communication Review, 37(3):73–78, 2007. [find](https://scholar.google.com/scholar?q=Konstantina+Papagiannaki%2E+Author+feedback+experiment+at+pam+2007%2E+ACM+SIGCOMM+Computer+Communication+Review%2C+37%283%29%3A73%E2%80%9378%2C+2007%2E)
31. Piitu Parmanne, Joonas Laajava, Noora Järvinen, Terttu Harju, Mauri Marttunen, and Pertti Saloheimo. Peer reviewers’ willingness to review, their recommendations and quality of reviews after the finnish medical journal switched from single-blind to double-blind peer review. Research Integrity and P… [find](https://scholar.google.com/scholar?q=Piitu+Parmanne%2C+Joonas+Laajava%2C+Noora+Ja%CC%88rvinen%2C+Terttu+Harju%2C+Mauri+Marttunen%2C+and+Pertti+Saloheimo%2E+Peer+reviewers%E2%80%99+willingness+to+review%2C+their+recommendations+and+quality+of+reviews+after+the+finn)
32. Shelly M. Pranić, Mario Malički, Stjepan Ljudevit Marušić, Bahar Mehmani, and Ana Marušić. Is the quality of reviews reflected in editors’ and authors’ satisfaction with peer review? a cross-sectional study in 12 journals across four research fields. Learned Publishing, 34(2):187–197, 2021. [find](https://scholar.google.com/scholar?q=Shelly+M%2E+Pranic%CC%81%2C+Mario+Malic%CC%8Cki%2C+Stjepan+Ljudevit+Marus%CC%8Cic%CC%81%2C+Bahar+Mehmani%2C+and+Ana+Marus%CC%8Cic%CC%81%2E+Is+the+quality+of+reviews+reflected+in+editors%E2%80%99+and+authors%E2%80%99+satisfaction+with+peer+review%3F+a+cross%2Dsec)
33. Magnus Roos, Jörg Rothe, Joachim Rudolph, Björn Scheuermann, and Dietrich Stoyan. A statistical approach to calibrating the scores of biased reviewers: The linear vs. the nonlinear model. In Multidisciplinary Workshop on Advances in Preference Handling, 2012. [find](https://scholar.google.com/scholar?q=Magnus+Roos%2C+Jo%CC%88rg+Rothe%2C+Joachim+Rudolph%2C+Bjo%CC%88rn+Scheuermann%2C+and+Dietrich+Stoyan%2E+A+statistical+approach+to+calibrating+the+scores+of+biased+reviewers%3A+The+linear+vs%2E+the+nonlinear+model%2E+In+Multidi)
34. Charvi Rastogi, Ivan Stelmakh, Alina Beygelzimer, Yann N Dauphin, Percy Liang, Jennifer Wortman Vaughan, Zhenyu Xue, Hal Daumé III, Emma Pierson, and Nihar B Shah. How do authors’ perceptions of their papers compare with co-authors’ perceptions and peer-review decisions? arXiv:2211.12966. Short blo… [find](https://scholar.google.com/scholar?q=Charvi+Rastogi%2C+Ivan+Stelmakh%2C+Alina+Beygelzimer%2C+Yann+N+Dauphin%2C+Percy+Liang%2C+Jennifer+Wortman+Vaughan%2C+Zhenyu+Xue%2C+Hal+Daume%CC%81+III%2C+Emma+Pierson%2C+and+Nihar+B+Shah%2E+How+do+authors%E2%80%99+perceptions+of+thei)
35. Sara Schroter, Nick Black, Stephen Evans, James Carpenter, Fiona Godlee, and Richard Smith. Effects of training on quality of peer review: randomised controlled trial. Bmj, 328(7441):673, 2004. [find](https://scholar.google.com/scholar?q=Sara+Schroter%2C+Nick+Black%2C+Stephen+Evans%2C+James+Carpenter%2C+Fiona+Godlee%2C+and+Richard+Smith%2E+Effects+of+training+on+quality+of+peer+review%3A+randomised+controlled+trial%2E+Bmj%2C+328%287441%29%3A673%2C+2004%2E)
36. Cecilia Superchi, José Antonio González, Ivan Solà, Erik Cobo, Darko Hren, and Isabelle Boutron. Tools used to assess the quality of peer review reports: a methodological systematic review. BMC medical research methodology, 19:1–14, 2019. [find](https://scholar.google.com/scholar?q=Cecilia+Superchi%2C+Jose%CC%81+Antonio+Gonza%CC%81lez%2C+Ivan+Sola%CC%80%2C+Erik+Cobo%2C+Darko+Hren%2C+and+Isabelle+Boutron%2E+Tools+used+to+assess+the+quality+of+peer+review+reports%3A+a+methodological+systematic+review%2E+BMC+med)
37. Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. [A long way to go: Investigating length correlations in RLHF](https://md.research.elamur.ai/free/en/papers/singhal-2023-long-way-go-investigating-length/). arXiv preprint arXiv:2310.03716, 2023.
38. Nihar B Shah. Challenges, experiments, and computational solutions in peer review. Communications of the ACM, 65(6):76–87, 2022. [find](https://scholar.google.com/scholar?q=Nihar+B+Shah%2E+Challenges%2C+experiments%2C+and+computational+solutions+in+peer+review%2E+Communications+of+the+ACM%2C+65%286%29%3A76%E2%80%9387%2C+2022%2E)
39. Siddarth Srinivasan and Jamie Morgenstern. Auctions and prediction markets for scientific peer review. arXiv preprint arXiv:2109.00923, 2021. — [arXiv:2109.00923](https://arxiv.org/abs/2109.00923) [find](https://scholar.google.com/scholar?q=Siddarth+Srinivasan+and+Jamie+Morgenstern%2E+Auctions+and+prediction+markets+for+scientific+peer+review%2E+arXiv+preprint+arXiv%3A2109%2E00923%2C+2021%2E+%E2%80%94+%5BarXiv%3A2109%2E00923%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2109%2E00923%29)
40. Ivan Stelmakh, Nihar B Shah, Aarti Singh, and Hal Daumé III. A novice-reviewer experiment to address scarcity of qualified reviewers in large conferences. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 4785–4793, 2021. [find](https://scholar.google.com/scholar?q=Ivan+Stelmakh%2C+Nihar+B+Shah%2C+Aarti+Singh%2C+and+Hal+Daume%CC%81+III%2E+A+novice%2Dreviewer+experiment+to+address+scarcity+of+qualified+reviewers+in+large+conferences%2E+In+Proceedings+of+the+AAAI+Conference+on+Art)
41. Nihar B. Shah, Behzad Tabibian, Krikamol Muandet, Isabelle Guyon, and Ulrike von Luxburg. Design and analysis of the NIPS 2016 review process. Journal of Machine Learning Research, 19(49):1–34, 2018. [find](https://scholar.google.com/scholar?q=Nihar+B%2E+Shah%2C+Behzad+Tabibian%2C+Krikamol+Muandet%2C+Isabelle+Guyon%2C+and+Ulrike+von+Luxburg%2E+Design+and+analysis+of+the+NIPS+2016+review+process%2E+Journal+of+Machine+Learning+Research%2C+19%2849%29%3A1%E2%80%9334%2C+2018%2E)
42. Alexander Ugarov. Peer prediction for peer review: arXiv:2303.16855, 2023. — [arXiv:2303.16855](https://arxiv.org/abs/2303.16855) [find](https://scholar.google.com/scholar?q=Alexander+Ugarov%2E+Peer+prediction+for+peer+review%3A+arXiv%3A2303%2E16855%2C+2023%2E+%E2%80%94+%5BarXiv%3A2303%2E16855%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2303%2E16855%29)
43. T. N. Vijaykumar. Potential organized fraud in ACM/IEEE computer architecture conferences. https://medium. com/@tnvijayk/potential-organized-fraud-in-acm-ieee-computer-architecture-conferences-ccd61169370d, 2020. [find](https://scholar.google.com/scholar?q=T%2E+N%2E+Vijaykumar%2E+Potential+organized+fraud+in+ACM%2FIEEE+computer+architecture+conferences%2E+https%3A%2F%2Fmedium%2E+com%2F%40tnvijayk%2Fpotential%2Dorganized%2Dfraud%2Din%2Dacm%2Dieee%2Dcomputer%2Darchitecture%2Dconferences%2Dccd6116)
44. Susan Van Rooyen, Nick Black, and Fiona Godlee. Development of the review quality instrument (rqi) for assessing peer reviews of manuscripts. Journal of clinical epidemiology, 52(7):625–629, 1999. [find](https://scholar.google.com/scholar?q=Susan+Van+Rooyen%2C+Nick+Black%2C+and+Fiona+Godlee%2E+Development+of+the+review+quality+instrument+%28rqi%29+for+assessing+peer+reviews+of+manuscripts%2E+Journal+of+clinical+epidemiology%2C+52%287%29%3A625%E2%80%93629%2C+1999%2E)
45. Susan Van Rooyen, Tony Delamothe, and Stephen JW Evans. Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial. Bmj, 341, 2010. designing a marketplace for ideas. arXiv preprint [find](https://scholar.google.com/scholar?q=Susan+Van+Rooyen%2C+Tony+Delamothe%2C+and+Stephen+JW+Evans%2E+Effect+on+peer+review+of+telling+reviewers+that+their+signed+reviews+might+be+posted+on+the+web%3A+randomised+controlled+trial%2E+Bmj%2C+341%2C+2010%2E+de)
46. Susan Van Rooyen, Fiona Godlee, Stephen Evans, Nick Black, and Richard Smith. Effect of open peer review on quality of reviews and on reviewers’ recommendations: a randomised trial. Bmj, 318(7175):23–27, 1999. [find](https://scholar.google.com/scholar?q=Susan+Van+Rooyen%2C+Fiona+Godlee%2C+Stephen+Evans%2C+Nick+Black%2C+and+Richard+Smith%2E+Effect+of+open+peer+review+on+quality+of+reviews+and+on+reviewers%E2%80%99+recommendations%3A+a+randomised+trial%2E+Bmj%2C+318%287175%29%3A23%E2%80%93)
47. Ellen J Weber, Patricia P Katz, Joseph F Waeckerle, and Michael L Callaham. Author perception of peer review: impact of review quality and acceptance on satisfaction. JAMA, 287(21):2790–2793, 2002. [find](https://scholar.google.com/scholar?q=Ellen+J+Weber%2C+Patricia+P+Katz%2C+Joseph+F+Waeckerle%2C+and+Michael+L+Callaham%2E+Author+perception+of+peer+review%3A+impact+of+review+quality+and+acceptance+on+satisfaction%2E+JAMA%2C+287%2821%29%3A2790%E2%80%932793%2C+2002%2E)
48. Elizabeth Walsh, Maeve Rooney, Louis Appleby, and Greg Wilkinson. Open peer review: A randomised controlled trial. The British Journal of Psychiatry, 176(1):47–51, 2000. [find](https://scholar.google.com/scholar?q=Elizabeth+Walsh%2C+Maeve+Rooney%2C+Louis+Appleby%2C+and+Greg+Wilkinson%2E+Open+peer+review%3A+A+randomised+controlled+trial%2E+The+British+Journal+of+Psychiatry%2C+176%281%29%3A47%E2%80%9351%2C+2000%2E)
49. Jingyan Wang and Nihar B Shah. Your 2 is my 1, your 3 is my 9: Handling arbitrary miscalibrations in ratings. In AAMAS, 2019. [find](https://scholar.google.com/scholar?q=Jingyan+Wang+and+Nihar+B+Shah%2E+Your+2+is+my+1%2C+your+3+is+my+9%3A+Handling+arbitrary+miscalibrations+in+ratings%2E+In+AAMAS%2C+2019%2E)
50. Jingyan Wang, Ivan Stelmakh, Yuting Wei, and Nihar B Shah. Debiasing evaluations that are biased by evaluations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10120–10128, 2021. [find](https://scholar.google.com/scholar?q=Jingyan+Wang%2C+Ivan+Stelmakh%2C+Yuting+Wei%2C+and+Nihar+B+Shah%2E+Debiasing+evaluations+that+are+biased+by+evaluations%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volume+35%2C+pages+1012)
51. Yuanzhang Xiao, Florian Dörfler, and Mihaela van der Schaar. Rating and matching in peer review systems. In Allerton conference, 2014. [find](https://scholar.google.com/scholar?q=Yuanzhang+Xiao%2C+Florian+Do%CC%88rfler%2C+and+Mihaela+van+der+Schaar%2E+Rating+and+matching+in+peer+review+systems%2E+In+Allerton+conference%2C+2014%2E)
52. Yuanzhang Xiao, Florian Dörfler, and Mihaela Van Der Schaar. Incentive design in peer review: Rating and repeated endogenous matching. IEEE Transactions on Network Science and Engineering, 2018. [find](https://scholar.google.com/scholar?q=Yuanzhang+Xiao%2C+Florian+Do%CC%88rfler%2C+and+Mihaela+Van+Der+Schaar%2E+Incentive+design+in+peer+review%3A+Rating+and+repeated+endogenous+matching%2E+IEEE+Transactions+on+Network+Science+and+Engineering%2C+2018%2E)
53. Yichong Xu, Han Zhao, Xiaofei Shi, and Nihar Shah. On strategyproof conference review. In IJCAI, 2019. [find](https://scholar.google.com/scholar?q=Yichong+Xu%2C+Han+Zhao%2C+Xiaofei+Shi%2C+and+Nihar+Shah%2E+On+strategyproof+conference+review%2E+In+IJCAI%2C+2019%2E)

*Updated 3 November 2026*