---
title: "Illusions of the Gold Standard — paper by Mei et al. (2026)"
description: "Drawing on reproducibility research, they build a codebook of 37 questions, 20 of which are binary core criteria. 21 references."
authors: ["Katelyn X. Mei", "Yi-Li Hsu", "Minjoon Choi", "Zongwan Cao", "Chenjun Xu", "Bingbing Wen", "Su Lin Blodgett", "Lucy Lu Wang"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.635/"
doi: "10.18653/v1/2026.acl-long.635"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mei-2026-illusions-gold-standard-large-scale/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mei-2026-illusions-gold-standard-large-scale/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/33/) / Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generation

# Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generation

**Authors:** Katelyn X. Mei, Yi-Li Hsu, Minjoon Choi, Zongwan Cao, Chenjun Xu, Bingbing Wen, [Su Lin Blodgett](https://md.research.elamur.ai/free/en/authors/su-lin-blodgett/) and [Lucy Lu Wang](https://md.research.elamur.ai/free/en/authors/lucy-lu-wang/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.635/) · [PDF](https://aclanthology.org/2026.acl-long.635.pdf) · [DOI 10.18653/v1/2026.acl-long.635](https://doi.org/10.18653/v1/2026.acl-long.635) · [OpenAlex](https://openalex.org/W7166864851)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary

Human judgment is still treated as the most trustworthy way to assess long-form or open-ended text produced by LLMs. Yet whether such studies can be interpreted or repeated depends on how thoroughly authors describe their protocols. The authors audit this documentation across *CL venues from 2023 to 2025. Drawing on reproducibility research, they build a codebook of 37 questions, 20 of which are binary core criteria. These are grouped into task documentation, annotation design, and analysis & interpretation.

From more than 9.1k papers, they identify over 1,800 that evaluate long-form generation with human raters. They code 284 of these fully by hand and process the rest with LLM-assisted annotation.

Most papers name the evaluated dimensions, the sample size, and some annotator details. Many other elements are frequently missing:

- task guidelines
- justification for the chosen dimensions
- power analysis
- statistical confidence

The share of papers relying on human evaluation is also shrinking, while LLM-judges are spreading. This leaves real uncertainty about what was measured and by whom. The paper ends with practical reporting recommendations for the community.

## Key points

- **Corpus:** 9.1k+ *CL papers (2023–2025). Of these, 1,800+ study long-form generation with human evaluation; 356 were sampled and 284 manually annotated. Roughly half of 2025 *CL papers involve long-form generation.
- **Criteria:** 20 core criteria, split into task documentation (4), annotation design (9), and analysis & interpretation (7). The codebook was refined over four rounds, each applied to five papers. A typical study is expected to satisfy about 15–16 criteria.
- **Under-reporting:** Only about half of papers provide annotator guidelines or a rationale for the evaluated dimensions. Power analysis and confidence reporting are very uncommon.
- **Trend:** Use of human evaluation is falling, while use of LLM-judges is rising. Human judgments increasingly serve to meta-evaluate those judges.
- **Resources and validation:** LLM annotations served only as supplementary evidence and had to reach a validation accuracy threshold of 0.75. Code and the annotated dataset are available at github.com/larchlab/Illusions-of-the-Gold-Standard.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 353 works · 200,043 citations
- [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) — Research institute, Seattle, United States · 275 works · 97,634 citations
- [Seoul National University](https://md.research.elamur.ai/free/en/universities/seoul-national-university/) — University, Seoul, South Korea · 41 works · 45,671 citations
- [Mila - Quebec Artificial Intelligence Institute](https://md.research.elamur.ai/free/en/labs/mila-quebec-artificial-intelligence-institute/) — Research institute, Montreal, Canada · 54 works · 13,565 citations
- [National Tsing Hua University](https://md.research.elamur.ai/free/en/universities/national-tsing-hua-university/) — University, Hsinchu, Taiwan · 8 works · 834 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 General Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/general-social-sciences/) · 227 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [✍️ Text generation](https://md.research.elamur.ai/free/en/topics/text-generation/) · 202 works

## Cite

Cite as: Katelyn X. Mei, Yi-Li Hsu, Minjoon Choi, Zongwan Cao, Chenjun Xu, Bingbing Wen, Su Lin Blodgett and Lucy Lu Wang. 2026. Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mei2026illusions,
  title = {{Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generation}},
  author = {Katelyn X. Mei, Yi-Li Hsu, Minjoon Choi, Zongwan Cao, Chenjun Xu, Bingbing Wen, Su Lin Blodgett and Lucy Lu Wang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.635},
  url = {https://doi.org/10.18653/v1/2026.acl-long.635}
}
```

- [← previous IIITK_SpeechScape@DravidianLangTech 2026: Dialect based speech recognition and classification using Speech Foundation Models and Deep Learning Techniques](https://md.research.elamur.ai/free/en/papers/sekar-2026-iiitk-speechscape-dravidianlangtech-2026-dialect/)
- [next → ILSIC: Corpora for Identifying Indian Legal Statutes from Queries by Laymen](https://md.research.elamur.ai/free/en/papers/paul-2026-ilsic-corpora-identifying-indian-legal/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/33/)

## References

21 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. evaluation framework: Rethinking human evaluation for generative large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1137–1160. [find](https://scholar.google.com/scholar?q=evaluation+framework%3A+Rethinking+human+evaluation+for+generative+large+language+models%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Paper)
2. Zhiwei Fei, Songyang Zhang, Xiaoyu Shen, Dawei Zhu, Xiao Wang, Jidong Ge, and Vincent Ng. 2025. Internlm-law: An open-sourced chinese legal large language model. In Proceedings of the 31st International Conference on Computational Linguistics, pages 9376–9392. [find](https://scholar.google.com/scholar?q=Zhiwei+Fei%2C+Songyang+Zhang%2C+Xiaoyu+Shen%2C+Dawei+Zhu%2C+Xiao+Wang%2C+Jidong+Ge%2C+and+Vincent+Ng%2E+2025%2E+Internlm%2Dlaw%3A+An+open%2Dsourced+chinese+legal+large+language+model%2E+In+Proceedings+of+the+31st+Internation)
3. Jennifer Fereday and Eimear Muir-Cochrane. 2006. Demonstrating rigor using thematic analysis: A hybrid approach of inductive and deductive coding and theme development. International journal of qualitative methods, 5(1):80–92. [find](https://scholar.google.com/scholar?q=Jennifer+Fereday+and+Eimear+Muir%2DCochrane%2E+2006%2E+Demonstrating+rigor+using+thematic+analysis%3A+A+hybrid+approach+of+inductive+and+deductive+coding+and+theme+development%2E+International+journal+of+qualit)
4. Eve Fleisig, Su Lin Blodgett, Dan Klein, and Zeerak Talat. 2024. The perspectivist paradigm shift: Assumptions and challenges of capturing human labels. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (… [find](https://scholar.google.com/scholar?q=Eve+Fleisig%2C+Su+Lin+Blodgett%2C+Dan+Klein%2C+and+Zeerak+Talat%2E+2024%2E+The+perspectivist+paradigm+shift%3A+Assumptions+and+challenges+of+capturing+human+labels%2E+In+Proceedings+of+the+2024+Conference+of+the+No)
5. David Fraile Navarro, Enrico Coiera, Thomas W Hambly, Zoe Triplett, Nahyan Asif, Anindya Susanto, Anamika Chowdhury, Amaya Azcoaga Lorenzo, Mark Dras, and Shlomo Berkovsky. 2025. Expert evaluation of large language models for clinical dialogue summarization. Scientific reports, 15(1):1195. [find](https://scholar.google.com/scholar?q=David+Fraile+Navarro%2C+Enrico+Coiera%2C+Thomas+W+Hambly%2C+Zoe+Triplett%2C+Nahyan+Asif%2C+Anindya+Susanto%2C+Anamika+Chowdhury%2C+Amaya+Azcoaga+Lorenzo%2C+Mark+Dras%2C+and+Shlomo+Berkovsky%2E+2025%2E+Expert+evaluation+of+)
6. David M Howcroft, Anja Belz, Miruna Clinciu, Dimitra Gkatzia, Sadid A Hasan, Saad Mahamood, Simon Mille, Emiel Van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: Nlg needs evaluation sheets and standardised definitions. In Proceedings of the 13… [find](https://scholar.google.com/scholar?q=David+M+Howcroft%2C+Anja+Belz%2C+Miruna+Clinciu%2C+Dimitra+Gkatzia%2C+Sadid+A+Hasan%2C+Saad+Mahamood%2C+Simon+Mille%2C+Emiel+Van+Miltenburg%2C+Sashank+Santhanam%2C+and+Verena+Rieser%2E+2020%2E+Twenty+years+of+confusion+in+)
7. Maximilian Idahl and Zahra Ahmadi. 2025. OpenReviewer: A specialized large language model for generating critical scientific paper reviews. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Syst… [find](https://scholar.google.com/scholar?q=Maximilian+Idahl+and+Zahra+Ahmadi%2E+2025%2E+OpenReviewer%3A+A+specialized+large+language+model+for+generating+critical+scientific+paper+reviews%2E+In+Proceedings+of+the+2025+Conference+of+the+Nations+of+the+)
8. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534–46594. [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+et+al%2E+2023%2E+Self%2Drefine%3A+Iterative+refinement+with+self%2Df)
9. Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and Hannaneh Hajishirzi. 2024. [Fine-grained hallucination detection and editing for language models](https://md.research.elamur.ai/free/en/papers/mishra-2024-fine-grained-hallucination-detection-editing/). In First Conference on Language Modeling.
10. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pages 220–229. [find](https://scholar.google.com/scholar?q=Margaret+Mitchell%2C+Simone+Wu%2C+Andrew+Zaldivar%2C+Parker+Barnes%2C+Lucy+Vasserman%2C+Ben+Hutchinson%2C+Elena+Spitzer%2C+Inioluwa+Deborah+Raji%2C+and+Timnit+Gebru%2E+2019%2E+Model+cards+for+model+reporting%2E+In+Proceedi)
11. Marcus R Munafò, Brian A Nosek, Dorothy VM Bishop, Katherine S Button, Christopher D Chambers, Nathalie Percie du Sert, Uri Simonsohn, Eric-Jan Wagenmakers, Jennifer J Ware, and John PA Ioannidis. 2017. A manifesto for reproducible science. Nature human behaviour, 1(1):0021. [find](https://scholar.google.com/scholar?q=Marcus+R+Munaf%C3%B2%2C+Brian+A+Nosek%2C+Dorothy+VM+Bishop%2C+Katherine+S+Button%2C+Christopher+D+Chambers%2C+Nathalie+Percie+du+Sert%2C+Uri+Simonsohn%2C+Eric%2DJan+Wagenmakers%2C+Jennifer+J+Ware%2C+and+John+PA+Ioannidis%2E+201)
12. Eric A Posner and Shivam Saran. 2025. Judge ai: Assessing large language models in judicial decision-making. University of Chicago Coase-Sandor Institute for Law & Economics Research Paper, (2503). [find](https://scholar.google.com/scholar?q=Eric+A+Posner+and+Shivam+Saran%2E+2025%2E+Judge+ai%3A+Assessing+large+language+models+in+judicial+decision%2Dmaking%2E+University+of+Chicago+Coase%2DSandor+Institute+for+Law+%26+Economics+Research+Paper%2C+%282503%29%2E)
13. Juan-Pablo Rivera, Gabriel Mukobi, Anka Reuel, Max Lamparth, Chandler Smith, and Jacquelyn Schneider. 2024. Escalation risks from language models in military and diplomatic decision-making. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 836–898. [find](https://scholar.google.com/scholar?q=Juan%2DPablo+Rivera%2C+Gabriel+Mukobi%2C+Anka+Reuel%2C+Max+Lamparth%2C+Chandler+Smith%2C+and+Jacquelyn+Schneider%2E+2024%2E+Escalation+risks+from+language+models+in+military+and+diplomatic+decision%2Dmaking%2E+In+Proceed)
14. Anna Rogers, Timothy Baldwin, and Kobi Leins. 2021. ‘just what do you think you’re doing, dave?’ a checklist for responsible data use in NLP. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4821–4833, Punta Cana, Dominican Republic. Association for Computational Lingu… [find](https://scholar.google.com/scholar?q=Anna+Rogers%2C+Timothy+Baldwin%2C+and+Kobi+Leins%2E+2021%2E+%E2%80%98just+what+do+you+think+you%E2%80%99re+doing%2C+dave%3F%E2%80%99+a+checklist+for+responsible+data+use+in+NLP%2E+In+Findings+of+the+Association+for+Computational+Linguisti)
15. Shaurya Rohatgi. 2022. Acl anthology corpus with full text. Github. [find](https://scholar.google.com/scholar?q=Shaurya+Rohatgi%2E+2022%2E+Acl+anthology+corpus+with+full+text%2E+Github%2E)
16. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A Smith. 2019. [The risk of racial bias in hate speech detection](https://md.research.elamur.ai/free/en/papers/sap-2019-risk-racial-bias-hate-speech/). In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 1668–1678.
17. Anastasia Shimorina and Anja Belz. 2022. The human evaluation datasheet: A template for recording details of human evaluation experiments in nlp. In Proceedings of the 2nd Workshop on Human Evaluation of NLP Systems (HumEval), pages 54–75. [find](https://scholar.google.com/scholar?q=Anastasia+Shimorina+and+Anja+Belz%2E+2022%2E+The+human+evaluation+datasheet%3A+A+template+for+recording+details+of+human+evaluation+experiments+in+nlp%2E+In+Proceedings+of+the+2nd+Workshop+on+Human+Evaluation)
18. Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. 2025. Judging the judges: Evaluating alignment and vulnerabilities in llms-as-judges. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM 2 ), pages 404–430. [find](https://scholar.google.com/scholar?q=Aman+Singh+Thakur%2C+Kartik+Choudhary%2C+Venkat+Srinik+Ramayapally%2C+Sankaran+Vaidyanathan%2C+and+Dieuwke+Hupkes%2E+2025%2E+Judging+the+judges%3A+Evaluating+alignment+and+vulnerabilities+in+llms%2Das%2Djudges%2E+In+Proc)
19. Craig Thomson, Ehud Reiter, and Anya Belz. 2024. Common flaws in running human evaluation experiments in nlp. Computational Linguistics, 50(2):795– 805. [find](https://scholar.google.com/scholar?q=Craig+Thomson%2C+Ehud+Reiter%2C+and+Anya+Belz%2E+2024%2E+Common+flaws+in+running+human+evaluation+experiments+in+nlp%2E+Computational+Linguistics%2C+50%282%29%3A795%E2%80%93+805%2E)
20. Lucy Lu Wang, Yulia Otmakhova, Jay DeYoung, Thinh Hung Truong, Bailey Kuehl, Erin Bransom, and Byron C. Wallace. 2023. Automated metrics for medical multi-document summarization disagree with human evaluations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Lucy+Lu+Wang%2C+Yulia+Otmakhova%2C+Jay+DeYoung%2C+Thinh+Hung+Truong%2C+Bailey+Kuehl%2C+Erin+Bransom%2C+and+Byron+C%2E+Wallace%2E+2023%2E+Automated+metrics+for+medical+multi%2Ddocument+summarization+disagree+with+human+ev)
21. Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi. 2023. [A critical evaluation of evaluations for long-form question answering](https://md.research.elamur.ai/free/en/papers/xu-2023-critical-evaluation-evaluations-long-form/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3225–3245, Toronto, Canada. Association for C…

*Updated 1 November 2026*