--- title: "Never Truly Out of Fashion — paper by Schmidtová et al. (2026)" description: "Automatic metrics and LLM judges are cheap, which tempts researchers to drop human evaluation of generated… 82.4% recall. 4 references." authors: ["Patrícia Schmidtová", "Saad Mahamood", "Ondřej Dušek"] year: 2026 venue: "Proceedings of the 1st Symposium on Natural Language Generation Evaluations" source: "https://aclanthology.org/2026.retroeval-main.8/" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/schmidtova-2026-never-truly-out-fashion-retrospective/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/schmidtova-2026-never-truly-out-fashion-retrospective/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/38/) / Never Truly Out of Fashion: A Retrospective Look at Evaluation in NLG # Never Truly Out of Fashion: A Retrospective Look at Evaluation in NLG **Authors:** [Patrícia Schmidtová](https://md.research.elamur.ai/free/en/authors/patricia-schmidtova/), [Saad Mahamood](https://md.research.elamur.ai/free/en/authors/saad-mahamood/) and [Ondřej Dušek](https://md.research.elamur.ai/free/en/authors/ondrej-dusek/) · **Year:** 2026 · **Venue:** [Proceedings of the 1st Symposium on Natural Language Generation Evaluations](https://md.research.elamur.ai/free/en/venues/retroeval/) ([Symposium on Natural Language Generation Evaluations 2026](https://md.research.elamur.ai/free/en/venues/retroeval/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.retroeval-main.8/) · [PDF](https://aclanthology.org/2026.retroeval-main.8.pdf) ## Summary Automatic metrics and LLM judges are cheap, which tempts researchers to drop human evaluation of generated text. The authors ask how NLG evaluation has actually changed over time. They compiled 85,792 ACL Anthology papers from ACL-OCL plus their own post-2022 crawl of nine core venues. Filtering left 24,291 generation-focused papers spanning 1952–2025. They then used precision-oriented regular expressions to detect evaluation types, criteria from the Howcroft et al. (2020) taxonomy, and annotation methods. An LLM-based extraction attempt had poor recall and was set aside. The results show that papers with no evaluation dominated before 1990, while evaluation is now nearly universal. Human evaluation has stayed flat at roughly 35–45% of papers over the last decade. LLM-as-a-judge has spread quickly since 2023, mostly alongside human evaluation, but often without it. Faithfulness has returned as a criterion, driven by hallucination research, while grammaticality and adequacy are fading. The study gives the field a measurable baseline for future comparison. It also argues for calibrating LLM judges and for standardised faithfulness annotation. ## Key points - **Human evaluation is stable:** 34–42% of NLG papers per year since 2018, with no significant trend over 2015–2025 (logistic regression β = 0.0023, p = 0.69). INLG reaches 53.1% compared with 36.0% at core NLP venues. - **LLM judges are growing fast:** 19 papers in 2023, 109 in 2024 and 224 in 2025. Of these, 61.4% also report human evaluation, while 38.6% rely on the LLM alone. - **Faithfulness is the fastest-rising criterion:** its share of human-evaluation papers went from 1.7% in 2015 to 29.8% in 2025. Among the 8,894 human-evaluation papers, the most frequent criteria are fluency (22.5%), relevance (21.6%) and coherence (16.6%). - **Methods and modalities:** - Automatic-only evaluation is the most common modality at 57.8%, and human-only evaluation has dropped to 2–3%. - Likert scales are the leading annotation method (19.0%), followed by post-editing (15.8%). - MT has the lowest human evaluation rate, at about 28%. - **Validation and data release:** the regex approach achieved 82.4% recall on 51 manually annotated papers. Code and the extracted signals (no full texts) are available at github.com/patuchen/trends_in_nlg_eval. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works 🧭 Topics (1): [✍️ Text generation](https://md.research.elamur.ai/free/en/topics/text-generation/) · 202 works ## Cite Cite as: Patrícia Schmidtová, Saad Mahamood and Ondřej Dušek. 2026. Never Truly Out of Fashion: A Retrospective Look at Evaluation in NLG. Proceedings of the 1st Symposium on Natural Language Generation Evaluations. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{schmidtova2026never, title = {{Never Truly Out of Fashion: A Retrospective Look at Evaluation in NLG}}, author = {Patrícia Schmidtová, Saad Mahamood and Ondřej Dušek}, booktitle = {Proceedings of the 1st Symposium on Natural Language Generation Evaluations}, year = {2026}, url = {https://aclanthology.org/2026.retroeval-main.8/} } ``` - [← previous Never Care For What They Say ? Platform vs Genre Rules in Online Horror Narratives (2007–2024)](https://md.research.elamur.ai/free/en/papers/rollin-2026-never-care-what-they-say/) - [next → New Compendium of a Myriad of Plants: A New Dataset Describing Ancient Chinese Plants](https://md.research.elamur.ai/free/en/papers/shen-2026-new-compendium-myriad-plants-new/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/38/) ## References 4 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Sur… — [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://md.research.elamur.ai/free/en/papers/bavaresco-2024-llms-instead-human-judges-large/) 2. Anja Belz and Ehud Reiter. 2006. Comparing Automatic and Human Evaluation of NLG Systems. In 11th Conference of the European Chapter of the Association for Computational Linguistics, pages 313– 320, Trento, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Anja+Belz+and+Ehud+Reiter%2E+2006%2E+Comparing+Automatic+and+Human+Evaluation+of+NLG+Systems%2E+In+11th+Conference+of+the+European+Chapter+of+the+Association+for+Computational+Linguistics%2C+pages+313%E2%80%93+320%2C+T) 3. Anya Belz, Simon Mille, and David M Howcroft. 2020. Disentangling the properties of human evaluation methods: A classification system to support comparability, meta-evaluation and reproducibility testing. Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Anya+Belz%2C+Simon+Mille%2C+and+David+M+Howcroft%2E+2020%2E+Disentangling+the+properties+of+human+evaluation+methods%3A+A+classification+system+to+support+comparability%2C+meta%2Devaluation+and+reproducibility+test) 4. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022. [Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust](https://md.research.elamur.ai/free/en/papers/freitag-2022-results-wmt22-metrics-shared-task/). In Proceedings of the Seventh Conferen… *Updated 1 November 2026*