---
title: "Position — paper by Zaghouani (2026) — summary and ideas"
description: "Drawing on psychometrics and the construct validity tradition of Cronbach and Meehl (1955), the author describes evaluation… 15 references."
authors: ["Wajdi Zaghouani"]
year: 2026
venue: "Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)"
source: "https://aclanthology.org/2026.gem-main.79/"
doi: "10.18653/v1/2026.gem-main.79"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zaghouani-2026-position-what-are-we-measuring/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zaghouani-2026-position-what-are-we-measuring/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/45/) / Position: What Are We Measuring? Rethinking Evaluation in Natural Language Generation

# Position: What Are We Measuring? Rethinking Evaluation in Natural Language Generation

**Authors:** [Wajdi Zaghouani](https://md.research.elamur.ai/free/en/authors/wajdi-zaghouani/) · **Year:** 2026 · **Venue:** [Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)](https://md.research.elamur.ai/free/en/venues/gem/) ([GEM 2026](https://md.research.elamur.ai/free/en/venues/gem/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.gem-main.79/) · [PDF](https://aclanthology.org/2026.gem-main.79.pdf) · [DOI 10.18653/v1/2026.gem-main.79](https://doi.org/10.18653/v1/2026.gem-main.79) · [OpenAlex](https://openalex.org/W7166855080)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
This position paper argues that natural language generation evaluation lacks a theory of what its many metrics capture, rather than lacking metrics. Drawing on psychometrics and the construct validity tradition of Cronbach and Meehl (1955), the author describes evaluation as a chain: a latent construct, its operationalization, a metric, and an observed score. Current practice merges these levels. The paper examines four paradigms: reference-based metrics (BLEU, ROUGE), embedding-based metrics (BERTScore), LLM-as-judge, and human evaluation. Each relies on implicit, unjustified assumptions about "quality" and falls short on content, convergent, discriminant, and consequential validity. The work offers no new dataset or metric. Instead, it calls for three changes:
- explicit definitions of constructs such as fluency, adequacy, and faithfulness,
- validation with established measurement concepts,
- openness about the theoretical commitments behind evaluation choices.

It also sketches a preliminary six-construct taxonomy. The framing matters because it explains recurring problems such as leaderboard chasing, poor reproducibility of NLG evaluations, and weak attention to real-world impact.

## Key points
- The paper analyzes four paradigms: reference-based, embedding-based, LLM-as-judge, and human evaluation. It argues that each blurs construct definition with operationalization.
- Its example of a discriminant validity failure is BERTScore applied to summarization faithfulness. The metric rewards closeness to a reference even when that reference contains claims the source does not support.
- Prior evidence it cites includes Reiter (2018), which reviewed 284 correlations from 34 papers and advised against using BLEU as primary evidence. Braggaar et al. (2026) reviewed 122 task-oriented dialogue evaluation studies.
- The paper shows that explicit construct specification is feasible by pointing to existing work:
  - MQM (Lommel et al., 2014)
  - the QCET taxonomy (Belz et al., 2025), with 114 criteria derived from 933 experiments
  - the ASA questionnaire, with 19 constructs
- Proposed future work includes multi-trait multi-method validation, construct-grounded benchmark design, and reproducibility audits aligned with ReproNLP.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Northwestern University in Qatar](https://md.research.elamur.ai/free/en/universities/northwestern-university-in-qatar/) — University, Doha, Qatar · 39 works · 148 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [✍️ Text generation](https://md.research.elamur.ai/free/en/topics/text-generation/) · 214 works

## Cite

Cite as: Wajdi Zaghouani. 2026. Position: What Are We Measuring? Rethinking Evaluation in Natural Language Generation. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zaghouani2026position,
  title = {{Position: What Are We Measuring? Rethinking Evaluation in Natural Language Generation}},
  author = {Wajdi Zaghouani},
  booktitle = {Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)},
  year = {2026},
  doi = {10.18653/v1/2026.gem-main.79},
  url = {https://doi.org/10.18653/v1/2026.gem-main.79}
}
```

- [← previous Position: Scores Without Context? Rethinking the Role of Evaluation in the Era of LLMs](https://md.research.elamur.ai/free/en/papers/zhou-2026-position-scores-without-context-rethinking/)
- [next → Post Hoc Agentic Refinement for Improving Precision in Multilingual Clinical Text De-identification](https://md.research.elamur.ai/free/en/papers/xu-2026-post-hoc-agentic-refinement-improving/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/45/)

## References

15 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anya Belz. 2022. A metrological perspective on reproducibility in NLP. Computational Linguistics, 48(4):1125–1135. https://doi.org/10.1162/ coli_a_00448 [find](https://scholar.google.com/scholar?q=Anya+Belz%2E+2022%2E+A+metrological+perspective+on+reproducibility+in+NLP%2E+Computational+Linguistics%2C+48%284%29%3A1125%E2%80%931135%2E+https%3A%2F%2Fdoi%2Eorg%2F10%2E1162%2F+coli%5Fa%5F00448)
2. Anya Belz, Simon Mille, and Craig Thomson. 2025. Standard quality criteria derived from current NLP evaluations for guiding evaluation design and grounding comparability and AI compliance assessments. In Findings of the Association for Computational Linguistics: ACL 2025, pages 26685– 26715, Vienna,… [find](https://scholar.google.com/scholar?q=Anya+Belz%2C+Simon+Mille%2C+and+Craig+Thomson%2E+2025%2E+Standard+quality+criteria+derived+from+current+NLP+evaluations+for+guiding+evaluation+design+and+grounding+comparability+and+AI+compliance+assessments%2E)
3. Lee J. Cronbach and Paul E. Meehl. 1955. Construct validity in psychological tests. Psychological Bulletin, 52(4):281–302. https://doi.org/10.1037/ h0040957 [find](https://scholar.google.com/scholar?q=Lee+J%2E+Cronbach+and+Paul+E%2E+Meehl%2E+1955%2E+Construct+validity+in+psychological+tests%2E+Psychological+Bulletin%2C+52%284%29%3A281%E2%80%93302%2E+https%3A%2F%2Fdoi%2Eorg%2F10%2E1037%2F+h0040957)
4. Siska Fitrianie, Merijn Bruijnes, Deborah Richards, Andrea Bönsch, and Willem-Paul Brinkman. 2020. The 19 unifying questionnaire constructs of artificial social agents: An IVA community analysis. In Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents (IVA ’20), Article… — [doi:10.1145/3383652](https://doi.org/10.1145/3383652) [find](https://scholar.google.com/scholar?q=Siska+Fitrianie%2C+Merijn+Bruijnes%2C+Deborah+Richards%2C+Andrea+B%C3%B6nsch%2C+and+Willem%2DPaul+Brinkman%2E+2020%2E+The+19+unifying+questionnaire+constructs+of+artificial+social+agents%3A+An+IVA+community+analysis%2E+In+P)
5. Mingqi Gao, Xinyu Hu, Xunjian Yin, Jie Ruan, Xiao Pu, and Xiaojun Wan. 2025. [LLM-based NLG evaluation: Current status and challenges](https://md.research.elamur.ai/free/en/papers/gao-2025-llm-based-nlg-evaluation-current/). Computational Linguistics, 51(2):661–687. https://doi.org/10. 1162/coli_a_00561
6. Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondrej Dusek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori H… — [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://md.research.elamur.ai/free/en/papers/gehrmann-2021-gem-benchmark-natural-language-generation/)
7. David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions. In Proceedings… [find](https://scholar.google.com/scholar?q=David+M%2E+Howcroft%2C+Anya+Belz%2C+Miruna%2DAdriana+Clinciu%2C+Dimitra+Gkatzia%2C+Sadid+A%2E+Hasan%2C+Saad+Mahamood%2C+Simon+Mille%2C+Emiel+van+Miltenburg%2C+Sashank+Santhanam%2C+and+Verena+Rieser%2E+2020%2E+Twenty+years+of+con)
8. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric… — [ROUGE: A Package for Automatic Evaluation of Summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/)
9. Jekaterina Novikova, Ondrej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. [Why we need new evaluation metrics for NLG](https://md.research.elamur.ai/free/en/papers/novikova-2017-why-we-need-new-evaluation/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241–2252, Copenhagen, Denmark. Association for Computational Linguistics. htt…
10. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: A method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
11. David Schlangen. 2021. Targeting the benchmark: On methodology in current natural language processing research. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Paper… [find](https://scholar.google.com/scholar?q=David+Schlangen%2E+2021%2E+Targeting+the+benchmark%3A+On+methodology+in+current+natural+language+processing+research%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguisti)
12. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. [BERTscore: Evaluating text generation with BERT](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Proceedings of the 8th International Conference on Learning Representations (ICLR 2020). [https://openreview.net/forum?id=SkeHuCVFDr](https://openreview.net/forum?id=SkeHuCVFDr) Lianmin Zheng, Wei-Lin Chiang,… [↗](https://openreview.net/forum?id=SkeHuCVFDr)
13. Yan Zhuang, Qi Liu, Zachary A. Pardos, Patrick C. Kyllonen, Jiyun Zu, Zhenya Huang, Shijin Wang, and Enhong Chen. 2025. Position: AI evaluation should learn from how we test humans. In Proceedings of the 42nd International Conference on Machine Learning (ICML 2025). https://arxiv.org/abs/ 2306.10512 [↗](https://arxiv.org/abs/)
14. Cécile Paris, Nathalie Colineau, and Ross Wilkinson. 2006. Evaluations of NLG systems: Common corpus and tasks or common dimensions and metrics? In Proceedings of the Fourth International Natural Language Generation Conference, pages 127–129, Sydney, Australia. Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=C%C3%A9cile+Paris%2C+Nathalie+Colineau%2C+and+Ross+Wilkinson%2E+2006%2E+Evaluations+of+NLG+systems%3A+Common+corpus+and+tasks+or+common+dimensions+and+metrics%3F+In+Proceedings+of+the+Fourth+International+Natural+Lang)
15. Ehud Reiter. 2018. [A structured review of the validity of BLEU](https://md.research.elamur.ai/free/en/papers/reiter-2018-structured-review-validity-bleu/). Computational Linguistics, 44(3):393– 401. [https://direct.mit.edu/coli/article/](https://direct.mit.edu/coli/article/) 44/3/393/1598/ [↗](https://direct.mit.edu/coli/article/)

*Updated 2 November 2026*