---
title: "LLM Comparative Assessment — paper by Liusie et al. (2024)"
description: "The authors also find that the LLMs often favour one answer position, and they propose a reweighting… 36 references, cited by 66 papers here."
authors: ["Adian Liusie", "Potsawee Manakul", "Mark Gales"]
year: 2024
venue: "Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.eacl-long.8/"
doi: "10.18653/v1/2024.eacl-long.8"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liusie-2024-llm-comparative-assessment-zero-shot/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liusie-2024-llm-comparative-assessment-zero-shot/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models

# LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models

**Authors:** [Adian Liusie](https://md.research.elamur.ai/free/en/authors/adian-liusie/), Potsawee Manakul and [Mark Gales](https://md.research.elamur.ai/free/en/authors/mark-gales/) · **Year:** 2024 · **Venue:** [Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2024](https://md.research.elamur.ai/free/en/venues/eacl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.eacl-long.8/) · [PDF](https://aclanthology.org/2024.eacl-long.8.pdf) · [DOI 10.18653/v1/2024.eacl-long.8](https://doi.org/10.18653/v1/2024.eacl-long.8) · [OpenAlex](https://openalex.org/W4411630288)

**Citations:** 20 citations (OpenAlex, 2026-10-14) · [66](https://md.research.elamur.ai/free/en/papers/liusie-2024-llm-comparative-assessment-zero-shot/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Judging the quality of generated text usually depends on costly human annotation or on metrics built for a single task and attribute. Earlier zero-shot methods ask an LLM for an absolute score, but these tend to work well only with very large models of 175B+ parameters. Liusie, Manakul and Gales instead have an LLM decide which of two candidate outputs is better on a given attribute. The decisions come either from label-word probabilities (a prompt-based classifier) or from sampled text answers. Each candidate is then ranked by its win ratio across comparisons. Tested with moderate open-source models (FlanT5, Llama2-chat) on SummEval and TopicalChat, pairwise judging clearly beats absolute prompt scoring. It sometimes comes close to, or matches, state-of-the-art evaluators. The authors also find that the LLMs often favour one answer position, and they propose a reweighting step that corrects this bias and raises accuracy. The approach matters because it is general and needs little prompt engineering. It also lets mid-sized local models act as evaluators for any attribute.

## Key points
- Models tested: FlanT5-3B, FlanT5-11B, Llama2-chat-7B and Llama2-chat-13B (up to 13B parameters). The prompts are minimal templates in which only the attribute word changes, such as coherent, consistent or engaging.
- On SummEval (16 summaries per context, 240 possible ordered pairs), pairwise accuracy falls roughly between 60% and 80%. Debiasing raises it, for example FlanT5-11B consistency goes from 70.3 to 76.7.
- Positional bias is strong for some prompt and model combinations. Correcting it gives state-of-the-art results for FlanT5-11B on TopicalChat.
- To avoid all N·(N−1) comparisons, the paper tests three subset strategies: random, no-repeat and symmetric. Symmetric selection loses only about 1 Spearman point compared with no-repeat. Debiasing helps most when few comparisons are used.
- The main limitation is the quadratic number of comparisons. The experiments cover at most 16 candidates and only open-source LLMs.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Cambridge](https://md.research.elamur.ai/free/en/universities/university-of-cambridge/) — University, Cambridge, United Kingdom · 155 works · 115,562 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [✍️ Text generation](https://md.research.elamur.ai/free/en/topics/text-generation/) · 145 works

## Cite

Cite as: Adian Liusie, Potsawee Manakul and Mark Gales. 2024. LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{liusie2024llm,
  title = {{LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models}},
  author = {Adian Liusie, Potsawee Manakul and Mark Gales},
  booktitle = {Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.eacl-long.8},
  url = {https://doi.org/10.18653/v1/2024.eacl-long.8}
}
```

- [← previous Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment](https://md.research.elamur.ai/free/en/papers/lu-2024-large-language-models-are-superpositions/)
- [next → Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models](https://md.research.elamur.ai/free/en/papers/shi-2024-math-llava-bootstrapping-mathematical-reasoning/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

36 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Satanjeev Banerjee and Alon Lavie. 2005. [METEOR: An automatic metric for MT evaluation with improved correlation with human judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/). In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Ass…
2. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). arXiv:2004.05150.
3. Anja Belz and Ehud Reiter. 2006. Comparing automatic and human evaluation of NLG systems. In 11th Conference of the European Chapter of the Association for Computational Linguistics, pages 313– 320, Trento, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Anja+Belz+and+Ehud+Reiter%2E+2006%2E+Comparing+automatic+and+human+evaluation+of+NLG+systems%2E+In+11th+Conference+of+the+European+Chapter+of+the+Association+for+Computational+Linguistics%2C+pages+313%E2%80%93+320%2C+T)
4. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
5. Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey. arXiv preprint arXiv:2006.14799. [find](https://scholar.google.com/scholar?q=Asli+Celikyilmaz%2C+Elizabeth+Clark%2C+and+Jianfeng+Gao%2E+2020%2E+Evaluation+of+text+generation%3A+A+survey%2E+arXiv+preprint+arXiv%3A2006%2E14799%2E)
6. Lucian Lita, Monica Rogati, and Alon Lavie. 2005. BLANC: Learning evaluation metrics for MT. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 740–747, Vancouver, British Columbia, Canada. Association for Computational Li… [find](https://scholar.google.com/scholar?q=Lucian+Lita%2C+Monica+Rogati%2C+and+Alon+Lavie%2E+2005%2E+BLANC%3A+Learning+evaluation+metrics+for+MT%2E+In+Proceedings+of+Human+Language+Technology+Conference+and+Conference+on+Empirical+Methods+in+Natural+Langu)
7. Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607–15631, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Cheng%2DHan+Chiang+and+Hung%2Dyi+Lee%2E+2023%2E+Can+large+language+models+be+an+alternative+to+human+evaluations%3F+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Vo)
8. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chiang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Ying+Sheng%2C+Zhanghao+Wu%2C+Hao+Zhang%2C+Lianmin+Zheng%2C+Siyuan+Zhuang%2C+Yonghao+Zhuang%2C+Joseph+E%2E+Gonzalez%2C+Ion+Stoica%2C+and+Eric+P%2E+Xing%2E+2023%2E+Vicuna%3A+An+open%2Dsource+cha)
9. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. [Scaling instruction-finetuned language models](https://md.research.elamur.ai/free/en/papers/chung-2022-scaling-instruction-finetuned-language-models/). arXiv preprint arXiv:2210.11416.
10. Ondřej Dušek and Zdeněk Kasner. 2020. Evaluating semantic accuracy of data-to-text generation with natural language inference. In Proceedings of the 13th International Conference on Natural Language Generation, pages 131–137, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Du%C5%A1ek+and+Zdene%CC%8Ck+Kasner%2E+2020%2E+Evaluating+semantic+accuracy+of+data%2Dto%2Dtext+generation+with+natural+language+inference%2E+In+Proceedings+of+the+13th+International+Conference+on+Natural+Language)
11. Alexander R Fabbri, Wojciech Kryściński, Bryan Mc-Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. [find](https://scholar.google.com/scholar?q=Alexander+R+Fabbri%2C+Wojciech+Krys%CC%81cin%CC%81ski%2C+Bryan+Mc%2DCann%2C+Caiming+Xiong%2C+Richard+Socher%2C+and+Dragomir+Radev%2E+2021%2E+Summeval%3A+Re%2Devaluating+summarization+evaluation%2E+Transactions+of+the+Association+for)
12. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166. [find](https://scholar.google.com/scholar?q=Jinlan+Fu%2C+See%2DKiong+Ng%2C+Zhengbao+Jiang%2C+and+Pengfei+Liu%2E+2023%2E+Gptscore%3A+Evaluate+as+you+desire%2E+arXiv+preprint+arXiv%3A2302%2E04166%2E)
13. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Yang+Liu%2C+Dan+Iter%2C+Yichong+Xu%2C+Shuohang+Wang%2C+Ruochen+Xu%2C+and+Chenguang+Zhu%2E+2023%2E+G%2Deval%3A+NLG+evaluation+using+gpt%2D4+with+better+human+alignment%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+M)
14. Adian Liusie, Potsawee Manakul, and Mark J. F. Gales. 2023. Mitigating word bias in zero-shot prompt-based classifiers. [find](https://scholar.google.com/scholar?q=Adian+Liusie%2C+Potsawee+Manakul%2C+and+Mark+J%2E+F%2E+Gales%2E+2023%2E+Mitigating+word+bias+in+zero%2Dshot+prompt%2Dbased+classifiers%2E)
15. Potsawee Manakul and Mark JF Gales. 2022. Podcast summary assessment: A resource for evaluating summary assessment methods. arXiv preprint arXiv:2208.13265. — [arXiv:2208.13265](https://arxiv.org/abs/2208.13265) [find](https://scholar.google.com/scholar?q=Potsawee+Manakul+and+Mark+JF+Gales%2E+2022%2E+Podcast+summary+assessment%3A+A+resource+for+evaluating+summary+assessment+methods%2E+arXiv+preprint+arXiv%3A2208%2E13265%2E+%E2%80%94+%5BarXiv%3A2208%2E13265%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
16. Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023a. Mqag: Multiple-choice question answering and generation for assessing information consistency in summarization. arXiv preprint arXiv:2301.12307. [find](https://scholar.google.com/scholar?q=Potsawee+Manakul%2C+Adian+Liusie%2C+and+Mark+JF+Gales%2E+2023a%2E+Mqag%3A+Multiple%2Dchoice+question+answering+and+generation+for+assessing+information+consistency+in+summarization%2E+arXiv+preprint+arXiv%3A2301%2E1230)
17. Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023b. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896. [find](https://scholar.google.com/scholar?q=Potsawee+Manakul%2C+Adian+Liusie%2C+and+Mark+JF+Gales%2E+2023b%2E+Selfcheckgpt%3A+Zero%2Dresource+black%2Dbox+hallucination+detection+for+generative+large+language+models%2E+arXiv+preprint+arXiv%3A2303%2E08896%2E)
18. Shikib Mehri and Maxine Eskenazi. 2020a. [Unsupervised evaluation of interactive dialog with DialoGPT](https://md.research.elamur.ai/free/en/papers/mehri-2020-unsupervised-evaluation-interactive-dialog-dialogpt/). In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 225–235, 1st virtual meeting. Association for Computational Linguistics.
19. Shikib Mehri and Maxine Eskenazi. 2020b. [USR: An unsupervised and reference free evaluation metric for dialog generation](https://md.research.elamur.ai/free/en/papers/mehri-2020-usr-unsupervised-reference-free-evaluation/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681–707, Online. Association for Computational Linguistics.
20. Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. [Creating training corpora for NLG micro-planners](https://md.research.elamur.ai/free/en/papers/gardent-2017-creating-training-corpora-nlg-micro/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 179–188, Vancouver, Canada. Association…
21. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. arXiv preprint arXiv:2302.14520. [find](https://scholar.google.com/scholar?q=Tom+Kocmi+and+Christian+Federmann%2E+2023%2E+Large+language+models+are+state%2Dof%2Dthe%2Dart+evaluators+of+translation+quality%2E+arXiv+preprint+arXiv%3A2302%2E14520%2E)
22. Alice Lai and Joel Tetreault. 2018. [Discourse coherence in the wild: A dataset, evaluation and methods](https://md.research.elamur.ai/free/en/papers/lai-2018-discourse-coherence-wild-dataset-evaluation/). In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pages 214–223, Melbourne, Australia. Association for Computational Linguistics.
23. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
24. Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, et al. 2023. [Large language models are effective text rankers with pairwise ranking prompting](https://md.research.elamur.ai/free/en/papers/qin-2024-large-language-models-are-effective/). arXiv preprint arXiv:2306.17563.
25. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). The Journal of Machine Learning Research, 21(1):5485–5551.
26. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. [COMET: A neural framework for MT evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics.
27. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+Xing%2C+et+al%2E+2023%2E+Judging+llm%2Das%2Da%2Djudge+with+mt%2Dbench+and+chatbot+arena%2E+a)
28. Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. [Towards a unified multi-dimensional evaluator for text generation](https://md.research.elamur.ai/free/en/papers/zhong-2022-unified-multi-dimensional-evaluator-text/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2023– 2038, Abu Dhabi,…
29. Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. [QuestEval: Summarization asks for fact-based evaluation](https://md.research.elamur.ai/free/en/papers/scialom-2021-questeval-summarization-asks-fact-based/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6594–6604, Online…
30. Louis L Thurstone. 1927. A law of comparative judgment. Psychological review, 34(4):273. [find](https://scholar.google.com/scholar?q=Louis+L+Thurstone%2E+1927%2E+A+law+of+comparative+judgment%2E+Psychological+review%2C+34%284%29%3A273%2E)
31. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023%2E+Llama+2%3A+Open+foundation+and+fi)
32. Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008–5020, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Kyunghyun+Cho%2C+and+Mike+Lewis%2E+2020%2E+Asking+and+answering+questions+to+evaluate+the+factual+consistency+of+summaries%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Comput)
33. Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. [Is chatgpt a good nlg evaluator? a preliminary study](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-nlg-evaluator-preliminary/). arXiv preprint arXiv:2303.04048.
34. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. 2022. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. arXiv preprint arXiv:2204.… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Swaroop+Mishra%2C+Pegah+Alipoormolabashi%2C+Yeganeh+Kordi%2C+Amirreza+Mirzaei%2C+Anjana+Arunkumar%2C+Arjun+Ashok%2C+Arut+Selvan+Dhanasekaran%2C+Atharva+Naik%2C+David+Stap%2C+et+al%2E+2022%2E+Super%2Dnaturalinst)
35. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). arXiv preprint arXiv:2206.07682.
36. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. *Figure 5: Assessment Performance when only a subset of comparisons are considered (extending the results of Fi… [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+2019%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+In+International+Conference+on+Learning+Representations%2E+%2AFigure+5%3A+)

*Updated 24 October 2026*