---
title: "Leveraging Large Language Models for NLG Evaluation:"
description: "Automatic metrics for natural language generation have serious limits. 132 references, cited by 40 papers here."
authors: ["Zhen Li", "Xiaohan Xu", "Tao Shen", "Can Xu", "Jia-Chen Gu", "Yuxuan Lai", "Chongyang Tao", "Shuai Ma"]
year: 2024
venue: "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2024.emnlp-main.896/"
doi: "10.18653/v1/2024.emnlp-main.896"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2024-leveraging-large-language-models-nlg/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2024-leveraging-large-language-models-nlg/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / Leveraging Large Language Models for NLG Evaluation: Advances and Challenges

# Leveraging Large Language Models for NLG Evaluation: Advances and Challenges

**Authors:** [Zhen Li](https://md.research.elamur.ai/free/en/authors/zhen-li/), Xiaohan Xu, [Tao Shen](https://md.research.elamur.ai/free/en/authors/tao-shen/), [Can Xu](https://md.research.elamur.ai/free/en/authors/can-xu/), [Jia-Chen Gu](https://md.research.elamur.ai/free/en/authors/jia-chen-gu/), Yuxuan Lai, [Chongyang Tao](https://md.research.elamur.ai/free/en/authors/chongyang-tao/) and Shuai Ma · **Year:** 2024 · **Venue:** [Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2024](https://md.research.elamur.ai/free/en/venues/emnlp/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.emnlp-main.896/) · [PDF](https://aclanthology.org/2024.emnlp-main.896.pdf) · [DOI 10.18653/v1/2024.emnlp-main.896](https://doi.org/10.18653/v1/2024.emnlp-main.896) · [OpenAlex](https://openalex.org/W4404782301)

**Citations:** 23 citations (OpenAlex, 2026-10-14) · [40](https://md.research.elamur.ai/free/en/papers/li-2024-leveraging-large-language-models-nlg/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

Automatic metrics for natural language generation have serious limits. N-gram overlap scores such as BLEU and ROUGE mostly capture surface differences. Embedding-based metrics often agree poorly with human ratings and explain little about their scores. This EMNLP 2024 survey reviews the fast-growing use of LLMs as judges of generated text.

The authors model evaluation as a function of three inputs: the candidate output, an optional source and optional references. They then classify prior work along three axes: the NLG task, whether references are used, and how the LLM is applied (matching-based or generative). The survey concentrates on generative evaluators. It separates prompt-based methods, which use off-the-shelf models, from tuning-based methods, which fine-tune open models to act as judges. Each group is further divided by scoring protocol. The paper also catalogues meta-evaluation benchmarks for each task and compares LLM judges with traditional metrics.

It closes with open problems: bias, robustness, circular self-evaluation, domain-specific evaluation and unified evaluation. This gives researchers a structured map of the field and its weak spots.

## Key points

- **Framework and taxonomy.** Evaluation is formalized as E = f_t(h, s, r): hypothesis h, source s, reference r, task t. Work is sorted by task (MT, TS, DG, SG, IC, D2T, general generation), by reference-based vs. reference-free setting, and by matching-based vs. generative function.
- **Scoring protocols.** Generative evaluators fall into six protocols: score-based, probability-based, Likert-style, pairwise, ensemble and advanced.
  - Prompt-based examples: GEMBA, GPTSCORE, G-Eval, ChatEval, FACTSCORE.
  - Tuning-based examples: PandaLM, AUTO-J, Prometheus/PROMETHEUS-2, INSTRUCTSCORE, TIGERScore.
- **Meta-evaluation benchmarks.** The survey lists benchmarks for each task, including MQM, WMT Metrics, SummEval, FRANK, Topical-Chat, WebNLG, OpenMEVA, MT-bench and AlpacaEval.
- **Evaluator biases.** Beyond social stereotypes, LLM judges show three biases specific to their role:
  - order bias (favoring options by their position),
  - egocentric bias (favoring their own outputs),
  - length bias (favoring longer or shorter texts).
- **Robustness and future directions.** LLM judges are sensitive to prompt wording and to adversarial inputs. The authors call for domain-aware evaluators (e.g., law, medicine, finance) and unified evaluators. They cite MT-Bench and Auto-J (validated on 58 scenarios) as early steps toward unified evaluation.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 244 works · 39,359 citations
- [University of California, Los Angeles](https://md.research.elamur.ai/free/en/universities/university-of-california-los-angeles/) — University, Los Angeles, United States · 87 works · 54,961 citations
- [University of Hong Kong](https://md.research.elamur.ai/free/en/universities/university-of-hong-kong/) — University, Pok Fu Lam, China · 89 works · 11,774 citations
- [Beihang University](https://md.research.elamur.ai/free/en/universities/beihang-university/) — University, Beijing, China · 65 works · 8,452 citations
- [University of Technology Sydney](https://md.research.elamur.ai/free/en/universities/university-of-technology-sydney/) — University, Sydney, Australia · 18 works · 12,833 citations

Also on the byline: [Open University of China](https://ror.org/02maxav80).

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [✍️ Text generation](https://md.research.elamur.ai/free/en/topics/text-generation/) · 202 works

## Cite

Cite as: Zhen Li, Xiaohan Xu, Tao Shen, Can Xu, Jia-Chen Gu, Yuxuan Lai, Chongyang Tao and Shuai Ma. 2024. Leveraging Large Language Models for NLG Evaluation: Advances and Challenges. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{li2024leveraging,
  title = {{Leveraging Large Language Models for NLG Evaluation: Advances and Challenges}},
  author = {Zhen Li, Xiaohan Xu, Tao Shen, Can Xu, Jia-Chen Gu, Yuxuan Lai, Chongyang Tao and Shuai Ma},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.896},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.896}
}
```

- [← previous Can Language Models Handle Recursively Nested Grammatical Structures? A Case Study on Comparing Models and Humans](https://md.research.elamur.ai/free/en/papers/lampinen-2024-can-language-models-handle-recursively/)
- [next → Revolutionizing Finance with LLMs: An Overview of Applications and Insights](https://md.research.elamur.ai/free/en/papers/zhao-2024-revolutionizing-finance-llms-overview-applications/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

132 references, 66 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Somak Aditya, Yezhou Yang, Chitta Baral, Cornelia Fermuller, and Yiannis Aloimonos. 2015. From images to sentences through scene description graphs using commonsense reasoning and knowledge. arXiv preprint arXiv:1511.03292. — [arXiv:1511.03292](https://arxiv.org/abs/1511.03292) [find](https://scholar.google.com/scholar?q=Somak+Aditya%2C+Yezhou+Yang%2C+Chitta+Baral%2C+Cornelia+Fermuller%2C+and+Yiannis+Aloimonos%2E+2015%2E+From+images+to+sentences+through+scene+description+graphs+using+commonsense+reasoning+and+knowledge%2E+arXiv+pre)
2. Hong Chen, Duc Vo, Hiroya Takamura, Yusuke Miyao, and Hideki Nakayama. 2022. StoryER: Automatic story evaluation via ranking, rating and reasoning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1739–1753, Abu Dhabi, United Arab Emirates. Association… [find](https://scholar.google.com/scholar?q=Hong+Chen%2C+Duc+Vo%2C+Hiroya+Takamura%2C+Yusuke+Miyao%2C+and+Hideki+Nakayama%2E+2022%2E+StoryER%3A+Automatic+story+evaluation+via+ranking%2C+rating+and+reasoning%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+M)
3. Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. [Towards a human-like open-domain chatbot](https://md.research.elamur.ai/free/en/papers/adiwardana-2020-towards-human-like-open-domain/). arXiv preprint arXiv:2001.09977.
4. Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu. 2023. Exploring the use of large language models for reference-free text quality evaluation: A preliminary empirical study. arXiv preprint arXiv:2304.00723. — [arXiv:2304.00723](https://arxiv.org/abs/2304.00723) [find](https://scholar.google.com/scholar?q=Yi+Chen%2C+Rui+Wang%2C+Haiyun+Jiang%2C+Shuming+Shi%2C+and+Ruifeng+Xu%2E+2023%2E+Exploring+the+use+of+large+language+models+for+reference%2Dfree+text+quality+evaluation%3A+A+preliminary+empirical+study%2E+arXiv+preprint)
5. Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al. 2023. Benchmarking foundation models with language-model-as-an-examiner. arXiv preprint arXiv:2306.04181. [find](https://scholar.google.com/scholar?q=Yushi+Bai%2C+Jiahao+Ying%2C+Yixin+Cao%2C+Xin+Lv%2C+Yuze+He%2C+Xiaozhi+Wang%2C+Jifan+Yu%2C+Kaisheng+Zeng%2C+Yijia+Xiao%2C+Haozhe+Lyu%2C+et+al%2E+2023%2E+Benchmarking+foundation+models+with+language%2Dmodel%2Das%2Dan%2Dexaminer%2E+arXiv)
6. Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607–15631, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Cheng%2DHan+Chiang+and+Hung%2Dyi+Lee%2E+2023%2E+Can+large+language+models+be+an+alternative+to+human+evaluations%3F+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Vo)
7. Ankur Bapna, Naveen Arivazhagan, and Orhan Firat. 2019. [Simple, scalable adaptation for neural machine translation](https://md.research.elamur.ai/free/en/papers/bapna-2019-simple-scalable-adaptation-neural-machine/). arXiv preprint arXiv:1909.08478.
8. Loïc Barrault, Magdalena Biesialska, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljubešić, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Sant… — [Findings of the 2020 Conference on Machine Translation (WMT20)](https://md.research.elamur.ai/free/en/papers/barrault-2020-findings-2020-conference-machine-translation/)
9. Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. [Re-evaluating evaluation in text summarization](https://md.research.elamur.ai/free/en/papers/bhandari-2020-re-evaluating-evaluation-text-summarization/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9347–9359, Online. Association for Computational Linguist…
10. Thiago Castro Ferreira, Claire Gardent, Nikolai Ilinykh, Chris van der Lee, Simon Mille, Diego Moussallem, and Anastasia Shimorina. 2020. [find](https://scholar.google.com/scholar?q=Thiago+Castro+Ferreira%2C+Claire+Gardent%2C+Nikolai+Ilinykh%2C+Chris+van+der+Lee%2C+Simon+Mille%2C+Diego+Moussallem%2C+and+Anastasia+Shimorina%2E+2020%2E)
11. [The 2020 bilingual, bi-directional WebNLG+ shared task: Overview and evaluation results (WebNLG+ 2020)](https://md.research.elamur.ai/free/en/papers/ferreira-2020-2020-bilingual-bi-directional-webnlg/). In Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+), pages 55–76, Dublin, Ireland (Virtual). Association for Computational Linguistics.
12. Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey. arXiv preprint arXiv:2006.14799. [find](https://scholar.google.com/scholar?q=Asli+Celikyilmaz%2C+Elizabeth+Clark%2C+and+Jianfeng+Gao%2E+2020%2E+Evaluation+of+text+generation%3A+A+survey%2E+arXiv+preprint+arXiv%3A2006%2E14799%2E)
13. Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. [Chateval: Towards better llm-based evaluators through multi-agent debate](https://md.research.elamur.ai/free/en/papers/chan-2023-chateval-better-llm-based/). arXiv preprint arXiv:2308.07201.
14. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Yunxuan+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+et+al%2E+2022%2E+Scaling+instruction%2Dfinetuned+language+models%2E+a)
15. Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. 2023a. [Chatlaw: Open-source legal large language model with integrated external knowledge bases](https://md.research.elamur.ai/free/en/papers/cui-2023-chatlaw-multi-agent-legal-assistant/). arXiv preprint arXiv:2306.16092.
16. Junyun Cui, Xiaoyu Shen, and Shaochun Wen. 2023b. A survey on legal judgment prediction: Datasets, metrics, models and challenges. IEEE Access. [find](https://scholar.google.com/scholar?q=Junyun+Cui%2C+Xiaoyu+Shen%2C+and+Shaochun+Wen%2E+2023b%2E+A+survey+on+legal+judgment+prediction%3A+Datasets%2C+metrics%2C+models+and+challenges%2E+IEEE+Access%2E)
17. Yin Cui, Guandao Yang, Andreas Veit, Xun Huang, and Serge Belongie. 2018. Learning to evaluate image captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5804–5812. [find](https://scholar.google.com/scholar?q=Yin+Cui%2C+Guandao+Yang%2C+Andreas+Veit%2C+Xun+Huang%2C+and+Serge+Belongie%2E+2018%2E+Learning+to+evaluate+image+captioning%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pages)
18. Sören Dreano, Derek Molloy, and Noel Murphy. 2023. Embed_Llama: Using LLM embeddings for the metrics shared task. In Proceedings of the Eighth Conference on Machine Translation, pages 738–745, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=S%C3%B6ren+Dreano%2C+Derek+Molloy%2C+and+Noel+Murphy%2E+2023%2E+Embed%5FLlama%3A+Using+LLM+embeddings+for+the+metrics+shared+task%2E+In+Proceedings+of+the+Eighth+Conference+on+Machine+Translation%2C+pages+738%E2%80%93745%2C+Singapo)
19. Alexander R. Fabbri, Wojciech Kryściński, Bryan Mc-Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021a. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. [find](https://scholar.google.com/scholar?q=Alexander+R%2E+Fabbri%2C+Wojciech+Krys%CC%81cin%CC%81ski%2C+Bryan+Mc%2DCann%2C+Caiming+Xiong%2C+Richard+Socher%2C+and+Dragomir+Radev%2E+2021a%2E+SummEval%3A+Re%2Devaluating+summarization+evaluation%2E+Transactions+of+the+Association+f)
20. Alexander R. Fabbri, Wojciech Kryściński, Bryan Mc-Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021b. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. [find](https://scholar.google.com/scholar?q=Alexander+R%2E+Fabbri%2C+Wojciech+Krys%CC%81cin%CC%81ski%2C+Bryan+Mc%2DCann%2C+Caiming+Xiong%2C+Richard+Socher%2C+and+Dragomir+Radev%2E+2021b%2E+SummEval%3A+Re%2Devaluating+summarization+evaluation%2E+Transactions+of+the+Association+f)
21. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. [Hierarchical neural story generation](https://md.research.elamur.ai/free/en/papers/fan-2018-hierarchical-neural-story-generation/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898, Melbourne, Australia. Association for Computational Linguistics.
22. Akhbardeh Farhad, Arkhangorodsky Arkady, Biesialska Magdalena, Bojar Ondřej, Chatterjee Rajen, Chaudhary Vishrav, Marta R Costa-jussa, España-Bonet Cristina, Fan Angela, Federmann Christian, et al. 2021. [Findings of the 2021 conference on machine translation (wmt21)](https://md.research.elamur.ai/free/en/papers/akhbardeh-2021-findings-2021-conference-machine-translation/). In Proceedings of the Sixth Con…
23. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021a. [Experts, errors, and context: A large-scale study of human evaluation for machine translation](https://md.research.elamur.ai/free/en/papers/freitag-2021-experts-errors-context-large-scale/). Transactions of the Association for Computational Linguistics, 9:1460–1474.
24. Markus Freitag, David Grangier, and Isaac Caswell. 2020. [BLEU might be guilty but references are not innocent](https://md.research.elamur.ai/free/en/papers/freitag-2020-bleu-might-be-guilty-but/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 61–71, Online. Association for Computational Linguistics.
25. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022. [Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust](https://md.research.elamur.ai/free/en/papers/freitag-2022-results-wmt22-metrics-shared-task/). In Proceedings of the Seventh Conferen…
26. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021b. [Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain](https://md.research.elamur.ai/free/en/papers/freitag-2021-results-wmt21-metrics-shared-task/). In Proceedings of the Sixth Conference on Machine Trans…
27. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166. [find](https://scholar.google.com/scholar?q=Jinlan+Fu%2C+See%2DKiong+Ng%2C+Zhengbao+Jiang%2C+and+Pengfei+Liu%2E+2023%2E+Gptscore%3A+Evaluate+as+you+desire%2E+arXiv+preprint+arXiv%3A2302%2E04166%2E)
28. Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. 2023. Human-like summarization evaluation with chatgpt. arXiv preprint arXiv:2304.02554. [find](https://scholar.google.com/scholar?q=Mingqi+Gao%2C+Jie+Ruan%2C+Renliang+Sun%2C+Xunjian+Yin%2C+Shiping+Yang%2C+and+Xiaojun+Wan%2E+2023%2E+Human%2Dlike+summarization+evaluation+with+chatgpt%2E+arXiv+preprint+arXiv%3A2304%2E02554%2E)
29. Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elkind, and Idan Szpektor. 2023. [Trueteacher: Learning factual consistency evaluation with large language models](https://md.research.elamur.ai/free/en/papers/gekhman-2023-trueteacher-learning-factual-consistency-evaluation/). arXiv preprint arXiv:2305.11171.
30. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. [Chatgpt outperforms crowd-workers for text-annotation tasks](https://md.research.elamur.ai/free/en/papers/gilardi-2023-chatgpt-outperforms-crowd-workers-text/). arXiv preprint arXiv:2303.15056.
31. Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. [Samsum corpus: A human-annotated dialogue dataset for abstractive summarization](https://md.research.elamur.ai/free/en/papers/gliwa-2019-samsum-corpus-human-annotated-dialogue/). arXiv preprint arXiv:1911.12237.
32. Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Z. Hakkani-Tür. 2019. Topical-chat: Towards knowledge-grounded open-domain conversations. ArXiv, abs/2308.11995. [find](https://scholar.google.com/scholar?q=Karthik+Gopalakrishnan%2C+Behnam+Hedayatnia%2C+Qinlang+Chen%2C+Anna+Gottardi%2C+Sanjeev+Kwatra%2C+Anu+Venkatesh%2C+Raefer+Gabriel%2C+and+Dilek+Z%2E+Hakkani%2DT%C3%BCr%2E+2019%2E+Topical%2Dchat%3A+Towards+knowledge%2Dgrounded+open%2Ddom)
33. Rupali Goyal, Parteek Kumar, and VP Singh. 2023. A systematic survey on automated text generation tools and techniques: application, evaluation, and challenges. Multimedia Tools and Applications, pages 1–56. [find](https://scholar.google.com/scholar?q=Rupali+Goyal%2C+Parteek+Kumar%2C+and+VP+Singh%2E+2023%2E+A+systematic+survey+on+automated+text+generation+tools+and+techniques%3A+application%2C+evaluation%2C+and+challenges%2E+Multimedia+Tools+and+Applications%2C+page)
34. Max Grusky, Mor Naaman, and Yoav Artzi. 2018. [Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies](https://md.research.elamur.ai/free/en/papers/grusky-2018-newsroom-dataset-1-3-million/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pa…
35. Jian Guan, Zhexin Zhang, Zhuoer Feng, Zitao Liu, Wenbiao Ding, Xiaoxi Mao, Changjie Fan, and Minlie Huang. 2021. OpenMEVA: A benchmark for evaluating open-ended story generation metrics. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internati… [find](https://scholar.google.com/scholar?q=Jian+Guan%2C+Zhexin+Zhang%2C+Zhuoer+Feng%2C+Zitao+Liu%2C+Wenbiao+Ding%2C+Xiaoxi+Mao%2C+Changjie+Fan%2C+and+Minlie+Huang%2E+2021%2E+OpenMEVA%3A+A+benchmark+for+evaluating+open%2Dended+story+generation+metrics%2E+In+Proceeding)
36. Hosein Hasanbeig, Hiteshi Sharma, Leo Betthauser, Felipe Vieira Frujeri, and Ida Momennejad. 2023. Allure: A systematic protocol for auditing and improving llm-based evaluation of text using iterative in-context-learning. arXiv preprint arXiv:2309.13701. — [arXiv:2309.13701](https://arxiv.org/abs/2309.13701) [find](https://scholar.google.com/scholar?q=Hosein+Hasanbeig%2C+Hiteshi+Sharma%2C+Leo+Betthauser%2C+Felipe+Vieira+Frujeri%2C+and+Ida+Momennejad%2E+2023%2E+Allure%3A+A+systematic+protocol+for+auditing+and+improving+llm%2Dbased+evaluation+of+text+using+iterative)
37. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. [Teaching machines to read and comprehend](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/). Advances in neural information processing systems, 28.
38. Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Framing image description as a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research, 47:853–899. [find](https://scholar.google.com/scholar?q=Micah+Hodosh%2C+Peter+Young%2C+and+Julia+Hockenmaier%2E+2013%2E+Framing+image+description+as+a+ranking+task%3A+Data%2C+models+and+evaluation+metrics%2E+Journal+of+Artificial+Intelligence+Research%2C+47%3A853%E2%80%93899%2E)
39. Fan Huang, Haewoon Kwak, and Jisun An. 2023. Is chatgpt better than human annotators? potential and limitations of chatgpt in explaining implicit hate speech. arXiv preprint arXiv:2302.07736. [find](https://scholar.google.com/scholar?q=Fan+Huang%2C+Haewoon+Kwak%2C+and+Jisun+An%2E+2023%2E+Is+chatgpt+better+than+human+annotators%3F+potential+and+limitations+of+chatgpt+in+explaining+implicit+hate+speech%2E+arXiv+preprint+arXiv%3A2302%2E07736%2E)
40. Sameer Jain, Vaishakh Keshava, Swarnashree Mysore Sathyendra, Patrick Fernandes, Pengfei Liu, Graham Neubig, and Chunting Zhou. 2023. Multi-dimensional evaluation of text summarization with in-context learning. arXiv preprint arXiv:2306.01200. [find](https://scholar.google.com/scholar?q=Sameer+Jain%2C+Vaishakh+Keshava%2C+Swarnashree+Mysore+Sathyendra%2C+Patrick+Fernandes%2C+Pengfei+Liu%2C+Graham+Neubig%2C+and+Chunting+Zhou%2E+2023%2E+Multi%2Ddimensional+evaluation+of+text+summarization+with+in%2Dcontext)
41. Yunjie Ji, Yan Gong, Yiping Peng, Chao Ni, Peiyan Sun, Dongyu Pan, Baochang Ma, and Xiangang Li. 2023. Exploring chatgpt’s ability to rank content: A preliminary study on consistency with human preferences. arXiv preprint arXiv:2303.07610. — [arXiv:2303.07610](https://arxiv.org/abs/2303.07610) [find](https://scholar.google.com/scholar?q=Yunjie+Ji%2C+Yan+Gong%2C+Yiping+Peng%2C+Chao+Ni%2C+Peiyan+Sun%2C+Dongyu+Pan%2C+Baochang+Ma%2C+and+Xiangang+Li%2E+2023%2E+Exploring+chatgpt%E2%80%99s+ability+to+rank+content%3A+A+preliminary+study+on+consistency+with+human+prefer)
42. Qi Jia, Siyu Ren, Yizhu Liu, and Kenny Q Zhu. 2023. Zero-shot faithfulness evaluation for text summarization with foundation language model. arXiv preprint arXiv:2310.11648. [find](https://scholar.google.com/scholar?q=Qi+Jia%2C+Siyu+Ren%2C+Yizhu+Liu%2C+and+Kenny+Q+Zhu%2E+2023%2E+Zero%2Dshot+faithfulness+evaluation+for+text+summarization+with+foundation+language+model%2E+arXiv+preprint+arXiv%3A2310%2E11648%2E)
43. Dongfu Jiang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, and Wenhu Chen. 2023. [Tigerscore: Towards building explainable metric for all text generation tasks](https://md.research.elamur.ai/free/en/papers/jiang-2023-tigerscore-towards-building-explainable-metric/). arXiv preprint arXiv:2310.00752.
44. Liqiang Jing, Xuemeng Song, Xuming Lin, Zhongzhou Zhao, Wei Zhou, and Liqiang Nie. 2023. Stylized data-to-text generation: A case study in the e-commerce domain. ACM Transactions on Information Systems. [find](https://scholar.google.com/scholar?q=Liqiang+Jing%2C+Xuemeng+Song%2C+Xuming+Lin%2C+Zhongzhou+Zhao%2C+Wei+Zhou%2C+and+Liqiang+Nie%2E+2023%2E+Stylized+data%2Dto%2Dtext+generation%3A+A+case+study+in+the+e%2Dcommerce+domain%2E+ACM+Transactions+on+Information+System)
45. Katharina Kann, Abteen Ebrahimi, Joewie Koh, Shiran Dudy, and Alessandro Roncone. 2022. Open-domain dialogue generation: What we can do, cannot do, and should do next. In Proceedings of the 4th Workshop on NLP for Conversational AI, pages 148–165. [find](https://scholar.google.com/scholar?q=Katharina+Kann%2C+Abteen+Ebrahimi%2C+Joewie+Koh%2C+Shiran+Dudy%2C+and+Alessandro+Roncone%2E+2022%2E+Open%2Ddomain+dialogue+generation%3A+What+we+can+do%2C+cannot+do%2C+and+should+do+next%2E+In+Proceedings+of+the+4th+Worksh)
46. Sudipta Kar, Suraj Maharjan, A. Pastor López-Monroy, and Thamar Solorio. 2018. MPST: A corpus of movie plot synopses with tags. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Sudipta+Kar%2C+Suraj+Maharjan%2C+A%2E+Pastor+L%C3%B3pez%2DMonroy%2C+and+Thamar+Solorio%2E+2018%2E+MPST%3A+A+corpus+of+movie+plot+synopses+with+tags%2E+In+Proceedings+of+the+Eleventh+International+Conference+on+Language+Reso)
47. Pei Ke, Bosi Wen, Zhuoer Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, et al. 2023. [Critiquellm: Scaling llm-as-critic for effective and explainable evaluation of large language model generation](https://md.research.elamur.ai/free/en/papers/ke-2023-critiquellm-informative-critique-generation/). arXiv preprint arXiv:2311.18702.
48. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. 2023a. [Prometheus: Inducing fine-grained evaluation capability in language models](https://md.research.elamur.ai/free/en/papers/kim-2023-prometheus-inducing-fine-grained-evaluation/). arXiv preprint arXiv:2310.08491.
49. Tae Soo Kim, Yoonjoo Lee, Jamin Shin, Young-Ho Kim, and Juho Kim. 2023b. Evallm: Interactive evaluation of large language model prompts on user-defined criteria. arXiv preprint arXiv:2309.13633. — [arXiv:2309.13633](https://arxiv.org/abs/2309.13633) [find](https://scholar.google.com/scholar?q=Tae+Soo+Kim%2C+Yoonjoo+Lee%2C+Jamin+Shin%2C+Young%2DHo+Kim%2C+and+Juho+Kim%2E+2023b%2E+Evallm%3A+Interactive+evaluation+of+large+language+model+prompts+on+user%2Ddefined+criteria%2E+arXiv+preprint+arXiv%3A2309%2E13633%2E+%E2%80%94+%5Bar)
50. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193–203, Tampere, Finland. European Association for Machine Translation. [find](https://scholar.google.com/scholar?q=Tom+Kocmi+and+Christian+Federmann%2E+2023%2E+Large+language+models+are+state%2Dof%2Dthe%2Dart+evaluators+of+translation+quality%2E+In+Proceedings+of+the+24th+Annual+Conference+of+the+European+Association+for+Mach)
51. Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. 2023. Benchmarking cognitive biases in large language models as evaluators. arXiv preprint arXiv:2309.17012. [find](https://scholar.google.com/scholar?q=Ryan+Koo%2C+Minhwa+Lee%2C+Vipul+Raheja%2C+Jong+Inn+Park%2C+Zae+Myung+Kim%2C+and+Dongyeop+Kang%2E+2023%2E+Benchmarking+cognitive+biases+in+large+language+models+as+evaluators%2E+arXiv+preprint+arXiv%3A2309%2E17012%2E)
52. Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022. SummaC: Re-visiting NLI-based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics, 10:163–177. [find](https://scholar.google.com/scholar?q=Philippe+Laban%2C+Tobias+Schnabel%2C+Paul+N%2E+Bennett%2C+and+Marti+A%2E+Hearst%2E+2022%2E+SummaC%3A+Re%2Dvisiting+NLI%2Dbased+models+for+inconsistency+detection+in+summarization%2E+Transactions+of+the+Association+for+Comp)
53. Christoph Leiter, Juri Opitz, Daniel Deutsch, Yang Gao, Rotem Dror, and Steffen Eger. 2023. The eval4nlp 2023 shared task on prompting large language models as explainable metrics. arXiv preprint arXiv:2310.19792. [find](https://scholar.google.com/scholar?q=Christoph+Leiter%2C+Juri+Opitz%2C+Daniel+Deutsch%2C+Yang+Gao%2C+Rotem+Dror%2C+and+Steffen+Eger%2E+2023%2E+The+eval4nlp+2023+shared+task+on+prompting+large+language+models+as+explainable+metrics%2E+arXiv+preprint+arXi)
54. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. [Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). arXiv preprint arXiv:1910.13461.
55. Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. 2023a. [Generative judge for evaluating alignment](https://md.research.elamur.ai/free/en/papers/li-2023-generative-judge-evaluating-alignment/). arXiv preprint arXiv:2310.05470.
56. Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. 2023b. [Generative judge for evaluating alignment](https://md.research.elamur.ai/free/en/papers/li-2023-generative-judge-evaluating-alignment/). CoRR, abs/2310.05470.
57. Ruosen Li, Teerth Patel, and Xinya Du. 2023c. Prd: Peer rank and discussion improve large language model based evaluations. arXiv preprint arXiv:2307.02762. [find](https://scholar.google.com/scholar?q=Ruosen+Li%2C+Teerth+Patel%2C+and+Xinya+Du%2E+2023c%2E+Prd%3A+Peer+rank+and+discussion+improve+large+language+model+based+evaluations%2E+arXiv+preprint+arXiv%3A2307%2E02762%2E)
58. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023d. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval). [↗](https://github.com/tatsu-lab/alpaca_eval)
59. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110. [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+et+al%2E+2022%2E+Holistic+evaluation+of+language+models%2E+)
60. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
61. Yen-Ting Lin and Yun-Nung Chen. 2023. [Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models](https://md.research.elamur.ai/free/en/papers/lin-2023-llm-eval-unified-multi-dimensional/). arXiv preprint arXiv:2305.13711.
62. Yupian Lin, Tong Ruan, Jingping Liu, and Haofen Wang. 2023. A survey on neural data-to-text generation. IEEE Transactions on Knowledge and Data Engineering. [find](https://scholar.google.com/scholar?q=Yupian+Lin%2C+Tong+Ruan%2C+Jingping+Liu%2C+and+Haofen+Wang%2E+2023%2E+A+survey+on+neural+data%2Dto%2Dtext+generation%2E+IEEE+Transactions+on+Knowledge+and+Data+Engineering%2E)
63. Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. [How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation](https://md.research.elamur.ai/free/en/papers/liu-2016-how-not-evaluate-your-dialogue/). arXiv preprint arXiv:1603.08023.
64. Minqian Liu, Ying Shen, Zhiyang Xu, Yixin Cao, Eunah Cho, Vaibhav Kumar, Reza Ghanadan, and Lifu Huang. 2023a. X-eval: Generalizable multi-aspect text evaluation via augmented instruction tuning with auxiliary evaluation aspects. arXiv preprint arXiv:2311.08788. [find](https://scholar.google.com/scholar?q=Minqian+Liu%2C+Ying+Shen%2C+Zhiyang+Xu%2C+Yixin+Cao%2C+Eunah+Cho%2C+Vaibhav+Kumar%2C+Reza+Ghanadan%2C+and+Lifu+Huang%2E+2023a%2E+X%2Deval%3A+Generalizable+multi%2Daspect+text+evaluation+via+augmented+instruction+tuning+with+)
65. Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, et al. 2023b. [Alignbench: Benchmarking chinese alignment of large language models](https://md.research.elamur.ai/free/en/papers/liu-2023-alignbench-benchmarking-chinese-alignment/). arXiv preprint arXiv:2311.18743.
66. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023c. [Gpteval: Nlg evaluation using gpt-4 with better human alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/). arXiv preprint arXiv:2303.16634.
67. Yiqi Liu, Nafise Sadat Moosavi, and Chenghua Lin. 2023d. Llms as narcissistic evaluators: When ego inflates evaluation scores. arXiv preprint arXiv:2311.09766. [find](https://scholar.google.com/scholar?q=Yiqi+Liu%2C+Nafise+Sadat+Moosavi%2C+and+Chenghua+Lin%2E+2023d%2E+Llms+as+narcissistic+evaluators%3A+When+ego+inflates+evaluation+scores%2E+arXiv+preprint+arXiv%3A2311%2E09766%2E)
68. Yixin Liu and Pengfei Liu. 2021. [SimCLS: A simple framework for contrastive learning of abstractive summarization](https://md.research.elamur.ai/free/en/papers/liu-2021-simcls-simple-framework-contrastive-learning/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Pa…
69. Yongkang Liu, Shi Feng, Daling Wang, Yifei Zhang, and Hinrich Schütze. 2023e. Evaluate what you can’t evaluate: Unassessable generated responses quality. arXiv preprint arXiv:2305.14658. — [arXiv:2305.14658](https://arxiv.org/abs/2305.14658) [find](https://scholar.google.com/scholar?q=Yongkang+Liu%2C+Shi+Feng%2C+Daling+Wang%2C+Yifei+Zhang%2C+and+Hinrich+Sch%C3%BCtze%2E+2023e%2E+Evaluate+what+you+can%E2%80%99t+evaluate%3A+Unassessable+generated+responses+quality%2E+arXiv+preprint+arXiv%3A2305%2E14658%2E+%E2%80%94+%5BarXiv%3A2305)
70. Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. 2023f. [Calibrating llm-based evaluator](https://md.research.elamur.ai/free/en/papers/liu-2024-calibrating-llm-based-evaluator/). arXiv preprint arXiv:2309.13308.
71. Qingyu Lu, Baopu Qiu, Liang Ding, Liping Xie, and Dacheng Tao. 2023. [Error analysis prompting enables human-like translation evaluation in large language models](https://md.research.elamur.ai/free/en/papers/lu-2024-error-analysis-prompting-enables-human/): A case study on chatgpt. arXiv preprint arXiv:2303.13809.
72. Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. [Chatgpt as a factual inconsistency evaluator for abstractive text summarization](https://md.research.elamur.ai/free/en/papers/luo-2023-chatgpt-factual-inconsistency-evaluator-text/). arXiv preprint arXiv:2303.15621.
73. François Mairesse, Milica Gašić, Filip Jurčíček, Simon Keizer, Blaise Thomson, Kai Yu, and Steve Young. 2010. Phrase-based statistical language generation using graphical models and active learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages… [find](https://scholar.google.com/scholar?q=Fran%C3%A7ois+Mairesse%2C+Milica+Ga%C5%A1ic%CC%81%2C+Filip+Jurc%CC%8C%C3%ADc%CC%8Cek%2C+Simon+Keizer%2C+Blaise+Thomson%2C+Kai+Yu%2C+and+Steve+Young%2E+2010%2E+Phrase%2Dbased+statistical+language+generation+using+graphical+models+and+active+learning)
74. Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020. [Results of the WMT20 metrics shared task](https://md.research.elamur.ai/free/en/papers/mathur-2020-results-wmt20-metrics-shared-task/). In Proceedings of the Fifth Conference on Machine Translation, pages 688–725, Online. Association for Computational Linguistics.
75. Shikib Mehri and Maxine Eskenazi. 2020a. [Unsupervised evaluation of interactive dialog with DialoGPT](https://md.research.elamur.ai/free/en/papers/mehri-2020-unsupervised-evaluation-interactive-dialog-dialogpt/). In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 225–235, 1st virtual meeting. Association for Computational Linguistics.
76. Shikib Mehri and Maxine Eskenazi. 2020b. [USR: An unsupervised and reference free evaluation metric for dialog generation](https://md.research.elamur.ai/free/en/papers/mehri-2020-usr-unsupervised-reference-free-evaluation/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681–707, Online. Association for Computational Linguistics.
77. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. [Factscore: Fine-grained atomic evaluation of factual precision in long form text generation](https://md.research.elamur.ai/free/en/papers/min-2023-factscore-fine-grained-atomic-evaluation/). arXiv preprint arXiv:2305.14251.
78. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
79. Phil Ostheimer, Mayank Nagda, Marius Kloft, and Sophie Fellenz. 2023. Text style transfer evaluation using large language models. arXiv preprint arXiv:2308.13577. [find](https://scholar.google.com/scholar?q=Phil+Ostheimer%2C+Mayank+Nagda%2C+Marius+Kloft%2C+and+Sophie+Fellenz%2E+2023%2E+Text+style+transfer+evaluation+using+large+language+models%2E+arXiv+preprint+arXiv%3A2308%2E13577%2E)
80. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744.
81. Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. [Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics](https://md.research.elamur.ai/free/en/papers/pagnoni-2021-understanding-factuality-abstractive-summarization-frank/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Languag…
82. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
83. Yiwei Qin, Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2022. T5score: Discriminative fine-tuning of generative evaluation metrics. [find](https://scholar.google.com/scholar?q=Yiwei+Qin%2C+Weizhe+Yuan%2C+Graham+Neubig%2C+and+Pengfei+Liu%2E+2022%2E+T5score%3A+Discriminative+fine%2Dtuning+of+generative+evaluation+metrics%2E)
84. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. [COMET: A neural framework for MT evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics.
85. Ananya B Sai, Akash Kumar Mohankumar, and Mitesh M Khapra. 2022. A survey of evaluation metrics used for nlg systems. ACM Computing Surveys (CSUR), 55(2):1–39. [find](https://scholar.google.com/scholar?q=Ananya+B+Sai%2C+Akash+Kumar+Mohankumar%2C+and+Mitesh+M+Khapra%2E+2022%2E+A+survey+of+evaluation+metrics+used+for+nlg+systems%2E+ACM+Computing+Surveys+%28CSUR%29%2C+55%282%29%3A1%E2%80%9339%2E)
86. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. [BLEURT: Learning robust metrics for text generation](https://md.research.elamur.ai/free/en/papers/sellam-2020-bleurt-learning-robust-metrics-text/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics.
87. Yuchen Shen and Xiaojun Wan. 2023. Opinsummeval: Revisiting automated evaluation for opinion summarization. arXiv preprint arXiv:2310.18122. — [arXiv:2310.18122](https://arxiv.org/abs/2310.18122) [find](https://scholar.google.com/scholar?q=Yuchen+Shen+and+Xiaojun+Wan%2E+2023%2E+Opinsummeval%3A+Revisiting+automated+evaluation+for+opinion+summarization%2E+arXiv+preprint+arXiv%3A2310%2E18122%2E+%E2%80%94+%5BarXiv%3A2310%2E18122%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310%2E18122%29)
88. Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 90–121, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+pages+90%E2%80%93121%2C+Online%2E+Association+for+Computational+Linguistics%2E)
89. Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2021. [Societal biases in language generation: Progress and challenges](https://md.research.elamur.ai/free/en/papers/sheng-2021-societal-biases-language-generation-progress/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (V…
90. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.
91. Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2023. [Large language models encode clinical knowledge](https://md.research.elamur.ai/free/en/papers/singhal-2022-large-language-models-encode-clinical/). Nature, 620(7972):172–180.
92. Ondrej Skopek, Rahul Aralikatte, Sian Gooding, and Victor Carbune. 2023. Towards better evaluation of instruction-following: A case-study in summarization. arXiv preprint arXiv:2310.08394. [find](https://scholar.google.com/scholar?q=Ondrej+Skopek%2C+Rahul+Aralikatte%2C+Sian+Gooding%2C+and+Victor+Carbune%2E+2023%2E+Towards+better+evaluation+of+instruction%2Dfollowing%3A+A+case%2Dstudy+in+summarization%2E+arXiv+preprint+arXiv%3A2310%2E08394%2E)
93. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. [A study of translation edit rate with targeted human annotation](https://md.research.elamur.ai/free/en/papers/snover-2006-study-translation-edit-rate-targeted/). In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pages 223–231.
94. Andrea Sottana, Bin Liang, Kai Zou, and Zheng Yuan. 2023. [Evaluation metrics in the era of gpt-4: Reliably evaluating large language models on sequence to sequence tasks](https://md.research.elamur.ai/free/en/papers/sottana-2023-evaluation-metrics-era-gpt-4/). arXiv preprint arXiv:2310.13800.
95. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). arXiv preprint arXiv:2206.04615.
96. Tianyi Tang, Hongyuan Lu, Yuchen Eleanor Jiang, Haoyang Huang, Dongdong Zhang, Wayne Xin Zhao, and Furu Wei. 2023. Not all metrics are guilty: Improving nlg evaluation with llm paraphrasing. arXiv preprint arXiv:2305.15067. — [arXiv:2305.15067](https://arxiv.org/abs/2305.15067) [find](https://scholar.google.com/scholar?q=Tianyi+Tang%2C+Hongyuan+Lu%2C+Yuchen+Eleanor+Jiang%2C+Haoyang+Huang%2C+Dongdong+Zhang%2C+Wayne+Xin+Zhao%2C+and+Furu+Wei%2E+2023%2E+Not+all+metrics+are+guilty%3A+Improving+nlg+evaluation+with+llm+paraphrasing%2E+arXiv+pre)
97. Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. 2018. Ruber: An unsupervised method for automatic evaluation of open-domain dialog systems. In Proceedings of the AAAI conference on artificial intelligence, volume 32. [find](https://scholar.google.com/scholar?q=Chongyang+Tao%2C+Lili+Mou%2C+Dongyan+Zhao%2C+and+Rui+Yan%2E+2018%2E+Ruber%3A+An+unsupervised+method+for+automatic+evaluation+of+open%2Ddomain+dialog+systems%2E+In+Proceedings+of+the+AAAI+conference+on+artificial+inte)
98. Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. 2022. Zerocap: Zero-shot image-to-text generation for visual-semantic arithmetic. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17918–17928. [find](https://scholar.google.com/scholar?q=Yoad+Tewel%2C+Yoav+Shalev%2C+Idan+Schwartz%2C+and+Lior+Wolf%2E+2022%2E+Zerocap%3A+Zero%2Dshot+image%2Dto%2Dtext+generation+for+visual%2Dsemantic+arithmetic%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+an)
99. Brian Thompson and Matt Post. 2020. [Automatic machine translation evaluation in many languages via zero-shot paraphrasing](https://md.research.elamur.ai/free/en/papers/thompson-2020-automatic-machine-translation-evaluation-many/). In Proceedings of the 2020
100. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. [Cider: Consensus-based image description evaluation](https://md.research.elamur.ai/free/en/papers/vedantam-2014-cider-consensus-based-image-description/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575.
101. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2016. Show and tell: Lessons learned from the 2015 mscoco image captioning challenge. IEEE transactions on pattern analysis and machine intelligence, 39(4):652–663. [find](https://scholar.google.com/scholar?q=Oriol+Vinyals%2C+Alexander+Toshev%2C+Samy+Bengio%2C+and+Dumitru+Erhan%2E+2016%2E+Show+and+tell%3A+Lessons+learned+from+the+2015+mscoco+image+captioning+challenge%2E+IEEE+transactions+on+pattern+analysis+and+machine)
102. Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020a. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008–5020, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Kyunghyun+Cho%2C+and+Mike+Lewis%2E+2020a%2E+Asking+and+answering+questions+to+evaluate+the+factual+consistency+of+summaries%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Compu)
103. Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020b. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008–5020, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Kyunghyun+Cho%2C+and+Mike+Lewis%2E+2020b%2E+Asking+and+answering+questions+to+evaluate+the+factual+consistency+of+summaries%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Compu)
104. Danqing Wang, Kevin Yang, Hanlin Zhu, Xiaomeng Yang, Andrew Cohen, Lei Li, and Yuandong Tian. 2023a. Learning personalized story evaluation. arXiv preprint arXiv:2310.03304. — [arXiv:2310.03304](https://arxiv.org/abs/2310.03304) [find](https://scholar.google.com/scholar?q=Danqing+Wang%2C+Kevin+Yang%2C+Hanlin+Zhu%2C+Xiaomeng+Yang%2C+Andrew+Cohen%2C+Lei+Li%2C+and+Yuandong+Tian%2E+2023a%2E+Learning+personalized+story+evaluation%2E+arXiv+preprint+arXiv%3A2310%2E03304%2E+%E2%80%94+%5BarXiv%3A2310%2E03304%5D%28https)
105. Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023b. [Is chatgpt a good nlg evaluator? a preliminary study](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-nlg-evaluator-preliminary/). arXiv preprint arXiv:2303.04048.
106. Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023c. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926. [find](https://scholar.google.com/scholar?q=Peiyi+Wang%2C+Lei+Li%2C+Liang+Chen%2C+Dawei+Zhu%2C+Binghuai+Lin%2C+Yunbo+Cao%2C+Qi+Liu%2C+Tianyu+Liu%2C+and+Zhifang+Sui%2E+2023c%2E+Large+language+models+are+not+fair+evaluators%2E+arXiv+preprint+arXiv%3A2305%2E17926%2E)
107. Tianlu Wang, Ping Yu, Xiaoqing Ellen Tan, Sean O’Brien, Ramakanth Pasunuru, Jane Dwivedi-Yu, Olga Golovneva, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2023d. [Shepherd: A critic for language model generation](https://md.research.elamur.ai/free/en/papers/wang-2023-shepherd-critic-language-model-generation/). arXiv preprint arXiv:2308.04592.
108. Yaqing Wang, Jiepu Jiang, Mingyang Zhang, Cheng Li, Yi Liang, Qiaozhu Mei, and Michael Bendersky. 2023e. Automated evaluation of personalized text generation using large language models. arXiv preprint arXiv:2310.11593. — [arXiv:2310.11593](https://arxiv.org/abs/2310.11593) [find](https://scholar.google.com/scholar?q=Yaqing+Wang%2C+Jiepu+Jiang%2C+Mingyang+Zhang%2C+Cheng+Li%2C+Yi+Liang%2C+Qiaozhu+Mei%2C+and+Michael+Bendersky%2E+2023e%2E+Automated+evaluation+of+personalized+text+generation+using+large+language+models%2E+arXiv+preprin)
109. Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. 2023f. [Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization](https://md.research.elamur.ai/free/en/papers/wang-2023-pandalm-automatic-evaluation-benchmark-llm/). arXiv preprint arXiv:2306.05087.
110. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023g. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A%2E+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2023g%2E+Self%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%2E+In+Pro)
111. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022a. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenRevie…
112. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022b. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems, 35:24824–24837.
113. Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015. [Semantically conditioned LSTM-based natural language generation for spoken dialogue systems](https://md.research.elamur.ai/free/en/papers/wen-2015-semantically-conditioned-lstm-based-natural/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1711–1721, L…
114. Kevin Yang, Dan Klein, Nanyun Peng, and Yuandong Tian. 2023b. [DOC: Improving long story coherence with detailed outline control](https://md.research.elamur.ai/free/en/papers/yang-2023-doc-improving-long-story-coherence/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3378–3465, Toronto, Canada. Association for Compu…
115. Kevin Yang, Yuandong Tian, Nanyun Peng, and Dan Klein. 2022. Re3: Generating longer stories with recursive reprompting and revision. arXiv preprint arXiv:2210.06774. [find](https://scholar.google.com/scholar?q=Kevin+Yang%2C+Yuandong+Tian%2C+Nanyun+Peng%2C+and+Dan+Klein%2E+2022%2E+Re3%3A+Generating+longer+stories+with+recursive+reprompting+and+revision%2E+arXiv+preprint+arXiv%3A2210%2E06774%2E)
116. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. [Bartscore: Evaluating generated text as text generation](https://md.research.elamur.ai/free/en/papers/yuan-2021-bartscore-evaluating-generated-text-text/). In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 27263–27277.
117. Xiang Yue, Boshi Wang, Kai Zhang, Ziru Chen, Yu Su, and Huan Sun. 2023. Automatic evaluation of attribution by large language models. arXiv preprint arXiv:2305.06311. [find](https://scholar.google.com/scholar?q=Xiang+Yue%2C+Boshi+Wang%2C+Kai+Zhang%2C+Ziru+Chen%2C+Yu+Su%2C+and+Huan+Sun%2E+2023%2E+Automatic+evaluation+of+attribution+by+large+language+models%2E+arXiv+preprint+arXiv%3A2305%2E06311%2E)
118. Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2023. Evaluating large language models at evaluating instruction following. arXiv preprint arXiv:2310.07641. [find](https://scholar.google.com/scholar?q=Zhiyuan+Zeng%2C+Jiatong+Yu%2C+Tianyu+Gao%2C+Yu+Meng%2C+Tanya+Goyal%2C+and+Danqi+Chen%2E+2023%2E+Evaluating+large+language+models+at+evaluating+instruction+following%2E+arXiv+preprint+arXiv%3A2310%2E07641%2E)
119. Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2023a. Summit: Iterative text summarization via chatgpt. arXiv preprint arXiv:2305.14835. [find](https://scholar.google.com/scholar?q=Haopeng+Zhang%2C+Xiao+Liu%2C+and+Jiawei+Zhang%2E+2023a%2E+Summit%3A+Iterative+text+summarization+via+chatgpt%2E+arXiv+preprint+arXiv%3A2305%2E14835%2E)
120. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. [Personalizing dialogue agents: I have a dog, do you have pets too?](https://md.research.elamur.ai/free/en/papers/zhang-2018-personalizing-dialogue-agents-i-have/) arXiv preprint arXiv:1801.07243.
121. Ning Wu, Ming Gong, Linjun Shou, Shining Liang, and Daxin Jiang. 2023a. [Large language models are diverse role-players for summarization evaluation](https://md.research.elamur.ai/free/en/papers/wu-2023-large-language-models-diverse-role/). arXiv preprint arXiv:2303.15078.
122. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+Bertscore%3A+Evaluating+text+generation+with+BERT%2E+In+8th+International+Conference+on+Learning+Representations%2C+ICLR+2)
123. Yunshu Wu, Hayate Iso, Pouya Pezeshkpour, Nikita Bhutani, and Estevam Hruschka. 2023b. Less is more for long document summary evaluation by llms. arXiv preprint arXiv:2309.07382. [find](https://scholar.google.com/scholar?q=Yunshu+Wu%2C+Hayate+Iso%2C+Pouya+Pezeshkpour%2C+Nikita+Bhutani%2C+and+Estevam+Hruschka%2E+2023b%2E+Less+is+more+for+long+document+summary+evaluation+by+llms%2E+arXiv+preprint+arXiv%3A2309%2E07382%2E)
124. Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li. 2023b. [Wider and deeper llm networks are fairer llm evaluators](https://md.research.elamur.ai/free/en/papers/zhang-2023-wider-deeper-llm-networks-fairer/). arXiv preprint arXiv:2308.01862.
125. Wenda Xu, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Yang Wang, and Lei Li. 2023. [Instructscore: Towards explainable text generation evaluation with automatic feedback](https://md.research.elamur.ai/free/en/papers/xu-2023-instructscore-explainable-text-generation-evaluation/). arXiv preprint arXiv:2305.14282.
126. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. [MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance](https://md.research.elamur.ai/free/en/papers/zhao-2019-moverscore-text-generation-evaluating-contextualized/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International…
127. Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. 2024. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116. — [arXiv:2402.13116](https://arxiv.org/abs/2402.13116) [find](https://scholar.google.com/scholar?q=Xiaohan+Xu%2C+Ming+Li%2C+Chongyang+Tao%2C+Tao+Shen%2C+Reynold+Cheng%2C+Jinyang+Li%2C+Can+Xu%2C+Dacheng+Tao%2C+and+Tianyi+Zhou%2E+2024%2E+A+survey+on+knowledge+distillation+of+large+language+models%2E+arXiv+preprint+arXiv%3A2)
128. Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang. 2023a. [Fingpt: Open-source financial large language models](https://md.research.elamur.ai/free/en/papers/yang-2023-fingpt-open-source-financial-large/). arXiv preprint arXiv:2306.06031.
129. Yilun Zhao, Haowei Zhang, Shengyun Si, Linyong Nan, Xiangru Tang, and Arman Cohan. 2023. Investigating table-to-text generation capabilities of llms in real-world information seeking scenarios. [find](https://scholar.google.com/scholar?q=Yilun+Zhao%2C+Haowei+Zhang%2C+Shengyun+Si%2C+Linyong+Nan%2C+Xiangru+Tang%2C+and+Arman+Cohan%2E+2023%2E+Investigating+table%2Dto%2Dtext+generation+capabilities+of+llms+in+real%2Dworld+information+seeking+scenarios%2E)
130. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). arXiv preprint arXiv:2306.05685.
131. Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. 2022. Towards language-free training for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17907–17917. [find](https://scholar.google.com/scholar?q=Yufan+Zhou%2C+Ruiyi+Zhang%2C+Changyou+Chen%2C+Chunyuan+Li%2C+Chris+Tensmeyer%2C+Tong+Yu%2C+Jiuxiang+Gu%2C+Jinhui+Xu%2C+and+Tong+Sun%2E+2022%2E+Towards+language%2Dfree+training+for+text%2Dto%2Dimage+generation%2E+In+Proceedings+o)
132. Terry Yue Zhuo. 2023. Large language models are state-of-the-art evaluators of code generation. arXiv preprint arXiv:2304.14317. [find](https://scholar.google.com/scholar?q=Terry+Yue+Zhuo%2E+2023%2E+Large+language+models+are+state%2Dof%2Dthe%2Dart+evaluators+of+code+generation%2E+arXiv+preprint+arXiv%3A2304%2E14317%2E)

*Updated 31 October 2026*