---
title: "Exploring the Use of Large Language Models for Reference-Free"
description: "The study offers practical guidance for using LLMs as evaluators where references are unavailable. 41 references, cited by 45 papers here."
authors: ["Yi Chen", "Rui Wang", "Haiyun Jiang", "Shuming Shi", "Ruifeng Xu"]
year: 2023
venue: "Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023 (Findings)"
source: "https://aclanthology.org/2023.findings-ijcnlp.32/"
doi: "10.18653/v1/2023.findings-ijcnlp.32"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2023-exploring-use-large-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2023-exploring-use-large-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Study

# Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Study

**Authors:** Yi Chen, [Rui Wang](https://md.research.elamur.ai/free/en/authors/rui-wang/), [Haiyun Jiang](https://md.research.elamur.ai/free/en/authors/haiyun-jiang/), [Shuming Shi](https://md.research.elamur.ai/free/en/authors/shuming-shi/) and [Ruifeng Xu](https://md.research.elamur.ai/free/en/authors/ruifeng-xu/) · **Year:** 2023 · **Venue:** [Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023 (Findings)](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2023](https://md.research.elamur.ai/free/en/venues/findings/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.findings-ijcnlp.32/) · [PDF](https://aclanthology.org/2023.findings-ijcnlp.32.pdf) · [DOI 10.18653/v1/2023.findings-ijcnlp.32](https://doi.org/10.18653/v1/2023.findings-ijcnlp.32) · [OpenAlex](https://openalex.org/W4392669905)

**Citations:** 68 citations (OpenAlex, 2026-10-14) · [45](https://md.research.elamur.ai/free/en/papers/chen-2023-exploring-use-large-language-models/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper asks whether large language models, mainly ChatGPT, can judge the quality of generated text without human-written references, and which way of prompting them works best. The authors compare three reference-free strategies. The first is an Explicit Score, where ChatGPT writes a numeric rating. The second is an Implicit Score, where text-davinci-001 and text-davinci-003 answer a yes/no question and the probability of "yes" serves as the score. The third is Pairwise Comparison, where ChatGPT picks the better of two texts. They test these on four generation tasks: summarization, dialogue, story generation and paraphrasing. Each method's agreement with human judgments is measured against established metrics such as ROUGE, BERTScore, BARTScore and ParaScore.

ChatGPT's Explicit Score proves the most accurate and stable option, beating most existing automatic metrics even with simple prompts. Pairwise comparison underperforms because ChatGPT considers most system outputs poor, so it struggles to separate two weak texts. The study offers practical guidance for using LLMs as evaluators where references are unavailable.

## Key points
- **Benchmarks:**
  - SummEval: 20 sampled news articles, 16 summaries each, rated on COH/FLU/CON/REL.
  - Dialogue-level FED: 124 conversations with fine-grained aspects; ERR was excluded.
  - OpenMEVA-ROC: 200 story beginnings with 5 storylines each.
  - Twitter (Extend): built on Twitter-Para, with 761 inputs and about 9.41 candidates per input.
- **Models:** ChatGPT (gpt-3.5-turbo-0301), text-davinci-003 and text-davinci-001.
  - The Implicit Score was estimated from the top-5 token log probabilities returned by the OpenAI API.
  - Greedy decoding is recommended over sampling for more reliable Explicit Scores.
- **Meta-evaluation:**
  - Individual scores were assessed with Spearman and Pearson correlations.
  - Pairwise comparison was assessed with Kendall's Tau-b on 200 randomly sampled pairs per dataset.
- **Implicit vs. Explicit Score:** Implicit Scores fall in a narrow, peaked distribution, while Explicit Scores spread more smoothly and so discriminate better between texts.
- **Prompt design:**
  - Rewording or shortening the prompt had little effect.
  - Detailed five-level scoring rules slightly hurt performance.
  - Asking for chain-of-thought justifications before the score also slightly hurt performance.
  - Code and data are released on GitHub (MilkWhite/LLMs_for_Reference_Free_Text_Quality_Evaluation).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Harbin Institute of Technology](https://md.research.elamur.ai/free/en/universities/harbin-institute-of-technology/) — University, Harbin, China · 76 works · 27,918 citations
- [Peng Cheng Laboratory](https://md.research.elamur.ai/free/en/labs/peng-cheng-laboratory/) — Research institute, Shenzhen, China · 18 works · 2,349 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 685 works

## Cite

Cite as: Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi and Ruifeng Xu. 2023. Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Study. Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023 (Findings).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{chen2023exploring,
  title = {{Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Study}},
  author = {Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi and Ruifeng Xu},
  booktitle = {Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023 (Findings)},
  year = {2023},
  doi = {10.18653/v1/2023.findings-ijcnlp.32},
  url = {https://doi.org/10.18653/v1/2023.findings-ijcnlp.32}
}
```

- [← previous RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation](https://md.research.elamur.ai/free/en/papers/xu-2023-recomp-improving-retrieval-augmented-lms/)
- [next → LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization](https://md.research.elamur.ai/free/en/papers/krishna-2023-longeval-guidelines-human-evaluation-faithfulness/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

41 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. on Methods for Optimizing and Evaluating Neural Language Generation, pages 82–89, Minneapolis, Minnesota. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=on+Methods+for+Optimizing+and+Evaluating+Neural+Language+Generation%2C+pages+82%E2%80%9389%2C+Minneapolis%2C+Minnesota%2E+Association+for+Computational+Linguistics%2E)
2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
3. Nuo Chen, Yan Wang, Haiyun Jiang, Deng Cai, Ziyang Chen, and Jia Li. 2022a. What would harry say? building dialogue agents for characters in a story. arXiv preprint arXiv:2211.06869. — [arXiv:2211.06869](https://arxiv.org/abs/2211.06869) [find](https://scholar.google.com/scholar?q=Nuo+Chen%2C+Yan+Wang%2C+Haiyun+Jiang%2C+Deng+Cai%2C+Ziyang+Chen%2C+and+Jia+Li%2E+2022a%2E+What+would+harry+say%3F+building+dialogue+agents+for+characters+in+a+story%2E+arXiv+preprint+arXiv%3A2211%2E06869%2E+%E2%80%94+%5BarXiv%3A2211%2E068)
4. Yi Chen, Haiyun Jiang, Lemao Liu, Rui Wang, Shuming Shi, and Ruifeng Xu. 2022b. Mcpg: A flexible multi-level controllable framework for unsupervised paraphrase generation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5948–5958. [find](https://scholar.google.com/scholar?q=Yi+Chen%2C+Haiyun+Jiang%2C+Lemao+Liu%2C+Rui+Wang%2C+Shuming+Shi%2C+and+Ruifeng+Xu%2E+2022b%2E+Mcpg%3A+A+flexible+multi%2Dlevel+controllable+framework+for+unsupervised+paraphrase+generation%2E+In+Findings+of+the+Associati)
5. Jian Guan, Zhexin Zhang, Zhuoer Feng, Zitao Liu, Wenbiao Ding, Xiaoxi Mao, Changjie Fan, and Minlie Huang. 2021. OpenMEVA: A benchmark for evaluating open-ended story generation metrics. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internati… [find](https://scholar.google.com/scholar?q=Jian+Guan%2C+Zhexin+Zhang%2C+Zhuoer+Feng%2C+Zitao+Liu%2C+Wenbiao+Ding%2C+Xiaoxi+Mao%2C+Changjie+Fan%2C+and+Minlie+Huang%2E+2021%2E+OpenMEVA%3A+A+benchmark+for+evaluating+open%2Dended+story+generation+metrics%2E+In+Proceeding)
6. Yunjie Ji, Yan Gong, Yiping Peng, Chao Ni, Peiyan Sun, Dongyu Pan, Baochang Ma, and Xiangang Li. 2023. Exploring chatgpt’s ability to rank content: A preliminary study on consistency with human preferences. [find](https://scholar.google.com/scholar?q=Yunjie+Ji%2C+Yan+Gong%2C+Yiping+Peng%2C+Chao+Ni%2C+Peiyan+Sun%2C+Dongyu+Pan%2C+Baochang+Ma%2C+and+Xiangang+Li%2E+2023%2E+Exploring+chatgpt%E2%80%99s+ability+to+rank+content%3A+A+preliminary+study+on+consistency+with+human+prefer)
7. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311.
8. M. G. Kendall. 1945. The treatment of ties in ranking problems. Biometrika, 33(3):239–251. [find](https://scholar.google.com/scholar?q=M%2E+G%2E+Kendall%2E+1945%2E+The+treatment+of+ties+in+ranking+problems%2E+Biometrika%2C+33%283%29%3A239%E2%80%93251%2E)
9. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. [Scaling instruction-finetuned language models](https://md.research.elamur.ai/free/en/papers/chung-2022-scaling-instruction-finetuned-language-models/). arXiv preprint arXiv:2210.11416.
10. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
11. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. [find](https://scholar.google.com/scholar?q=Tom+Kocmi+and+Christian+Federmann%2E+2023%2E+Large+language+models+are+state%2Dof%2Dthe%2Dart+evaluators+of+translation+quality%2E)
12. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. [GLM: General language model pretraining with autoregressive blank infilling](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320–335…
13. Alexander R. Fabbri, Wojciech Kryściński, Bryan Mc-Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021a. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. [find](https://scholar.google.com/scholar?q=Alexander+R%2E+Fabbri%2C+Wojciech+Krys%CC%81cin%CC%81ski%2C+Bryan+Mc%2DCann%2C+Caiming+Xiong%2C+Richard+Socher%2C+and+Dragomir+Radev%2E+2021a%2E+SummEval%3A+Re%2Devaluating+summarization+evaluation%2E+Transactions+of+the+Association+f)
14. Alexander R Fabbri, Wojciech Kryściński, Bryan Mc-Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021b. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. [find](https://scholar.google.com/scholar?q=Alexander+R+Fabbri%2C+Wojciech+Krys%CC%81cin%CC%81ski%2C+Bryan+Mc%2DCann%2C+Caiming+Xiong%2C+Richard+Socher%2C+and+Dragomir+Radev%2E+2021b%2E+Summeval%3A+Re%2Devaluating+summarization+evaluation%2E+Transactions+of+the+Association+fo)
15. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. [find](https://scholar.google.com/scholar?q=Jinlan+Fu%2C+See%2DKiong+Ng%2C+Zhengbao+Jiang%2C+and+Pengfei+Liu%2E+2023%2E+Gptscore%3A+Evaluate+as+you+desire%2E)
16. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. [Simcse: Simple contrastive learning of sentence embeddings](https://md.research.elamur.ai/free/en/papers/gao-2021-simcse-simple-contrastive-learning-sentence/). arXiv preprint arXiv:2104.08821.
17. Sarik Ghazarian, Johnny Wei, Aram Galstyan, and Nanyun Peng. 2019. Better automatic evaluation of open-domain dialogue systems with contextualized embeddings. In Proceedings of the Workshop Longxuan Ma, Ziyu Zhuang, Weinan Zhang, Mingda Li, and Ting Liu. 2022. SelF-eval: Self-supervised fine-grained… [find](https://scholar.google.com/scholar?q=Sarik+Ghazarian%2C+Johnny+Wei%2C+Aram+Galstyan%2C+and+Nanyun+Peng%2E+2019%2E+Better+automatic+evaluation+of+open%2Ddomain+dialogue+systems+with+contextualized+embeddings%2E+In+Proceedings+of+the+Workshop+Longxuan+M)
18. Shikib Mehri and Maxine Eskenazi. 2020a. [Unsupervised evaluation of interactive dialog with DialoGPT](https://md.research.elamur.ai/free/en/papers/mehri-2020-unsupervised-evaluation-interactive-dialog-dialogpt/). In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 225–235, 1st virtual meeting. Association for Computational Linguistics.
19. Shikib Mehri and Maxine Eskenazi. 2020b. [USR: An unsupervised and reference free evaluation metric for dialog generation](https://md.research.elamur.ai/free/en/papers/mehri-2020-usr-unsupervised-reference-free-evaluation/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681–707, Online. Association for Computational Linguistics.
20. Mohsen Mesgar, Sebastian Bücker, and Iryna Gurevych. 2020. [Dialogue coherence assessment without explicit dialogue act labels](https://md.research.elamur.ai/free/en/papers/mesgar-2020-dialogue-coherence-assessment-without-explicit/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1439–1450, Online. Association for Computational Linguistics.
21. Mavuto M Mukaka. 2012. A guide to appropriate use of correlation coefficient in medical research. Malawi medical journal, 24(3):69–71. [find](https://scholar.google.com/scholar?q=Mavuto+M+Mukaka%2E+2012%2E+A+guide+to+appropriate+use+of+correlation+coefficient+in+medical+research%2E+Malawi+medical+journal%2C+24%283%29%3A69%E2%80%9371%2E)
22. Tong Niu, Semih Yavuz, Yingbo Zhou, Nitish Shirish Keskar, Huan Wang, and Caiming Xiong. 2021. Unsupervised paraphrasing with pretrained language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5136–5150, Online and Punta Cana, Dominican Repub… [find](https://scholar.google.com/scholar?q=Tong+Niu%2C+Semih+Yavuz%2C+Yingbo+Zhou%2C+Nitish+Shirish+Keskar%2C+Huan+Wang%2C+and+Caiming+Xiong%2E+2021%2E+Unsupervised+paraphrasing+with+pretrained+language+models%2E+In+Proceedings+of+the+2021+Conference+on+Empir)
23. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744.
24. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
25. Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023a. [Is chatgpt a good nlg evaluator? a preliminary study](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-nlg-evaluator-preliminary/).
26. Rui Wang, Jianzhu Bao, Fei Mi, Yi Chen, Hongru Wang, Yasheng Wang, Yitong Li, Lifeng Shang, Kam-Fai Wong, and Ruifeng Xu. 2023b. Retrieval-free knowledge injection through multi-document traversal for dialogue models. In Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rui+Wang%2C+Jianzhu+Bao%2C+Fei+Mi%2C+Yi+Chen%2C+Hongru+Wang%2C+Yasheng+Wang%2C+Yitong+Li%2C+Lifeng+Shang%2C+Kam%2DFai+Wong%2C+and+Ruifeng+Xu%2E+2023b%2E+Retrieval%2Dfree+knowledge+injection+through+multi%2Ddocument+traversal+for)
27. Wei Xu, Chris Callison-Burch, and Bill Dolan. 2015. [SemEval-2015 task 1: Paraphrase and semantic similarity in Twitter (PIT)](https://md.research.elamur.ai/free/en/papers/xu-2015-semeval-2015-task-1-paraphrase/). In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 1–11, Denver, Colorado. Association for Computational Linguistics.
28. Wei Xu, Alan Ritter, Chris Callison-Burch, William B. Dolan, and Yangfeng Ji. 2014. Extracting lexically divergent paraphrases from Twitter. Transactions of the Association for Computational Linguistics, 2:435– 448. [find](https://scholar.google.com/scholar?q=Wei+Xu%2C+Alan+Ritter%2C+Chris+Callison%2DBurch%2C+William+B%2E+Dolan%2C+and+Yangfeng+Ji%2E+2014%2E+Extracting+lexically+divergent+paraphrases+from+Twitter%2E+Transactions+of+the+Association+for+Computational+Linguisti)
29. Lingfeng Shen, Haiyun Jiang, Lemao Liu, and Shuming Shi. 2023a. Sen2pro: A probabilistic perspective to sentence embedding from pre-trained language model. arXiv preprint arXiv:2306.02247. [find](https://scholar.google.com/scholar?q=Lingfeng+Shen%2C+Haiyun+Jiang%2C+Lemao+Liu%2C+and+Shuming+Shi%2E+2023a%2E+Sen2pro%3A+A+probabilistic+perspective+to+sentence+embedding+from+pre%2Dtrained+language+model%2E+arXiv+preprint+arXiv%3A2306%2E02247%2E)
30. Lingfeng Shen, Haiyun Jiang, Lemao Liu, and Shuming Shi. 2023b. A simple and plug-and-play method for unsupervised sentence representation enhancement. arXiv preprint arXiv:2305.07824. — [arXiv:2305.07824](https://arxiv.org/abs/2305.07824) [find](https://scholar.google.com/scholar?q=Lingfeng+Shen%2C+Haiyun+Jiang%2C+Lemao+Liu%2C+and+Shuming+Shi%2E+2023b%2E+A+simple+and+plug%2Dand%2Dplay+method+for+unsupervised+sentence+representation+enhancement%2E+arXiv+preprint+arXiv%3A2305%2E07824%2E+%E2%80%94+%5BarXiv%3A2305%2E0)
31. Lingfeng Shen, Lemao Liu, Haiyun Jiang, and Shuming Shi. 2022. On the evaluation metrics for paraphrase generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3178–3190, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Lingfeng+Shen%2C+Lemao+Liu%2C+Haiyun+Jiang%2C+and+Shuming+Shi%2E+2022%2E+On+the+evaluation+metrics+for+paraphrase+generation%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+Language+Proce)
32. Mingyang Song, Haiyun Jiang, Shuming Shi, Songfang Yao, Shilong Lu, Yi Feng, Huafeng Liu, and Liping Jing. 2023. Is chatgpt a good keyphrase generator? a preliminary study. arXiv preprint arXiv:2303.13001. — [arXiv:2303.13001](https://arxiv.org/abs/2303.13001) [find](https://scholar.google.com/scholar?q=Mingyang+Song%2C+Haiyun+Jiang%2C+Shuming+Shi%2C+Songfang+Yao%2C+Shilong+Lu%2C+Yi+Feng%2C+Huafeng+Liu%2C+and+Liping+Jing%2E+2023%2E+Is+chatgpt+a+good+keyphrase+generator%3F+a+preliminary+study%2E+arXiv+preprint+arXiv%3A2303%2E1)
33. Hong Sun and Ming Zhou. 2012. Joint learning of a dual SMT system for paraphrase generation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 38–42, Jeju Island, Korea. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Hong+Sun+and+Ming+Zhou%2E+2012%2E+Joint+learning+of+a+dual+SMT+system+for+paraphrase+generation%2E+In+Proceedings+of+the+50th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+2%3A+Short)
34. Brian Thompson and Matt Post. 2020. [Automatic machine translation evaluation in many languages via zero-shot paraphrasing](https://md.research.elamur.ai/free/en/papers/thompson-2020-automatic-machine-translation-evaluation-many/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 90–121, Online. Association for Computational Linguistics.
35. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
36. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. [Bartscore: Evaluating generated text as text generation](https://md.research.elamur.ai/free/en/papers/yuan-2021-bartscore-evaluating-generated-text-text/). In Advances in Neural Information Processing Systems, volume 34, pages 27263–27277. Curran Associates, Inc.
37. Jerrold H Zar. 2005. Spearman rank correlation. Encyclopedia of biostatistics, 7. [find](https://scholar.google.com/scholar?q=Jerrold+H+Zar%2E+2005%2E+Spearman+rank+correlation%2E+Encyclopedia+of+biostatistics%2C+7%2E)
38. Chen Zhang, Yiming Chen, Luis Fernando D’Haro, Yan Zhang, Thomas Friedrichs, Grandee Lee, and Haizhou Li. 2021. DynaEval: Unifying turn and dialogue level evaluation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conferenc… [find](https://scholar.google.com/scholar?q=Chen+Zhang%2C+Yiming+Chen%2C+Luis+Fernando+D%E2%80%99Haro%2C+Yan+Zhang%2C+Thomas+Friedrichs%2C+Grandee+Lee%2C+and+Haizhou+Li%2E+2021%2E+DynaEval%3A+Unifying+turn+and+dialogue+level+evaluation%2E+In+Proceedings+of+the+59th+Annual)
39. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. [Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv preprint arXiv:2205.01068.
40. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2A%2C+Varsha+Kishore%2A%2C+Felix+Wu%2A%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+In+International+Conference+on+Learning+Representations%2E)
41. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. [MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance](https://md.research.elamur.ai/free/en/papers/zhao-2019-moverscore-text-generation-evaluating-contextualized/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International…

*Updated 24 October 2026*