---
title: "Calibrating LLM-Based Evaluator — paper by Liu et al. (2024)"
description: "Because many strong LLMs are closed or too costly to fine-tune, the authors propose AutoCalibrate… 38 references, cited by 63 papers here."
authors: ["Yuxuan Liu", "Tianchi Yang", "Shaohan Huang", "Zihan Zhang", "Haizhen Huang", "Furu Wei", "Weiwei Deng", "Feng Sun", "Qi Zhang"]
year: 2024
venue: "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)"
source: "https://aclanthology.org/2024.lrec-main.237/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2024-calibrating-llm-based-evaluator/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2024-calibrating-llm-based-evaluator/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Calibrating LLM-Based Evaluator

# Calibrating LLM-Based Evaluator

**Authors:** Yuxuan Liu, Tianchi Yang, [Shaohan Huang](https://md.research.elamur.ai/free/en/authors/shaohan-huang/), Zihan Zhang, Haizhen Huang, [Furu Wei](https://md.research.elamur.ai/free/en/authors/furu-wei/), Weiwei Deng, Feng Sun and [Qi Zhang](https://md.research.elamur.ai/free/en/authors/qi-zhang/) · **Year:** 2024 · **Venue:** [Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)](https://md.research.elamur.ai/free/en/venues/lrec/) ([LREC 2024](https://md.research.elamur.ai/free/en/venues/lrec/2024/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.lrec-main.237/) · [PDF](https://aclanthology.org/2024.lrec-main.237.pdf) · [OpenAlex](https://openalex.org/W7131789684)

**Citations:** 8 citations (OpenAlex, 2026-10-14) · [63](https://md.research.elamur.ai/free/en/papers/liu-2024-calibrating-llm-based-evaluator/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
LLMs are increasingly used as reference-free judges of generated text, but their scores often diverge from human experts. One cause is that evaluation prompts usually give only a score range and no explicit rubric. Because many strong LLMs are closed or too costly to fine-tune, the authors propose AutoCalibrate, a gradient-free way to align such evaluators with human judgment. Instead of modelling preferences directly, they encode them as a small golden set of expert-labelled samples. The LLM then works in stages:

- **Draft:** it infers candidate scoring criteria from randomly sampled few-shot labelled examples, using temperature sampling for variety.
- **Revisit:** the best-correlating candidates are kept.
- **Refine:** the LLM revises them using examples where its scores and the human scores clearly disagree.

The method was tested on text summarization, data-to-text generation and hallucination evaluation. Calibrated criteria noticeably raised correlation with expert ratings over uncalibrated prompts; exact figures fall in the omitted part of the provided text. This matters because the method works with API-only models, and the rubrics stay in readable natural language, so humans can inspect and adjust them.

## Key points
- The evaluation prompt is split into five parts: instructions, criteria, aspect, output format and sample. Only the criteria are optimized, to maximize correlation with the golden set D*.
- Refinement asks the LLM to apply four editing operations: Modification, Paraphrase, Adding Aspects or Details, and Calibrate.
- Algorithm 1 has three stages:
  - Monte-Carlo sampling over several few-shot sizes;
  - top-K filtering by correlation;
  - refinement on misaligned examples, followed by final selection of the best criteria.
- Datasets and aspects shown in the case studies:
  - SummEval (e.g., fluency and relevance);
  - SFRES (informativeness and naturalness, scored 1–6);
  - QAGS-CNN (factuality, scored 0–1).
- The mined optimal criteria sets are released in Appendix E of the paper. The method needs no gradients or model weights, so it suits API-based LLMs.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun and Qi Zhang. 2024. Calibrating LLM-Based Evaluator. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{liu2024calibrating,
  title = {{Calibrating LLM-Based Evaluator}},
  author = {Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun and Qi Zhang},
  booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year = {2024},
  doi = {10.63317/3posqwchnb6g},
  url = {https://doi.org/10.63317/3posqwchnb6g}
}
```

- [← previous Self-Taught Evaluators](https://md.research.elamur.ai/free/en/papers/wang-2024-self-taught-evaluators/)
- [next → Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA](https://md.research.elamur.ai/free/en/papers/wang-2024-leave-no-document-behind-benchmarking/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

38 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu. 2023. Exploring the use of large language models for reference-free text quality evaluation: A preliminary empirical study. arXiv preprint arXiv:2304.00723. — [arXiv:2304.00723](https://arxiv.org/abs/2304.00723) [find](https://scholar.google.com/scholar?q=Yi+Chen%2C+Rui+Wang%2C+Haiyun+Jiang%2C+Shuming+Shi%2C+and+Ruifeng+Xu%2E+2023%2E+Exploring+the+use+of+large+language+models+for+reference%2Dfree+text+quality+evaluation%3A+A+preliminary+empirical+study%2E+arXiv+preprint)
2. Mingkai Deng, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. 2021. [Compression, transduction, and creation: A unified framework for evaluating natural language generation](https://md.research.elamur.ai/free/en/papers/deng-2021-compression-transduction-creation-unified-framework/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7580–7605.
3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
4. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. [A survey for in-context learning](https://md.research.elamur.ai/free/en/papers/dong-2023-survey-context-learning/). arXiv preprint arXiv:2301.00234.
5. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. [Alpacafarm: A simulation framework for methods that learn from human feedback](https://md.research.elamur.ai/free/en/papers/dubois-2023-alpacafarm-simulation-framework-methods-that/). arXiv preprint arXiv:2305.14387.
6. Alexander R Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. [find](https://scholar.google.com/scholar?q=Alexander+R+Fabbri%2C+Wojciech+Kry%C5%9Bci%C5%84ski%2C+Bryan+McCann%2C+Caiming+Xiong%2C+Richard+Socher%2C+and+Dragomir+Radev%2E+2021%2E+Summeval%3A+Re%2Devaluating+summarization+evaluation%2E+Transactions+of+the+Association+for+Co)
7. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André FT Martins. 2022. [Results of wmt22 metrics shared task: Stop using bleu–neural metrics are better and more robust](https://md.research.elamur.ai/free/en/papers/freitag-2022-results-wmt22-metrics-shared-task/). In Proceedings of the Seventh Conference on…
8. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166. [find](https://scholar.google.com/scholar?q=Jinlan+Fu%2C+See%2DKiong+Ng%2C+Zhengbao+Jiang%2C+and+Pengfei+Liu%2E+2023%2E+Gptscore%3A+Evaluate+as+you+desire%2E+arXiv+preprint+arXiv%3A2302%2E04166%2E)
9. Max Grusky, Mor Naaman, and Yoav Artzi. 2018. [Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies](https://md.research.elamur.ai/free/en/papers/grusky-2018-newsroom-dataset-1-3-million/). In Proceedings of NAACL-HLT, pages 708–719.
10. Lishan Huang, Zheng Ye, Jinghui Qin, Liang Lin, and Xiaodan Liang. 2020. [Grade: Automatic graph-enhanced coherence metric for evaluating open-domain dialogue systems](https://md.research.elamur.ai/free/en/papers/huang-2020-grade-automatic-graph-enhanced-coherence/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9230–9240.
11. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. [Survey of hallucination in natural language generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/). ACM Computing Surveys, 55(12):1–38.
12. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. arXiv preprint arXiv:2302.14520. [find](https://scholar.google.com/scholar?q=Tom+Kocmi+and+Christian+Federmann%2E+2023%2E+Large+language+models+are+state%2Dof%2Dthe%2Dart+evaluators+of+translation+quality%2E+arXiv+preprint+arXiv%3A2302%2E14520%2E)
13. Chin-Yew Lin. 2004. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out, pages 74–81.
14. Pengfei Liu, Jinlan Fu, Yang Xiao, Weizhe Yuan, Shuaichen Chang, Junqi Dai, Yixin Liu, Zihuiwen Ye, and Graham Neubig. 2021. Explainaboard: An explainable leaderboard for nlp. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint… [find](https://scholar.google.com/scholar?q=Pengfei+Liu%2C+Jinlan+Fu%2C+Yang+Xiao%2C+Weizhe+Yuan%2C+Shuaichen+Chang%2C+Junqi+Dai%2C+Yixin+Liu%2C+Zihuiwen+Ye%2C+and+Graham+Neubig%2E+2021%2E+Explainaboard%3A+An+explainable+leaderboard+for+nlp%2E+In+Proceedings+of+the+59)
15. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634. [find](https://scholar.google.com/scholar?q=Yang+Liu%2C+Dan+Iter%2C+Yichong+Xu%2C+Shuohang+Wang%2C+Ruochen+Xu%2C+and+Chenguang+Zhu%2E+2023%2E+G%2Deval%3A+Nlg+evaluation+using+gpt%2D4+with+better+human+alignment%2E+arXiv+preprint+arXiv%3A2303%2E16634%2E)
16. Qingyu Lu, Baopu Qiu, Liang Ding, Liping Xie, and Dacheng Tao. 2023. [Error analysis prompting enables human-like translation evaluation in large language models](https://md.research.elamur.ai/free/en/papers/lu-2024-error-analysis-prompting-enables-human/): A case study on chatgpt. arXiv preprint arXiv:2303.13809.
17. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651. [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+et+al%2E+2023%2E+Self%2Drefine%3A+Iterative+refinement+with+self%2Df)
18. Shikib Mehri and Maxine Eskenazi. 2020. [Usr: An unsupervised and reference free evaluation metric for dialog generation](https://md.research.elamur.ai/free/en/papers/mehri-2020-usr-unsupervised-reference-free-evaluation/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681–707.
19. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. [Factscore: Fine-grained atomic evaluation of factual precision in long form text generation](https://md.research.elamur.ai/free/en/papers/min-2023-factscore-fine-grained-atomic-evaluation/). arXiv preprint arXiv:2305.14251.
20. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). preprint arXiv:2303.08774. arXiv Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Adva…
21. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
22. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. [Comet: A neural framework for mt evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702.
23. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/](https://github.com/tatsu-lab/) stanford_alpaca. [↗](https://github.com/tatsu-lab/)
24. Brian Thompson and Matt Post. 2020. [Automatic machine translation evaluation in many languages via zero-shot paraphrasing](https://md.research.elamur.ai/free/en/papers/thompson-2020-automatic-machine-translation-evaluation-many/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 90–121.
25. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. [Cider: Consensus-based image description evaluation](https://md.research.elamur.ai/free/en/papers/vedantam-2014-cider-consensus-based-image-description/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575.
26. Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020a. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008–5020. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Kyunghyun+Cho%2C+and+Mike+Lewis%2E+2020a%2E+Asking+and+answering+questions+to+evaluate+the+factual+consistency+of+summaries%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Compu)
27. Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020b. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008–5020. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Kyunghyun+Cho%2C+and+Mike+Lewis%2E+2020b%2E+Asking+and+answering+questions+to+evaluate+the+factual+consistency+of+summaries%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Compu)
28. Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023a. [Is chatgpt a good nlg evaluator? a preliminary study](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-nlg-evaluator-preliminary/). arXiv preprint arXiv:2303.04048.
29. Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 563–578. [find](https://scholar.google.com/scholar?q=Empirical+Methods+in+Natural+Language+Processing+and+the+9th+International+Joint+Conference+on+Natural+Language+Processing+%28EMNLP%2DIJCNLP%29%2C+pages+563%E2%80%93578%2E)
30. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. [Calibrate before use: Improving few-shot performance of language models](https://md.research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/). In International Conference on Machine Learning, pages 12697–12706. PMLR.
31. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+Xing%2C+et+al%2E+2023%2E+Judging+llm%2Das%2Da%2Djudge+with+mt%2Dbench+and+chatbot+arena%2E+a)
32. Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. [Towards a unified multi-dimensional evaluator for text generation](https://md.research.elamur.ai/free/en/papers/zhong-2022-unified-multi-dimensional-evaluator-text/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2023–2038.
33. Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023b. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926. [find](https://scholar.google.com/scholar?q=Peiyi+Wang%2C+Lei+Li%2C+Liang+Chen%2C+Dawei+Zhu%2C+Binghuai+Lin%2C+Yunbo+Cao%2C+Qi+Liu%2C+Tianyu+Liu%2C+and+Zhifang+Sui%2E+2023b%2E+Large+language+models+are+not+fair+evaluators%2E+arXiv+preprint+arXiv%3A2305%2E17926%2E)
34. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems, 35:24824–24837.
35. Tsung-Hsien Wen, Milica Gasic, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015. [Semantically conditioned lstm-based natural language generation for spoken dialogue systems](https://md.research.elamur.ai/free/en/papers/wen-2015-semantically-conditioned-lstm-based-natural/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1711–1721.
36. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. [Bartscore: Evaluating generated text as text generation](https://md.research.elamur.ai/free/en/papers/yuan-2021-bartscore-evaluating-generated-text-text/). Advances in Neural Information Processing Systems, 34:27263–27277.
37. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2A%2C+Varsha+Kishore%2A%2C+Felix+Wu%2A%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+In+International+Conference+on+Learning+Representations%2E)
38. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. 2019. [Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance](https://md.research.elamur.ai/free/en/papers/zhao-2019-moverscore-text-generation-evaluating-contextualized/). In Proceedings of the 2019 Conference on

*Updated 24 October 2026*