---
title: "Towards a Unified Multi-Dimensional Evaluator — Zhong 2022"
description: "Automatic evaluation of generated text still relies mostly on similarity to reference outputs, using metrics such as ROUGE, BLEU or BERTScore."
authors: ["Ming Zhong", "Yang Liu", "Da Yin", "Yuning Mao", "Yizhu Jiao", "Pengfei Liu", "Chenguang Zhu", "Heng Ji", "Jiawei Han"]
year: 2022
venue: "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2022.emnlp-main.131/"
doi: "10.18653/v1/2022.emnlp-main.131"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhong-2022-unified-multi-dimensional-evaluator-text/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhong-2022-unified-multi-dimensional-evaluator-text/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/) / Towards a Unified Multi-Dimensional Evaluator for Text Generation

# Towards a Unified Multi-Dimensional Evaluator for Text Generation

**Authors:** [Ming Zhong](https://md.research.elamur.ai/free/en/authors/ming-zhong/), [Yang Liu](https://md.research.elamur.ai/free/en/authors/yang-liu/), [Da Yin](https://md.research.elamur.ai/free/en/authors/da-yin/), [Yuning Mao](https://md.research.elamur.ai/free/en/authors/yuning-mao/), Yizhu Jiao, [Pengfei Liu](https://md.research.elamur.ai/free/en/authors/pengfei-liu/), [Chenguang Zhu](https://md.research.elamur.ai/free/en/authors/chenguang-zhu/), [Heng Ji](https://md.research.elamur.ai/free/en/authors/heng-ji/), [Jiawei Han](https://md.research.elamur.ai/free/en/authors/jiawei-han/) · **Year:** 2022 · **Venue:** Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.emnlp-main.131/) · [PDF](https://aclanthology.org/2022.emnlp-main.131.pdf) · [DOI 10.18653/v1/2022.emnlp-main.131](https://doi.org/10.18653/v1/2022.emnlp-main.131) · [OpenAlex](https://openalex.org/W4385572754)

**Citations:** 115 citations (openalex-snapshot · 2026-10-14) · [89](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Automatic evaluation of generated text still relies mostly on similarity to reference outputs, using metrics such as ROUGE, BLEU or BERTScore. Human evaluation, by contrast, judges text along several interpretable dimensions such as coherence, consistency, fluency and relevance. The authors propose UniEval, a single evaluator that scores each of these dimensions separately.

Every dimension is phrased as a yes/no question to a T5-based model. The score is the probability of "Yes" normalized against "No." Large human-annotated training sets are lacking, so the model is trained in two stages:
- **Intermediate stage:** multi-task learning on four related task types, all cast in the same Boolean QA format.
- **Evaluation stage:** training on pseudo data. Rule-based corruptions of reference texts produce negative examples for each dimension.

UniEval correlates with human judgments far better than prior unified evaluators on summarization and dialogue response generation. It also transfers zero-shot to new dimensions and tasks. This makes fine-grained automatic evaluation practical with one model rather than many dimension-specific ones.

## Key points
- Compared with the strongest unified evaluators (BARTScore and the dialogue evaluator of Mehri and Eskenazi), correlation with human scores improves by roughly 23% for summarization tasks and by more than 43% when evaluating generated dialogue responses.
- Negative training examples come from several rule-based corruptions:
  - Incoherent summaries are made by swapping in BM25-retrieved sentences.
  - Inconsistent ones use antonym, number and entity edits.
  - Disfluent ones repeat, delete or shuffle Poisson-length spans.
- Intermediate tasks cover NLI, opening sentence prediction, linguistics-related data and generic QA. In ablations, removing NLI or generic QA lowers the average Spearman correlation from 0.479 to about 0.46.
- In zero-shot data-to-text evaluation on SFRES and SFHOT, UniEval (Summ) reaches an average Spearman of 0.282, outperforming BARTScore on most dimensions.
- Limitations include:
  - The backbone is T5-large only.
  - The synthetic training data is noisy.
  - The model is a black box.
  - It is restricted to English.
- Code and models are released at github.com/maszhongming/UniEval.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

89 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 83 not yet.

- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://md.research.elamur.ai/free/en/papers/chan-2023-chateval-better-llm-based/)
- [A Survey on Evaluation of Large Language Models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/)
- [Evaluating Large Language Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/guo-2023-evaluating-large-language-models-comprehensive/)
- [Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation](https://md.research.elamur.ai/free/en/papers/liu-2022-revisiting-gold-standard-grounding-summarization/)
- [大语言模型的幻觉问题研究综述](https://md.research.elamur.ai/free/en/papers/liu-2024-hallucinations-large-language/)
- [AlignScore: Evaluating Factual Consistency with A Unified Alignment Function](https://md.research.elamur.ai/free/en/papers/zha-2023-alignscore-evaluating-factual-consistency-unified/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhong2022unified,
  title = {{Towards a Unified Multi-Dimensional Evaluator for Text Generation}},
  author = {Zhong, Ming and Liu, Yang and Yin, Da and Mao, Yuning and Jiao, Yizhu and Liu, Pengfei and Zhu, Chenguang and Ji, Heng and Han, Jiawei},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.131},
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.131}
}
```

- [← previous Towards a Cleaner Document-Oriented Multilingual Crawled Corpus](https://md.research.elamur.ai/free/en/papers/abadji-2022-towards-cleaner-document-oriented-multilingual/)
- [next → Towards Arabic Sentence Simplification via Classification and Generative Approaches](https://md.research.elamur.ai/free/en/papers/khallaf-2022-arabic-sentence-simplification-classification-generative/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/)

## References

53 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72. [find](https://scholar.google.com/scholar?q=Satanjeev+Banerjee+and+Alon+Lavie%2E+2005%2E+Meteor%3A+An+automatic+metric+for+mt+evaluation+with+improved+correlation+with+human+judgments%2E+In+Proceedings+of+the+acl+workshop+on+intrinsic+and+extrinsic+eva) [our card](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/)
2. Anja Belz and Albert Gatt. 2008. Intrinsic vs. extrinsic evaluation measures for referring expression generation. In Proceedings of ACL-08: HLT, Short Papers, pages 197–200. [find](https://scholar.google.com/scholar?q=Anja+Belz+and+Albert+Gatt%2E+2008%2E+Intrinsic+vs%2E+extrinsic+evaluation+measures+for+referring+expression+generation%2E+In+Proceedings+of+ACL%2D08%3A+HLT%2C+Short+Papers%2C+pages+197%E2%80%93200%2E)
3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+et+al%2E+2020%2E+Language+models+are+few%2Dshot+lear)
4. Meng Cao, Yue Dong, Jiapeng Wu, and Jackie Chi Kit Cheung. 2020. Factual error correction for abstractive summarization models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6251–6258. [find](https://scholar.google.com/scholar?q=Meng+Cao%2C+Yue+Dong%2C+Jiapeng+Wu%2C+and+Jackie+Chi+Kit+Cheung%2E+2020%2E+Factual+error+correction+for+abstractive+summarization+models%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+La) [our card](https://md.research.elamur.ai/free/en/papers/cao-2020-factual-error-correction-abstractive-summarization/)
5. Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In thirty-second AAAI conference on artificial intelligence. [find](https://scholar.google.com/scholar?q=Ziqiang+Cao%2C+Furu+Wei%2C+Wenjie+Li%2C+and+Sujian+Li%2E+2018%2E+Faithful+to+the+original%3A+Fact+aware+neural+abstractive+summarization%2E+In+thirty%2Dsecond+AAAI+conference+on+artificial+intelligence%2E)
6. Yiran Chen, Pengfei Liu, and Xipeng Qiu. 2021. Are factuality checkers reliable? adversarial meta-evaluation of factuality in summarization. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2082–2095. [find](https://scholar.google.com/scholar?q=Yiran+Chen%2C+Pengfei+Liu%2C+and+Xipeng+Qiu%2E+2021%2E+Are+factuality+checkers+reliable%3F+adversarial+meta%2Devaluation+of+factuality+in+summarization%2E+In+Findings+of+the+Association+for+Computational+Linguistic)
7. Yulong Chen, Yang Liu, Li Dong, Shuohang Wang, Chenguang Zhu, Michael Zeng, and Yue Zhang. 2022. Adaprompt: Adaptive model training for prompt-based nlp. arXiv preprint arXiv:2202.04824. [find](https://scholar.google.com/scholar?q=Yulong+Chen%2C+Yang+Liu%2C+Li+Dong%2C+Shuohang+Wang%2C+Chenguang+Zhu%2C+Michael+Zeng%2C+and+Yue+Zhang%2E+2022%2E+Adaprompt%3A+Adaptive+model+training+for+prompt%2Dbased+nlp%2E+arXiv+preprint+arXiv%3A2202%2E04824%2E)
8. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019a. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Christopher+Clark%2C+Kenton+Lee%2C+Ming%2DWei+Chang%2C+Tom+Kwiatkowski%2C+Michael+Collins%2C+and+Kristina+Toutanova%2E+2019a%2E+Boolq%3A+Exploring+the+surprising+difficulty+of+natural+yes%2Fno+questions%2E+In+Proceedings+o) [our card](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/)
9. Elizabeth Clark, Asli Celikyilmaz, and Noah A Smith. 2019b. Sentence mover’s similarity: Automatic evaluation for multi-sentence texts. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2748–2760. [find](https://scholar.google.com/scholar?q=Elizabeth+Clark%2C+Asli+Celikyilmaz%2C+and+Noah+A+Smith%2E+2019b%2E+Sentence+mover%E2%80%99s+similarity%3A+Automatic+evaluation+for+multi%2Dsentence+texts%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for)
10. Mingkai Deng, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. 2021. Compression, transduction, and creation: A unified framework for evaluating natural language generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7580–7605. [find](https://scholar.google.com/scholar?q=Mingkai+Deng%2C+Bowen+Tan%2C+Zhengzhong+Liu%2C+Eric+Xing%2C+and+Zhiting+Hu%2E+2021%2E+Compression%2C+transduction%2C+and+creation%3A+A+unified+framework+for+evaluating+natural+language+generation%2E+In+Proceedings+of+the)
11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N)
12. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of wikipedia: Knowledge-powered conversational agents. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. [find](https://scholar.google.com/scholar?q=Emily+Dinan%2C+Stephen+Roller%2C+Kurt+Shuster%2C+Angela+Fan%2C+Michael+Auli%2C+and+Jason+Weston%2E+2019%2E+Wizard+of+wikipedia%3A+Knowledge%2Dpowered+conversational+agents%2E+In+7th+International+Conference+on+Learning+R)
13. Bill Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (IWP2005). [find](https://scholar.google.com/scholar?q=Bill+Dolan+and+Chris+Brockett%2E+2005%2E+Automatically+constructing+a+corpus+of+sentential+paraphrases%2E+In+Third+International+Workshop+on+Paraphrasing+%28IWP2005%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/)
14. Esin Durmus, He He, and Mona Diab. 2020. Feqa: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055– 5070. [find](https://scholar.google.com/scholar?q=Esin+Durmus%2C+He+He%2C+and+Mona+Diab%2E+2020%2E+Feqa%3A+A+question+answering+evaluation+framework+for+faithfulness+assessment+in+abstractive+summarization%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Asso) [our card](https://md.research.elamur.ai/free/en/papers/durmus-2020-feqa-question-answering-evaluation-framework/)
15. Nouha Dziri, Ehsan Kamalloo, Kory Mathewson, and Osmar R Zaiane. 2019. Evaluating coherence in dialogue systems using entailment. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Shor… [find](https://scholar.google.com/scholar?q=Nouha+Dziri%2C+Ehsan+Kamalloo%2C+Kory+Mathewson%2C+and+Osmar+R+Zaiane%2E+2019%2E+Evaluating+coherence+in+dialogue+systems+using+entailment%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of)
16. Alexander R Fabbri, Wojciech Kryściński, Bryan Mc-Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. [find](https://scholar.google.com/scholar?q=Alexander+R+Fabbri%2C+Wojciech+Krys%CC%81cin%CC%81ski%2C+Bryan+Mc%2DCann%2C+Caiming+Xiong%2C+Richard+Socher%2C+and+Dragomir+Radev%2E+2021%2E+Summeval%3A+Re%2Devaluating+summarization+evaluation%2E+Transactions+of+the+Association+for)
17. Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, et al. 2020. Evaluating models’ local decision boundaries via contrast sets. In Findings of the Association for Computational Linguistics: EMNLP 2020, pag… [find](https://scholar.google.com/scholar?q=Matt+Gardner%2C+Yoav+Artzi%2C+Victoria+Basmov%2C+Jonathan+Berant%2C+Ben+Bogin%2C+Sihao+Chen%2C+Pradeep+Dasigi%2C+Dheeru+Dua%2C+Yanai+Elazar%2C+Ananth+Gottumukkala%2C+et+al%2E+2020%2E+Evaluating+models%E2%80%99+local+decision+boundar) [our card](https://md.research.elamur.ai/free/en/papers/gardner-2020-evaluating-models-local-decision-boundaries/)
18. Sebastian Gehrmann, Elizabeth Clark, and Thibault Sellam. 2022. Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text. arXiv preprint arXiv:2202.06935. [find](https://scholar.google.com/scholar?q=Sebastian+Gehrmann%2C+Elizabeth+Clark%2C+and+Thibault+Sellam%2E+2022%2E+Repairing+the+cracked+foundation%3A+A+survey+of+obstacles+in+evaluation+practices+for+generated+text%2E+arXiv+preprint+arXiv%3A2202%2E06935%2E)
19. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346– 361. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Daniel+Khashabi%2C+Elad+Segal%2C+Tushar+Khot%2C+Dan+Roth%2C+and+Jonathan+Berant%2E+2021%2E+Did+aristotle+use+a+laptop%3F+a+question+answering+benchmark+with+implicit+reasoning+strategies%2E+Transactions+of+)
20. Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tür. 2019. Topical-chat: Towards knowledge-grounded open-domain conversations. [find](https://scholar.google.com/scholar?q=Karthik+Gopalakrishnan%2C+Behnam+Hedayatnia%2C+Qinlang+Chen%2C+Anna+Gottardi%2C+Sanjeev+Kwatra%2C+Anu+Venkatesh%2C+Raefer+Gabriel%2C+and+Dilek+Hakkani%2DT%C3%BCr%2E+2019%2E+Topical%2Dchat%3A+Towards+knowledge%2Dgrounded+open%2Ddomain)
21. Proc. Interspeech 2019, pages 1891–1895. [find](https://scholar.google.com/scholar?q=Proc%2E+Interspeech+2019%2C+pages+1891%E2%80%931895%2E)
22. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. Advances in neural information processing systems, 28. [find](https://scholar.google.com/scholar?q=Karl+Moritz+Hermann%2C+Tomas+Kocisky%2C+Edward+Grefenstette%2C+Lasse+Espeholt%2C+Will+Kay%2C+Mustafa+Suleyman%2C+and+Phil+Blunsom%2E+2015%2E+Teaching+machines+to+read+and+comprehend%2E+Advances+in+neural+information+pr) [our card](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/)
23. Lishan Huang, Zheng Ye, Jinghui Qin, Liang Lin, and Xiaodan Liang. 2020. Grade: Automatic graph-enhanced coherence metric for evaluating open-domain dialogue systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9230–9240. [find](https://scholar.google.com/scholar?q=Lishan+Huang%2C+Zheng+Ye%2C+Jinghui+Qin%2C+Liang+Lin%2C+and+Xiaodan+Liang%2E+2020%2E+Grade%3A+Automatic+graph%2Denhanced+coherence+metric+for+evaluating+open%2Ddomain+dialogue+systems%2E+In+Proceedings+of+the+2020+Confer)
24. Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Lavinia Dunagan, Jacob Morrison, Alexander R Fabbri, Yejin Choi, and Noah A Smith. 2021. Bidimensional leaderboards: Generate and evaluate language hand in hand. arXiv preprint arXiv:2112.04139. [find](https://scholar.google.com/scholar?q=Jungo+Kasai%2C+Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Lavinia+Dunagan%2C+Jacob+Morrison%2C+Alexander+R+Fabbri%2C+Yejin+Choi%2C+and+Noah+A+Smith%2E+2021%2E+Bidimensional+leaderboards%3A+Generate+and+evaluate+language+hand+)
25. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:… [find](https://scholar.google.com/scholar?q=Daniel+Khashabi%2C+Snigdha+Chaturvedi%2C+Michael+Roth%2C+Shyam+Upadhyay%2C+and+Dan+Roth%2E+2018%2E+Looking+beyond+the+surface%3A+A+challenge+set+for+reading+comprehension+over+multiple+sentences%2E+In+Proceedings+of+) [our card](https://md.research.elamur.ai/free/en/papers/khashabi-2018-looking-beyond-surface-challenge-set/)
26. Daniel Khashabi, Tushar Khot, and Ashish Sabharwal. 2020. More bang for your buck: Natural perturbation for robust question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 163–170. [find](https://scholar.google.com/scholar?q=Daniel+Khashabi%2C+Tushar+Khot%2C+and+Ashish+Sabharwal%2E+2020%2E+More+bang+for+your+buck%3A+Natural+perturbation+for+robust+question+answering%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Nat)
27. Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332–9346. [find](https://scholar.google.com/scholar?q=Wojciech+Krys%CC%81cin%CC%81ski%2C+Bryan+McCann%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2020%2E+Evaluating+the+factual+consistency+of+abstractive+text+summarization%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+M)
28. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1– 67. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J+Liu%2E+2020%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext)
29. Philippe Laban, Tobias Schnabel, Paul N Bennett, and Marti A Hearst. 2021. Summac: Re-visiting nli-based models for inconsistency detection in summarization. arXiv preprint arXiv:2111.09525. [find](https://scholar.google.com/scholar?q=Philippe+Laban%2C+Tobias+Schnabel%2C+Paul+N+Bennett%2C+and+Marti+A+Hearst%2E+2021%2E+Summac%3A+Re%2Dvisiting+nli%2Dbased+models+for+inconsistency+detection+in+summarization%2E+arXiv+preprint+arXiv%3A2111%2E09525%2E)
30. Ehud Reiter and Anja Belz. 2009. An investigation into the validity of some metrics for automatically evaluating natural language generation systems. Computational Linguistics, 35(4):529–558. [find](https://scholar.google.com/scholar?q=Ehud+Reiter+and+Anja+Belz%2E+2009%2E+An+investigation+into+the+validity+of+some+metrics+for+automatically+evaluating+natural+language+generation+systems%2E+Computational+Linguistics%2C+35%284%29%3A529%E2%80%93558%2E)
31. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the… [find](https://scholar.google.com/scholar?q=Mike+Lewis%2C+Yinhan+Liu%2C+Naman+Goyal%2C+Marjan+Ghazvininejad%2C+Abdelrahman+Mohamed%2C+Omer+Levy%2C+Veselin+Stoyanov%2C+and+Luke+Zettlemoyer%2E+2020%2E+BART%3A+denoising+sequence%2Dto%2Dsequence+pre%2Dtraining+for+natural+l) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/)
32. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+2004%2E+Rouge%3A+A+package+for+automatic+evaluation+of+summaries%2E+In+Text+summarization+branches+out%2C+pages+74%E2%80%9381%2E)
33. Pengfei Liu, Jinlan Fu, Yang Xiao, Weizhe Yuan, Shuaichen Chang, Junqi Dai, Yixin Liu, Zihuiwen Ye, and Graham Neubig. 2021a. Explainaboard: An explainable leaderboard for nlp. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint… [find](https://scholar.google.com/scholar?q=Pengfei+Liu%2C+Jinlan+Fu%2C+Yang+Xiao%2C+Weizhe+Yuan%2C+Shuaichen+Chang%2C+Junqi+Dai%2C+Yixin+Liu%2C+Zihuiwen+Ye%2C+and+Graham+Neubig%2E+2021a%2E+Explainaboard%3A+An+explainable+leaderboard+for+nlp%2E+In+Proceedings+of+the+5)
34. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021b. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586. [find](https://scholar.google.com/scholar?q=Pengfei+Liu%2C+Weizhe+Yuan%2C+Jinlan+Fu%2C+Zhengbao+Jiang%2C+Hiroaki+Hayashi%2C+and+Graham+Neubig%2E+2021b%2E+Pre%2Dtrain%2C+prompt%2C+and+predict%3A+A+systematic+survey+of+prompting+methods+in+natural+language+processing%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/)
35. Shikib Mehri and Maxine Eskenazi. 2020. Usr: An unsupervised and reference free evaluation metric for dialog generation. arXiv preprint arXiv:2005.00456. [find](https://scholar.google.com/scholar?q=Shikib+Mehri+and+Maxine+Eskenazi%2E+2020%2E+Usr%3A+An+unsupervised+and+reference+free+evaluation+metric+for+dialog+generation%2E+arXiv+preprint+arXiv%3A2005%2E00456%2E) [our card](https://md.research.elamur.ai/free/en/papers/mehri-2020-usr-unsupervised-reference-free-evaluation/)
36. Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807. [find](https://scholar.google.com/scholar?q=Shashi+Narayan%2C+Shay+B+Cohen%2C+and+Mirella+Lapata%2E+2018%2E+Don%E2%80%99t+give+me+the+details%2C+just+the+summary%21+topic%2Daware+convolutional+neural+networks+for+extreme+summarization%2E+In+Proceedings+of+the+2018+Con)
37. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+annual+meeting+of+the+Association+for+Comp) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
38. GI Parisi, R Kemker, JL Part, C Kanan, and S Wermter. 2019. Continual lifelong learning with neural networks: A review. Neural Networks: the Official Journal of the International Neural Network Society, 113:54–71. [find](https://scholar.google.com/scholar?q=GI+Parisi%2C+R+Kemker%2C+JL+Part%2C+C+Kanan%2C+and+S+Wermter%2E+2019%2E+Continual+lifelong+learning+with+neural+networks%3A+A+review%2E+Neural+Networks%3A+the+Official+Journal+of+the+International+Neural+Network+Societ) [our card](https://md.research.elamur.ai/free/en/papers/parisi-2018-continual-lifelong-learning-neural-networks/)
39. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Information Retrieval, 3(4):333–389. [find](https://scholar.google.com/scholar?q=Stephen+Robertson+and+Hugo+Zaragoza%2E+2009%2E+The+probabilistic+relevance+framework%3A+Bm25+and+beyond%2E+Information+Retrieval%2C+3%284%29%3A333%E2%80%93389%2E)
40. Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. Questeval: Summarization asks for fact-based evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6594–6604. [find](https://scholar.google.com/scholar?q=Thomas+Scialom%2C+Paul%2DAlexis+Dray%2C+Sylvain+Lamprier%2C+Benjamin+Piwowarski%2C+Jacopo+Staiano%2C+Alex+Wang%2C+and+Patrick+Gallinari%2E+2021%2E+Questeval%3A+Summarization+asks+for+fact%2Dbased+evaluation%2E+In+Proceedings) [our card](https://md.research.elamur.ai/free/en/papers/scialom-2021-questeval-summarization-asks-fact-based/)
41. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083. [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Peter+J+Liu%2C+and+Christopher+D+Manning%2E+2017%2E+Get+to+the+point%3A+Summarization+with+pointer%2Dgenerator+networks%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computation)
42. Amanda Stent, Matthew Marge, and Mohit Singhai. 2005. Evaluating evaluation methods for generation in the presence of variation. In Proceedings of the 6th international conference on Computational Linguistics and Intelligent Text Processing, pages 341–351. [find](https://scholar.google.com/scholar?q=Amanda+Stent%2C+Matthew+Marge%2C+and+Mohit+Singhai%2E+2005%2E+Evaluating+evaluation+methods+for+generation+in+the+presence+of+variation%2E+In+Proceedings+of+the+6th+international+conference+on+Computational+Lin)
43. Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008–5020. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Kyunghyun+Cho%2C+and+Mike+Lewis%2E+2020%2E+Asking+and+answering+questions+to+evaluate+the+factual+consistency+of+summaries%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Comput)
44. Zhiguo Wang, Wael Hamza, and Radu Florian. 2017. Bilateral multi-perspective matching for natural language sentences. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 4144–4150. [find](https://scholar.google.com/scholar?q=Zhiguo+Wang%2C+Wael+Hamza%2C+and+Radu+Florian%2E+2017%2E+Bilateral+multi%2Dperspective+matching+for+natural+language+sentences%2E+In+Proceedings+of+the+26th+International+Joint+Conference+on+Artificial+Intelligen) [our card](https://md.research.elamur.ai/free/en/papers/wang-2017-bilateral-multi-perspective-matching-natural/)
45. Alex Warstadt, Amanpreet Singh, and Samuel Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+Bowman%2E+2019%2E+Neural+network+acceptability+judgments%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+7%3A625%E2%80%93641%2E)
46. Tsung-Hsien Wen, Milica Gasic, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned lstm-based natural language generation for spoken dialogue systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1711–1721. [find](https://scholar.google.com/scholar?q=Tsung%2DHsien+Wen%2C+Milica+Gasic%2C+Nikola+Mrk%C5%A1ic%CC%81%2C+Pei%2DHao+Su%2C+David+Vandyke%2C+and+Steve+Young%2E+2015%2E+Semantically+conditioned+lstm%2Dbased+natural+language+generation+for+spoken+dialogue+systems%2E+In+Proceed) [our card](https://md.research.elamur.ai/free/en/papers/wen-2015-semantically-conditioned-lstm-based-natural/)
47. Zheng Ye, Liucun Lu, Lishan Huang, Liang Lin, and Xiaodan Liang. 2021. Towards quantifiable dialogue coherence evaluation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1:… [find](https://scholar.google.com/scholar?q=Zheng+Ye%2C+Liucun+Lu%2C+Lishan+Huang%2C+Liang+Lin%2C+and+Xiaodan+Liang%2E+2021%2E+Towards+quantifiable+dialogue+coherence+evaluation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computationa)
48. Wenpeng Yin, Dragomir Radev, and Caiming Xiong. 2021. Docnli: A large-scale dataset for document-level natural language inference. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4913–4922. [find](https://scholar.google.com/scholar?q=Wenpeng+Yin%2C+Dragomir+Radev%2C+and+Caiming+Xiong%2E+2021%2E+Docnli%3A+A+large%2Dscale+dataset+for+document%2Dlevel+natural+language+inference%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL%2DIJC) [our card](https://md.research.elamur.ai/free/en/papers/yin-2021-docnli-large-scale-dataset-document/)
49. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34. [find](https://scholar.google.com/scholar?q=Weizhe+Yuan%2C+Graham+Neubig%2C+and+Pengfei+Liu%2E+2021%2E+Bartscore%3A+Evaluating+generated+text+as+text+generation%2E+Advances+in+Neural+Information+Processing+Systems%2C+34%2E) [our card](https://md.research.elamur.ai/free/en/papers/yuan-2021-bartscore-evaluating-generated-text-text/)
50. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+2019%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+arXiv+preprint+arXiv%3A1904%2E09675%2E)
51. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and William B Dolan. 2020. Dialogpt: Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational… [find](https://scholar.google.com/scholar?q=Yizhe+Zhang%2C+Siqi+Sun%2C+Michel+Galley%2C+Yen%2DChun+Chen%2C+Chris+Brockett%2C+Xiang+Gao%2C+Jianfeng+Gao%2C+Jingjing+Liu%2C+and+William+B+Dolan%2E+2020%2E+Dialogpt%3A+Large%2Dscale+generative+pre%2Dtraining+for+conversational+) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2019-dialogpt-large-scale-generative-pre/)
52. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. 2019. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International… [find](https://scholar.google.com/scholar?q=Wei+Zhao%2C+Maxime+Peyrard%2C+Fei+Liu%2C+Yang+Gao%2C+Christian+M+Meyer%2C+and+Steffen+Eger%2E+2019%2E+Moverscore%3A+Text+generation+evaluating+with+contextualized+embeddings+and+earth+mover+distance%2E+In+Proceedings+o) [our card](https://md.research.elamur.ai/free/en/papers/zhao-2019-moverscore-text-generation-evaluating-contextualized/)
53. Ming Zhong, Pengfei Liu, Danqing Wang, Xipeng Qiu, and Xuan-Jing Huang. 2019. Searching for effective neural extractive summarization: What works and what’s next. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1049–1058. [find](https://scholar.google.com/scholar?q=Ming+Zhong%2C+Pengfei+Liu%2C+Danqing+Wang%2C+Xipeng+Qiu%2C+and+Xuan%2DJing+Huang%2E+2019%2E+Searching+for+effective+neural+extractive+summarization%3A+What+works+and+what%E2%80%99s+next%2E+In+Proceedings+of+the+57th+Annual+Mee)