---
title: "MoverScore: Text Generation Evaluating with — Zhao 2019"
description: "Standard metrics such as BLEU and ROUGE judge generated text mostly by n-gram overlap."
authors: ["Wei Zhao", "Maxime Peyrard", "Fei Liu", "Yang Gao", "Christian M. Meyer", "Steffen Eger"]
year: 2019
venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)"
source: "https://aclanthology.org/D19-1053/"
doi: "10.18653/v1/d19-1053"
arxiv: "1909.02622"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhao-2019-moverscore-text-generation-evaluating-contextualized/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhao-2019-moverscore-text-generation-evaluating-contextualized/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) / MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance

# MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance

**Authors:** [Wei Zhao](https://md.research.elamur.ai/free/en/authors/wei-zhao/), [Maxime Peyrard](https://md.research.elamur.ai/free/en/authors/maxime-peyrard/), [Fei Liu](https://md.research.elamur.ai/free/en/authors/fei-liu/), [Yang Gao](https://md.research.elamur.ai/free/en/authors/yang-gao/), [Christian M. Meyer](https://md.research.elamur.ai/free/en/authors/christian-m-meyer/), [Steffen Eger](https://md.research.elamur.ai/free/en/authors/steffen-eger/) · **Year:** 2019 · **Venue:** Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-1053/) · [PDF](https://aclanthology.org/D19-1053.pdf) · [DOI 10.18653/v1/d19-1053](https://doi.org/10.18653/v1/d19-1053) · [arXiv 1909.02622](https://arxiv.org/abs/1909.02622) · [OpenAlex](https://openalex.org/W2970785793)

**Citations:** 477 citations (openalex-snapshot · 2026-10-14) · [345](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Standard metrics such as BLEU and ROUGE judge generated text mostly by n-gram overlap. They therefore miss outputs that express the reference meaning in different wording, and they track human judgments poorly. The authors frame evaluation as measuring semantic distance between system and reference texts and introduce MoverScore, an unsupervised metric. Both texts are encoded with contextualized embeddings, chiefly BERT, with ELMo and word2vec as comparisons. The texts are then compared using Word Mover's Distance, a special case of Earth Mover's Distance that the authors extend to n-grams weighted by IDF.

They test word-level (WMD-1, WMD-2) and sentence-level (SMD) variants. They also compare layer aggregation via p-means and routing, and BERT fine-tuned on tasks like MNLI. Evaluation covers machine translation (WMT), summarization, image captioning (MSCOCO) and data-to-text generation (BAGEL, SFHOTEL). MoverScore surpasses strong unsupervised baselines on three of the four tasks and frequently rivals or beats supervised, task-specific metrics. This makes it a promising general-purpose metric across NLG tasks. Code is publicly released.

## Key points
- On MT, the word mover metric averages 74.3 correlation with human judgments. This is 5.8 points above the best supervised metric (RUSE) and 2.4 above the contemporaneous BERTScore.
- Gains over unsupervised baselines include +5.7 over METEOR++ (MT), +3.0 over SPICE (captioning) and +2.2 over METEOR (dialogue response generation).
- Most of the improvement comes from BERT rather than word2vec or ELMo. Fine-tuning BERT on MNLI adds about 1.8 points on average across 7 language pairs.
- Soft alignment (MoverScore) is marginally better than hard alignment (BERTScore), and unigrams slightly outperform bigrams.
- MoverScore separates low- and high-quality translations better than SentBLEU, reaching roughly 0.6–0.7 correlation versus 0.4–0.5 for existing MT metrics.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

345 papers of our library cite it (of the 140884 whose bibliographies we read); 20 of them have a page here, 325 not yet.

- [Assessing the Role of Context in Chat Translation Evaluation: Is Context Helpful and Under What Conditions?](https://md.research.elamur.ai/free/en/papers/agrawal-2024-assessing-role-context-chat-translation/)
- [Affective and Contextual Embedding for Sarcasm Detection](https://md.research.elamur.ai/free/en/papers/babanejad-2020-affective-contextual-embedding-sarcasm-detection/)
- [Re-evaluating Evaluation in Text Summarization](https://md.research.elamur.ai/free/en/papers/bhandari-2020-re-evaluating-evaluation-text-summarization/)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://md.research.elamur.ai/free/en/papers/chan-2023-chateval-better-llm-based/)
- [A Rationale-centric Counterfactual Data Augmentation Method for Cross-Document Event Coreference Resolution](https://md.research.elamur.ai/free/en/papers/ding-2024-rationale-centric-counterfactual-data-augmentation/)
- [Ragas: Automated Evaluation of Retrieval Augmented Generation](https://md.research.elamur.ai/free/en/papers/es-2023-ragas-automated-evaluation-retrieval-augmented/)
- [Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation](https://md.research.elamur.ai/free/en/papers/liu-2022-revisiting-gold-standard-grounding-summarization/)
- [CLASE: A Hybrid Method for Chinese Legalese Stylistic Evaluation](https://md.research.elamur.ai/free/en/papers/ma-2026-clase-hybrid-method-chinese-legalese/)
- [LENS: A Learnable Evaluation Metric for Text Simplification](https://md.research.elamur.ai/free/en/papers/maddela-2023-lens-learnable-evaluation-metric-text/)
- [A Human-in/on-the-Loop Framework for Accessible Text Generation](https://md.research.elamur.ai/free/en/papers/moreno-2026-human-loop-framework-accessible-text/)
- [DART: Open-Domain Structured Data Record to Text Generation](https://md.research.elamur.ai/free/en/papers/nan-2020-dart-open-domain-structured-data/)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://md.research.elamur.ai/free/en/papers/pillutla-2021-mauve-measuring-gap-between-neural/)
- [COMET: A Neural Framework for MT Evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/)
- [Investigating Pretrained Language Models for Graph-to-Text Generation](https://md.research.elamur.ai/free/en/papers/ribeiro-2021-investigating-pretrained-language-models-graph/)
- [Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task](https://md.research.elamur.ai/free/en/papers/sellam-2020-learning-evaluate-translation-beyond-english/)
- [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-nlg-evaluator-preliminary/)
- [Revisiting Metric Reliability for Fine-grained Evaluation of Machine Translation and Summarization in Indian Languages](https://md.research.elamur.ai/free/en/papers/yari-2026-revisiting-metric-reliability-fine-grained/)
- [MORQA: Benchmarking Evaluation Metrics for Medical Open-Ended Question Answering](https://md.research.elamur.ai/free/en/papers/yim-2026-morqa-benchmarking-evaluation-metrics-medical/)
- [DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence](https://md.research.elamur.ai/free/en/papers/zhao-2023-discoscore-evaluating-text-generation-bert/)
- [Towards a Unified Multi-Dimensional Evaluator for Text Generation](https://md.research.elamur.ai/free/en/papers/zhong-2022-unified-multi-dimensional-evaluator-text/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhao2019moverscore,
  title = {{MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance}},
  author = {Zhao, Wei and Peyrard, Maxime and Liu, Fei and Gao, Yang and Meyer, Christian M. and Eger, Steffen},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1053},
  eprint = {1909.02622},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d19-1053}
}
```

- [← previous Morphologically Annotated Corpora for Seven Arabic Dialects: Taizi, Sanaani, Najdi, Jordanian, Syrian, Iraqi and Moroccan](https://md.research.elamur.ai/free/en/papers/aclbulk-2019-alshargi-morphologically-annotated-corpora-seven-arabic/)
- [next → MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension](https://md.research.elamur.ai/free/en/papers/fisch-2019-mrqa-2019-shared-task-evaluating/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/)

## References

49 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. SPICE: semantic propositional image caption evaluation. In Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V, pages 382–398. [find](https://scholar.google.com/scholar?q=Peter+Anderson%2C+Basura+Fernando%2C+Mark+Johnson%2C+and+Stephen+Gould%2E+2016%2E+SPICE%3A+semantic+propositional+image+caption+evaluation%2E+In+Computer+Vision+%2D+ECCV+2016+%2D+14th+European+Conference%2C+Amsterdam%2C+Th) [our card](https://md.research.elamur.ai/free/en/papers/anderson-2016-spice-semantic-propositional-image-caption/)
2. Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. 2019. Better rewards yield better summaries: Learning to summarise without references. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, Hong Kong, China. [find](https://scholar.google.com/scholar?q=Florian+Bo%CC%88hm%2C+Yang+Gao%2C+Christian+M%2E+Meyer%2C+Ori+Shapira%2C+Ido+Dagan%2C+and+Iryna+Gurevych%2E+2019%2E+Better+rewards+yield+better+summaries%3A+Learning+to+summarise+without+references%2E+In+Proceedings+of+the+20)
3. Ondrej Bojar, Yvette Graham, and Amir Kamran. 2017. Results of the WMT17 metrics shared task. In Proceedings of the Conference on Machine Translation (WMT). [find](https://scholar.google.com/scholar?q=Ondrej+Bojar%2C+Yvette+Graham%2C+and+Amir+Kamran%2E+2017%2E+Results+of+the+WMT17+metrics+shared+task%2E+In+Proceedings+of+the+Conference+on+Machine+Translation+%28WMT%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2017-results-wmt17-metrics-shared-task/)
4. Arun Chaganty, Stephen Mussmann, and Percy Liang. 2018. The price of debiasing automatic metrics in natural language evalaution. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 643–653. \#\# Conclusion We investigated new unsupe… [find](https://scholar.google.com/scholar?q=Arun+Chaganty%2C+Stephen+Mussmann%2C+and+Percy+Liang%2E+2018%2E+The+price+of+debiasing+automatic+metrics+in+natural+language+evalaution%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Comput)
5. Dorin Comaniciu and Peter Meer. 2002. Mean shift: A robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis & Machine Intelligence, (5):603–619. [find](https://scholar.google.com/scholar?q=Dorin+Comaniciu+and+Peter+Meer%2E+2002%2E+Mean+shift%3A+A+robust+approach+toward+feature+space+analysis%2E+IEEE+Transactions+on+Pattern+Analysis+%26+Machine+Intelligence%2C+%285%29%3A603%E2%80%93619%2E)
6. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Michel+Galley%2C+Chris+Brockett%2C+Jianfeng+Gao%2C+and+Bill+Dolan%2E+2016%2E+A+diversity%2Dpromoting+objective+function+for+neural+conversation+models%2E+In+Proceedings+of+the+North+American+Chapter+of+th) [our card](https://md.research.elamur.ai/free/en/papers/li-2016-diversity-promoting-objective-function-neural/)
7. Yin Cui, Guandao Yang, Andreas Veit, Xun Huang, and Serge Belongie. 2018. Learning to evaluate image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5804–5812. [find](https://scholar.google.com/scholar?q=Yin+Cui%2C+Guandao+Yang%2C+Andreas+Veit%2C+Xun+Huang%2C+and+Serge+Belongie%2E+2018%2E+Learning+to+evaluate+image+captioning%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+pages)
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+BERT%3A+pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv%3A1810%2E04805%2E)
9. Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of summaries. In Proceedings of ACL workshop on Text Summarization Branches Out, pages 74–81, Barcelona, Spain. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+2004%2E+ROUGE%3A+A+Package+for+Automatic+Evaluation+of+summaries%2E+In+Proceedings+of+ACL+workshop+on+Text+Summarization+Branches+Out%2C+pages+74%E2%80%9381%2C+Barcelona%2C+Spain%2E)
10. Steffen Eger, Gözde Gül Şahin, Andreas Rücklé, Ji-Ung Lee, Claudia Schulz, Mohsen Mesgar, Kr-ishnkant Swarnkar, Edwin Simpson, and Iryna Gurevych. 2019. Text processing like humans do: Visually attacking and shielding NLP systems. In Proceedings of the 2019 Conference of the North American Chap… [find](https://scholar.google.com/scholar?q=Steffen+Eger%2C+Go%CC%88zde+Gu%CC%88l+S%CC%A7ahin%2C+Andreas+Ru%CC%88ckle%CC%81%2C+Ji%2DUng+Lee%2C+Claudia+Schulz%2C+Mohsen+Mesgar%2C+Kr%2Dishnkant+Swarnkar%2C+Edwin+Simpson%2C+and+Iryna+Gurevych%2E+2019%2E+Text+processing+like+humans+do%3A+Visually+a)
11. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer. [find](https://scholar.google.com/scholar?q=Tsung%2DYi+Lin%2C+Michael+Maire%2C+Serge+Belongie%2C+James+Hays%2C+Pietro+Perona%2C+Deva+Ramanan%2C+Piotr+Dolla%CC%81r%2C+and+C+Lawrence+Zitnick%2E+2014%2E+Microsoft+coco%3A+Common+objects+in+context%2E+In+European+conference+on+) [our card](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/)
12. Chia-Wei Liu, Ryan Lowe, Iulian V. Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the Conference on Empirical Methods in Natural… [find](https://scholar.google.com/scholar?q=Chia%2DWei+Liu%2C+Ryan+Lowe%2C+Iulian+V%2E+Serban%2C+Michael+Noseworthy%2C+Laurent+Charlin%2C+and+Joelle+Pineau%2E+2016%2E+How+NOT+to+evaluate+your+dialogue+system%3A+An+empirical+study+of+unsupervised+evaluation+metrics)
13. Albert Gatt and Emiel Krahmer. 2018. Survey of the state of the art in natural language generation: Core tasks, applications and evaluation. Journal of Artificial Intelligence Research (JAIR). [find](https://scholar.google.com/scholar?q=Albert+Gatt+and+Emiel+Krahmer%2E+2018%2E+Survey+of+the+state+of+the+art+in+natural+language+generation%3A+Core+tasks%2C+applications+and+evaluation%2E+Journal+of+Artificial+Intelligence+Research+%28JAIR%29%2E)
14. Liyuan Liu, Xiang Ren, Jingbo Shang, Xiaotao Gu, Jian Peng, and Jiawei Han. 2018. Efficient contextualized representation: Language model pruning for sequence labeling. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Liyuan+Liu%2C+Xiang+Ren%2C+Jingbo+Shang%2C+Xiaotao+Gu%2C+Jian+Peng%2C+and+Jiawei+Han%2E+2018%2E+Efficient+contextualized+representation%3A+Language+model+pruning+for+sequence+labeling%2E+In+Proceedings+of+the+Conferenc)
15. Max Grusky, Mor Naaman, and Yoav Artzi. 2018. NEWSROOM: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). [find](https://scholar.google.com/scholar?q=Max+Grusky%2C+Mor+Naaman%2C+and+Yoav+Artzi%2E+2018%2E+NEWSROOM%3A+A+dataset+of+1%2E3+million+summaries+with+diverse+extractive+strategies%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+th) [our card](https://md.research.elamur.ai/free/en/papers/grusky-2018-newsroom-dataset-1-3-million/)
16. Nelson F Liu, Matt Gardner, Yonatan Belinkov, Matthew Peters, and Noah A Smith. 2019. Linguistic knowledge and transferability of contextual representations. arXiv preprint arXiv:1903.08855. [find](https://scholar.google.com/scholar?q=Nelson+F+Liu%2C+Matt+Gardner%2C+Yonatan+Belinkov%2C+Matthew+Peters%2C+and+Noah+A+Smith%2E+2019%2E+Linguistic+knowledge+and+transferability+of+contextual+representations%2E+arXiv+preprint+arXiv%3A1903%2E08855%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2019-linguistic-knowledge-transferability-contextual-representations/)
17. Yinuo Guo, Chong Ruan, and Junfeng Hu. 2018. Meteor++: Incorporating copy knowledge into machine translation evaluation. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 740–745. [find](https://scholar.google.com/scholar?q=Yinuo+Guo%2C+Chong+Ruan%2C+and+Junfeng+Hu%2E+2018%2E+Meteor%2B%2B%3A+Incorporating+copy+knowledge+into+machine+translation+evaluation%2E+In+Proceedings+of+the+Third+Conference+on+Machine+Translation%3A+Shared+Task+Pape)
18. Chi-kiu Lo. 2017. MEANT 2.0: Accurate semantic MT evaluation for any output language. In Proceedings of the Second Conference on Machine Translation, WMT 2017, Copenhagen, Denmark, September 7-8, 2017, pages 589–597. [find](https://scholar.google.com/scholar?q=Chi%2Dkiu+Lo%2E+2017%2E+MEANT+2%2E0%3A+Accurate+semantic+MT+evaluation+for+any+output+language%2E+In+Proceedings+of+the+Second+Conference+on+Machine+Translation%2C+WMT+2017%2C+Copenhagen%2C+Denmark%2C+September+7%2D8%2C+2017) [our card](https://md.research.elamur.ai/free/en/papers/lo-2017-meant-2-0-accurate-semantic/)
19. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of Neural Information Processing Systems (NIPS). [find](https://scholar.google.com/scholar?q=Karl+Moritz+Hermann%2C+Tomas+Kocisky%2C+Edward+Grefenstette%2C+Lasse+Espeholt%2C+Will+Kay%2C+Mustafa+Suleyman%2C+and+Phil+Blunsom%2E+2015%2E+Teaching+machines+to+read+and+comprehend%2E+In+Proceedings+of+Neural+Informat) [our card](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/)
20. Annie Louis and Ani Nenkova. 2013. Automatically assessing machine summary content without a gold standard. Computational Linguistics, 39(2):267– 300. [find](https://scholar.google.com/scholar?q=Annie+Louis+and+Ani+Nenkova%2E+2013%2E+Automatically+assessing+machine+summary+content+without+a+gold+standard%2E+Computational+Linguistics%2C+39%282%29%3A267%E2%80%93+300%2E) [our card](https://md.research.elamur.ai/free/en/papers/louis-2013-automatically-assessing-machine-summary-content/)
21. Eduard Hovy, Chin-Yew Lin, Liang Zhou, and Junichi Fukumoto. 2006. Automated summarization evaluation with basic elements. In Proceedings of the Fifth Conference on Language Resources and Evaluation (LREC 2006), pages 604–611. [find](https://scholar.google.com/scholar?q=Eduard+Hovy%2C+Chin%2DYew+Lin%2C+Liang+Zhou%2C+and+Junichi+Fukumoto%2E+2006%2E+Automated+summarization+evaluation+with+basic+elements%2E+In+Proceedings+of+the+Fifth+Conference+on+Language+Resources+and+Evaluation+%28)
22. Qingsong Ma, Ondrej Bojar, and Yvette Graham. 2018. Results of the WMT18 metrics shared task. In Proceedings of the Third Conference on Machine Translation (WMT). [find](https://scholar.google.com/scholar?q=Qingsong+Ma%2C+Ondrej+Bojar%2C+and+Yvette+Graham%2E+2018%2E+Results+of+the+WMT18+metrics+shared+task%2E+In+Proceedings+of+the+Third+Conference+on+Machine+Translation+%28WMT%29%2E)
23. Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, and Kilian Q. Weinberger. 2015. From word embeddings to document distances. In Proceedings of the International Conference on Machine Learning (ICML). [find](https://scholar.google.com/scholar?q=Matt+J%2E+Kusner%2C+Yu+Sun%2C+Nicholas+I%2E+Kolkin%2C+and+Kilian+Q%2E+Weinberger%2E+2015%2E+From+word+embeddings+to+document+distances%2E+In+Proceedings+of+the+International+Conference+on+Machine+Learning+%28ICML%29%2E)
24. François Mairesse, Milica Gašić, Filip Jurčı́ček, Simon Keizer, Blaise Thomson, Kai Yu, and Steve Young. 2010. Phrase-based statistical language generation Maxime Peyrard, Teresa Botschen, and Iryna Gurevych. 2017. Learning to score system summaries for better content selection evaluation. In P… [find](https://scholar.google.com/scholar?q=Franc%CC%A7ois+Mairesse%2C+Milica+Gas%CC%8Cic%CC%81%2C+Filip+Jurc%CC%8C%C4%B1%CC%81c%CC%8Cek%2C+Simon+Keizer%2C+Blaise+Thomson%2C+Kai+Yu%2C+and+Steve+Young%2E+2010%2E+Phrase%2Dbased+statistical+language+generation+Maxime+Peyrard%2C+Teresa+Botschen%2C+and+Ir)
25. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2019. Putting evaluation in context: Contextual embeddings improve machine translation evaluation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2799–2808, Florence, Italy. Association for Computatio… [find](https://scholar.google.com/scholar?q=Nitika+Mathur%2C+Timothy+Baldwin%2C+and+Trevor+Cohn%2E+2019%2E+Putting+evaluation+in+context%3A+Contextual+embeddings+improve+machine+translation+evaluation%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Ass)
26. Matt Post. 2018. A call for clarity in reporting bleu scores. In Proceedings of the Third Conference on Machine Translation (WMT). [find](https://scholar.google.com/scholar?q=Matt+Post%2E+2018%2E+A+call+for+clarity+in+reporting+bleu+scores%2E+In+Proceedings+of+the+Third+Conference+on+Machine+Translation+%28WMT%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/)
27. Ehud Reiter. 2018. A structured review of the validity of BLEU. Computational Linguistics, 44(3):393– 401. [find](https://scholar.google.com/scholar?q=Ehud+Reiter%2E+2018%2E+A+structured+review+of+the+validity+of+BLEU%2E+Computational+Linguistics%2C+44%283%29%3A393%E2%80%93+401%2E) [our card](https://md.research.elamur.ai/free/en/papers/reiter-2018-structured-review-validity-bleu/)
28. Ani Nenkova and Rebecca J. Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In Proceedings of the 2004 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 145–152. Association for Computatio… [find](https://scholar.google.com/scholar?q=Ani+Nenkova+and+Rebecca+J%2E+Passonneau%2E+2004%2E+Evaluating+content+selection+in+summarization%3A+The+pyramid+method%2E+In+Proceedings+of+the+2004+Conference+of+the+North+American+Chapter+of+the+Association+f)
29. Yossi Rubner, Carlo Tomasi, and Leonidas J. Guibas. 2000. The earth mover’s distance as a metric for image retrieval. International Journal of Computer Vision. [find](https://scholar.google.com/scholar?q=Yossi+Rubner%2C+Carlo+Tomasi%2C+and+Leonidas+J%2E+Guibas%2E+2000%2E+The+earth+mover%E2%80%99s+distance+as+a+metric+for+image+retrieval%2E+International+Journal+of+Computer+Vision%2E)
30. Andreas Rücklé, Steffen Eger, Maxime Peyrard, and Iryna Gurevych. 2018. Concatenated power mean word embeddings as universal cross-lingual sentence representations. CoRR, abs/1803.01400. — [arXiv:1803.01400](https://arxiv.org/abs/1803.01400) [find](https://scholar.google.com/scholar?q=Andreas+Ru%CC%88ckle%CC%81%2C+Steffen+Eger%2C+Maxime+Peyrard%2C+and+Iryna+Gurevych%2E+2018%2E+Concatenated+power+mean+word+embeddings+as+universal+cross%2Dlingual+sentence+representations%2E+CoRR%2C+abs%2F1803%2E01400%2E+%E2%80%94+%5BarXiv%3A18)
31. Jun-Ping Ng and Viktoria Abrecht. 2015. Better summarization evaluation with word embeddings for rouge. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1925–1930, Lisbon, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jun%2DPing+Ng+and+Viktoria+Abrecht%2E+2015%2E+Better+summarization+evaluation+with+word+embeddings+for+rouge%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages)
32. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Peter+J%2E+Liu%2C+and+Christopher+D%2E+Manning%2E+2017%2E+Get+to+the+point%3A+Summarization+with+pointer%2Dgenerator+networks%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Association+for+Computational+)
33. Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why We Need New Evaluation Metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241–2252, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jekaterina+Novikova%2C+Ondr%CC%8Cej+Dus%CC%8Cek%2C+Amanda+Cercas+Curry%2C+and+Verena+Rieser%2E+2017%2E+Why+We+Need+New+Evaluation+Metrics+for+NLG%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Lan)
34. Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. 2018. RUSE: Regressor using sentence embeddings for automatic machine translation evaluation. In Proceedings of the Third Conference on Machine Translation (WMT). [find](https://scholar.google.com/scholar?q=Hiroki+Shimanaka%2C+Tomoyuki+Kajiwara%2C+and+Mamoru+Komachi%2E+2018%2E+RUSE%3A+Regressor+using+sentence+embeddings+for+automatic+machine+translation+evaluation%2E+In+Proceedings+of+the+Third+Conference+on+Machine) [our card](https://md.research.elamur.ai/free/en/papers/shimanaka-2018-ruse-regressor-sentence-embeddings-automatic/)
35. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. arXiv preprint arXiv:1905.06316. [find](https://scholar.google.com/scholar?q=Ian+Tenney%2C+Patrick+Xia%2C+Berlin+Chen%2C+Alex+Wang%2C+Adam+Poliak%2C+R+Thomas+McCoy%2C+Najoung+Kim%2C+Benjamin+Van+Durme%2C+Samuel+R+Bowman%2C+Dipanjan+Das%2C+et+al%2E+2019%2E+What+do+you+learn+from+context%3F+probing+for+s) [our card](https://md.research.elamur.ai/free/en/papers/tenney-2019-learn-context-probing-sentence-structure/)
36. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, pages 311–318, Stroudsburg, PA, USA. Association for Computational Lingu… [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+BLEU%3A+A+Method+for+Automatic+Evaluation+of+Machine+Translation%2E+In+Proceedings+of+the+40th+Annual+Meeting+on+Association+for+Computat) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
37. Stephen Tratz and Eduard H Hovy. 2008. Summarization Evaluation Using Transformed Basic Elements. In Proceedings of the text analysing conference, (TAC 2008). [find](https://scholar.google.com/scholar?q=Stephen+Tratz+and+Eduard+H+Hovy%2E+2008%2E+Summarization+Evaluation+Using+Transformed+Basic+Elements%2E+In+Proceedings+of+the+text+analysing+conference%2C+%28TAC+2008%29%2E)
38. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL). [find](https://scholar.google.com/scholar?q=Matthew+E%2E+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+In+Proceedings+of+the+North+American+Ch)
39. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based Image Description Evaluation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 4566–4575. [find](https://scholar.google.com/scholar?q=Ramakrishna+Vedantam%2C+C%2E+Lawrence+Zitnick%2C+and+Devi+Parikh%2E+2015%2E+CIDEr%3A+Consensus%2Dbased+Image+Description+Evaluation%2E+In+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+CVPR+2015%2C+Boston%2C) [our card](https://md.research.elamur.ai/free/en/papers/vedantam-2014-cider-consensus-based-image-description/)
40. Maxime Peyrard. 2019a. A simple theoretical model of importance for summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1059–1073, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Maxime+Peyrard%2E+2019a%2E+A+simple+theoretical+model+of+importance+for+summarization%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+1059%E2%80%931073%2C+Florence)
41. Matt P Wand and M Chris Jones. 1994. Kernel smoothing. Chapman and Hall/CRC. [find](https://scholar.google.com/scholar?q=Matt+P+Wand+and+M+Chris+Jones%2E+1994%2E+Kernel+smoothing%2E+Chapman+and+Hall%2FCRC%2E)
42. Maxime Peyrard. 2019b. Studying summarization evaluation metrics in the appropriate scoring range. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5093–5100, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Maxime+Peyrard%2E+2019b%2E+Studying+summarization+evaluation+metrics+in+the+appropriate+scoring+range%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+509)
43. Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amapreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R+Bowman%2E+2018%2E+Glue%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+arXiv+preprint+arXiv) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
44. Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned lstm-based natural language generation for spoken dialogue systems. arXiv preprint arXiv:1508.01745. [find](https://scholar.google.com/scholar?q=Tsung%2DHsien+Wen%2C+Milica+Gasic%2C+Nikola+Mrksic%2C+Pei%2DHao+Su%2C+David+Vandyke%2C+and+Steve+Young%2E+2015%2E+Semantically+conditioned+lstm%2Dbased+natural+language+generation+for+spoken+dialogue+systems%2E+arXiv+prepr) [our card](https://md.research.elamur.ai/free/en/papers/wen-2015-semantically-conditioned-lstm-based-natural/)
45. Sam Wiseman, Stuart M. Shieber, and Alexander M. Rush. 2018. Learning neural templates for text generation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Sam+Wiseman%2C+Stuart+M%2E+Shieber%2C+and+Alexander+M%2E+Rush%2E+2018%2E+Learning+neural+templates+for+text+generation%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29) [our card](https://md.research.elamur.ai/free/en/papers/wiseman-2018-learning-neural-templates-text-generation/)
46. Suofei Zhang, Wei Zhao, Xiaofu Wu, and Quan Zhou. 2018. Fast dynamic routing based on weighted kernel density estimation. CoRR, abs/1805.10807. — [arXiv:1805.10807](https://arxiv.org/abs/1805.10807) [find](https://scholar.google.com/scholar?q=Suofei+Zhang%2C+Wei+Zhao%2C+Xiaofu+Wu%2C+and+Quan+Zhou%2E+2018%2E+Fast+dynamic+routing+based+on+weighted+kernel+density+estimation%2E+CoRR%2C+abs%2F1805%2E10807%2E+%E2%80%94+%5BarXiv%3A1805%2E10807%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1805%2E10807%29)
47. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with BERT. CoRR, abs/1904.09675. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2019%2E+Bertscore%3A+Evaluating+text+generation+with+BERT%2E+CoRR%2C+abs%2F1904%2E09675%2E)
48. Wei Zhao, Haiyun Peng, Steffen Eger, Erik Cambria, and Min Yang. 2019. Towards scalable and reliable capsule networks for challenging NLP applications. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1549–1559, Florence, Italy. Association for Comput… [find](https://scholar.google.com/scholar?q=Wei+Zhao%2C+Haiyun+Peng%2C+Steffen+Eger%2C+Erik+Cambria%2C+and+Min+Yang%2E+2019%2E+Towards+scalable+and+reliable+capsule+networks+for+challenging+NLP+applications%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the)
49. Wei Zhao, Jianbo Ye, Min Yang, Zeyang Lei, Suofei Zhang, and Zhou Zhao. 2018. Investigating capsule networks with dynamic routing for text classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wei+Zhao%2C+Jianbo+Ye%2C+Min+Yang%2C+Zeyang+Lei%2C+Suofei+Zhang%2C+and+Zhou+Zhao%2E+2018%2E+Investigating+capsule+networks+with+dynamic+routing+for+text+classification%2E+In+Proceedings+of+the+2016+Conference+on+Empi)