---
title: "Sentence Mover's Similarity — paper by Clark et al. (2019)"
description: "The work offers a cheap, meaning-aware way to evaluate and train multi-sentence generation. 58 references, cited by 73 papers here."
authors: ["Elizabeth Clark", "Asli Celikyilmaz", "Noah A. Smith"]
year: 2019
venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/P19-1264/"
doi: "10.18653/v1/p19-1264"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/clark-2019-sentence-mover-s-similarity-automatic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/clark-2019-sentence-mover-s-similarity-automatic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / Sentence Mover's Similarity: Automatic Evaluation for Multi-Sentence Texts

# Sentence Mover's Similarity: Automatic Evaluation for Multi-Sentence Texts

**Authors:** Elizabeth Clark, [Asli Celikyilmaz](https://md.research.elamur.ai/free/en/authors/asli-celikyilmaz/) and [Noah A. Smith](https://md.research.elamur.ai/free/en/authors/noah-a-smith/) · **Year:** 2019 · **Venue:** [Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2019](https://md.research.elamur.ai/free/en/venues/acl/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P19-1264/) · [PDF](https://aclanthology.org/P19-1264.pdf) · [DOI 10.18653/v1/p19-1264](https://doi.org/10.18653/v1/p19-1264) · [OpenAlex](https://openalex.org/W2947771965)

**Citations:** 135 citations (OpenAlex, 2026-10-14) · [73](https://md.research.elamur.ai/free/en/papers/clark-2019-sentence-mover-s-similarity-automatic/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Popular metrics such as ROUGE and BLEU score generated text by exact word overlap. As a result, they miss paraphrased or reordered content and correlate weakly with human judgments. The embedding-based word mover's distance (WMD) is more flexible, but it treats a document as an unordered bag of words and ignores how ideas group into sentences. Clark, Celikyilmaz and Smith extend this optimal-transport approach to longer texts. Their metric SMS represents each document as a set of sentence vectors, built by averaging word embeddings and weighted by sentence length. A second metric, S+WMS, puts word and sentence vectors into one shared transport problem. The authors compared the metrics with human ratings of machine-generated summaries (about 3.4 sentences) and human-written essays (about 7.5 sentences). Both new metrics agree with people significantly better than ROUGE-L and WMS, with either type-based or contextual embeddings. Used as a reinforcement learning reward for summarization, they produce summaries that human judges prefer. The work offers a cheap, meaning-aware way to evaluate and train multi-sentence generation.

## Key points
- Two metrics are introduced. SMS uses only sentence embeddings. S+WMS mixes word embeddings (weighted by frequency) and sentence embeddings (weighted by length), with weights normalized by 2|A|. WMD is turned into a similarity via exp(−distance), as in WMS.
- Gains over ROUGE-L were larger on generated summaries (avg. 3.4 sentences) than on human essays (avg. 7.5 sentences), but significant in both cases. The choice between type-based and contextual embeddings made no significant difference.
- The RL summarizers used a mixed MLE+RL loss. Human judges compared 100 summaries, with 3 judges per summary, on four criteria: non-redundancy, coherence, focus and overall quality. Models rewarded with WMS, SMS or S+WMS beat ROUGE-L at p \< 0.001.
- SMS achieved the strongest non-redundancy preference over ROUGE-L (69%). However, without word-level matching it can introduce factual slips in its summaries.
- Code and data are released at github.com/eaclark07/sms, which builds on the wmd-relax implementation.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 240 works · 123,537 citations
- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 263 works · 180,581 citations
- [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) — Research institute, Seattle, United States · 219 works · 89,712 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Elizabeth Clark, Asli Celikyilmaz and Noah A. Smith. 2019. Sentence Mover's Similarity: Automatic Evaluation for Multi-Sentence Texts. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{clark2019sentence,
  title = {{Sentence Mover's Similarity: Automatic Evaluation for Multi-Sentence Texts}},
  author = {Elizabeth Clark, Asli Celikyilmaz and Noah A. Smith},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1264},
  url = {https://doi.org/10.18653/v1/p19-1264}
}
```

- [← previous CONAN - COunter NArratives through Nichesourcing: a Multilingual Dataset of Responses to Fight Online Hate Speech](https://md.research.elamur.ai/free/en/papers/chung-2019-conan-counter-narratives-through-nichesourcing/)
- [next → Proactive Human-Machine Conversation with Explicit Conversation Goal](https://md.research.elamur.ai/free/en/papers/wu-2019-proactive-human-machine-conversation-explicit/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

58 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mohammed Alshahrani, Spyridon Samothrakis, and Maria Fasli. 2017. Word mover’s distance for affect detection. 2017 International Conference on the Frontiers and Advances in Data Science. [find](https://scholar.google.com/scholar?q=Mohammed+Alshahrani%2C+Spyridon+Samothrakis%2C+and+Maria+Fasli%2E+2017%2E+Word+mover%E2%80%99s+distance+for+affect+detection%2E+2017+International+Conference+on+the+Frontiers+and+Advances+in+Data+Science%2E)
2. Kubilay Atasu, Thomas P. Parnell, Celestine Dünner, Manolis Sifalakis, Haralampos Pozidis, Vasileios Vasileiadis, Michail Vlachos, Cesar Berrospi, and Abdel Labbi. 2017. Linear-complexity relaxed word mover’s distance with GPU acceleration. In IEEE International Conference on Big Data. [find](https://scholar.google.com/scholar?q=Kubilay+Atasu%2C+Thomas+P%2E+Parnell%2C+Celestine+Du%CC%88nner%2C+Manolis+Sifalakis%2C+Haralampos+Pozidis%2C+Vasileios+Vasileiadis%2C+Michail+Vlachos%2C+Cesar+Berrospi%2C+and+Abdel+Labbi%2E+2017%2E+Linear%2Dcomplexity+relaxed+wor)
3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). In ICLR.
4. Georgios Balikas, Charlotte Laclau, Ievgen Redko, and Massih-Reza Amini. 2018. Cross-lingual document retrieval using regularized Wasserstein distance. CoRR, abs/1805.04437. — [arXiv:1805.04437](https://arxiv.org/abs/1805.04437) [find](https://scholar.google.com/scholar?q=Georgios+Balikas%2C+Charlotte+Laclau%2C+Ievgen+Redko%2C+and+Massih%2DReza+Amini%2E+2018%2E+Cross%2Dlingual+document+retrieval+using+regularized+Wasserstein+distance%2E+CoRR%2C+abs%2F1805%2E04437%2E+%E2%80%94+%5BarXiv%3A1805%2E04437%5D%28https)
5. Satanjeev Banerjee and Alon Lavie. 2005. [METEOR: An automatic metric for MT evaluation with improved correlation with human judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/). In IEE-valuation@ACL.
6. Regina Barzilay and Mirala Lapata. 2008. [Modeling local coherence: An entity-based approach](https://md.research.elamur.ai/free/en/papers/barzilay-2008-modeling-local-coherence-entity-based/). In Computational Linguistics.
7. Anja Belz and Ehud Reiter. 2006. Comparing automatic and human evaluation of NLG systems. In EACL. [find](https://scholar.google.com/scholar?q=Anja+Belz+and+Ehud+Reiter%2E+2006%2E+Comparing+automatic+and+human+evaluation+of+NLG+systems%2E+In+EACL%2E)
8. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Second International Conference on Human Language Technology Research. [find](https://scholar.google.com/scholar?q=George+Doddington%2E+2002%2E+Automatic+evaluation+of+machine+translation+quality+using+n%2Dgram+co%2Doccurrence+statistics%2E+In+Second+International+Conference+on+Human+Language+Technology+Research%2E)
9. Ondrej Dusek, Jekaterina Novikova, and Verena Rieser. 2019. Evaluating the state-of-the-art of end-to-end natural language generation: The E2E NLG challenge. In Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ondrej+Dusek%2C+Jekaterina+Novikova%2C+and+Verena+Rieser%2E+2019%2E+Evaluating+the+state%2Dof%2Dthe%2Dart+of+end%2Dto%2Dend+natural+language+generation%3A+The+E2E+NLG+challenge%2E+In+Computational+Linguistics%2E)
10. Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. 2018. [Classical structured prediction losses for sequence to sequence learning](https://md.research.elamur.ai/free/en/papers/edunov-2018-classical-structured-prediction-losses-sequence/). In NAACL-HLT.
11. Desmond Elliott and Frank Keller. 2014. Comparing automatic evaluation measures for image description. In ACL. [find](https://scholar.google.com/scholar?q=Desmond+Elliott+and+Frank+Keller%2E+2014%2E+Comparing+automatic+evaluation+measures+for+image+description%2E+In+ACL%2E)
12. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2018. [AllenNLP: A deep semantic natural language processing platform](https://md.research.elamur.ai/free/en/papers/gardner-2018-allennlp-deep-semantic-natural-language/). In ACL workshop for NLP Open Source Software.
13. Yoav Goldberg, Graeme Hirst, Yang Liu, and Meng Zhang. 2018. Neural network methods for natural language processing. Computational Linguistics, 44(1). [find](https://scholar.google.com/scholar?q=Yoav+Goldberg%2C+Graeme+Hirst%2C+Yang+Liu%2C+and+Meng+Zhang%2E+2018%2E+Neural+network+methods+for+natural+language+processing%2E+Computational+Linguistics%2C+44%281%29%2E)
14. Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying human and statistical evaluation for natural language generation. In NAACL. [find](https://scholar.google.com/scholar?q=Tatsunori+B%2E+Hashimoto%2C+Hugh+Zhang%2C+and+Percy+Liang%2E+2019%2E+Unifying+human+and+statistical+evaluation+for+natural+language+generation%2E+In+NAACL%2E)
15. Asli Celikyilmaz, Antoine Bosselut, Xiadong He, and Yejin Choi. 2018. [Deep communicating agents for abstractive summarization](https://md.research.elamur.ai/free/en/papers/celikyilmaz-2018-deep-communicating-agents-abstractive-summarization/). In NAACL.
16. Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. [Teaching machines to read and comprehend](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/). In NeurIPS.
17. Arun Tejasvi Chaganty, Stephen Mussmann, and Percy Liang. 2018. The price of debiasing automatic metrics in natural language evaluation. In ACL. [find](https://scholar.google.com/scholar?q=Arun+Tejasvi+Chaganty%2C+Stephen+Mussmann%2C+and+Percy+Liang%2E+2018%2E+The+price+of+debiasing+automatic+metrics+in+natural+language+evaluation%2E+In+ACL%2E)
18. Gao Huang, Chuan Guo, Matt J. Kusner, Yu Sun, Fei Sha, and Kilian Q. Weinberger. 2016. Supervised word mover’s distance. In NeurIPS. [find](https://scholar.google.com/scholar?q=Gao+Huang%2C+Chuan+Guo%2C+Matt+J%2E+Kusner%2C+Yu+Sun%2C+Fei+Sha%2C+and+Kilian+Q%2E+Weinberger%2E+2016%2E+Supervised+word+mover%E2%80%99s+distance%2E+In+NeurIPS%2E)
19. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, and Phillipp Koehn. 2013. [One billion word benchmark for measuring progress in statistical language modeling](https://md.research.elamur.ai/free/en/papers/chelba-2013-one-billion-word-benchmark-measuring/). In INTERSPEECH.
20. Natasha Jaques, Shixiang Gu, Dxmitry Bahdanau, Jose Miguel Hernandez-Lobato, Richard E. Turner, and Douglas Eck. 2017. Counterfactual multi-agent policy gradients. In ICML. [find](https://scholar.google.com/scholar?q=Natasha+Jaques%2C+Shixiang+Gu%2C+Dxmitry+Bahdanau%2C+Jose+Miguel+Hernandez%2DLobato%2C+Richard+E%2E+Turner%2C+and+Douglas+Eck%2E+2017%2E+Counterfactual+multi%2Dagent+policy+gradients%2E+In+ICML%2E)
21. Liqun Chen, Yizhe Zhang, Ruiyi Zhang, Chenyang Tao, Zhe Gan, Haichao Zhang, Bai Li, Dinghan Shen, Changyou Chen, and Lawrence Carin. 2019. Improving sequence-to-sequence learning via optimal transport. In ICLR. [find](https://scholar.google.com/scholar?q=Liqun+Chen%2C+Yizhe+Zhang%2C+Ruiyi+Zhang%2C+Chenyang+Tao%2C+Zhe+Gan%2C+Haichao+Zhang%2C+Bai+Li%2C+Dinghan+Shen%2C+Changyou+Chen%2C+and+Lawrence+Carin%2E+2019%2E+Improving+sequence%2Dto%2Dsequence+learning+via+optimal+transport)
22. Mert Kilickaya, Aykut Erdem, Nazli Ikizler-Cinbis, and Erkut Erdem. 2017. Re-evaluating automatic metrics for image captioning. In EACL. [find](https://scholar.google.com/scholar?q=Mert+Kilickaya%2C+Aykut+Erdem%2C+Nazli+Ikizler%2DCinbis%2C+and+Erkut+Erdem%2E+2017%2E+Re%2Devaluating+automatic+metrics+for+image+captioning%2E+In+EACL%2E)
23. Jianpeng Cheng and Mirella Lapata. 2016. [Neural summarization by extracting sentences and words](https://md.research.elamur.ai/free/en/papers/cheng-2016-neural-summarization-extracting-sentences-words/). In ACL.
24. Alexis Conneau and Douwe Kiela. 2018. [SentEval: An evaluation toolkit for universal sentence representations](https://md.research.elamur.ai/free/en/papers/conneau-2018-senteval-evaluation-toolkit-universal-sentence/). In LREC.
25. Andrew M. Dai, Christopher Olah, and Quoc V. Le. 2015. Document embedding with paragraph vectors. In NeurIPS Deep Learning Workshop. [find](https://scholar.google.com/scholar?q=Andrew+M%2E+Dai%2C+Christopher+Olah%2C+and+Quoc+V%2E+Le%2E+2015%2E+Document+embedding+with+paragraph+vectors%2E+In+NeurIPS+Deep+Learning+Workshop%2E)
26. Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, and Kilian Q. Weinberger. 2015. From word embeddings to document distances. ICLR, 37. [find](https://scholar.google.com/scholar?q=Matt+J%2E+Kusner%2C+Yu+Sun%2C+Nicholas+I%2E+Kolkin%2C+and+Kilian+Q%2E+Weinberger%2E+2015%2E+From+word+embeddings+to+document+distances%2E+ICLR%2C+37%2E)
27. Mirella Lapata and Regina Barzilay. 2005. Automatic evaluation of text coherence: Models and representations. In IJCAI. [find](https://scholar.google.com/scholar?q=Mirella+Lapata+and+Regina+Barzilay%2E+2005%2E+Automatic+evaluation+of+text+coherence%3A+Models+and+representations%2E+In+IJCAI%2E)
28. Alon Lavie and Abhaya Agarwal. 2007. METEOR: An automatic metric for MT evaluation with high levels of with human judgments. In Second Workshop on Statistical Machine Translation. [find](https://scholar.google.com/scholar?q=Alon+Lavie+and+Abhaya+Agarwal%2E+2007%2E+METEOR%3A+An+automatic+metric+for+MT+evaluation+with+high+levels+of+with+human+judgments%2E+In+Second+Workshop+on+Statistical+Machine+Translation%2E)
29. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In ACL.
30. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. [How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation](https://md.research.elamur.ai/free/en/papers/liu-2016-how-not-evaluate-your-dialogue/). In EMNLP.
31. Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. [Advances in pre-training distributed word representations](https://md.research.elamur.ai/free/en/papers/mikolov-2017-advances-pre-training-distributed-word/). In LREC.
32. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. [SummaRuNNer: A recurrent neural network based sequence model for extractive summarization of documents](https://md.research.elamur.ai/free/en/papers/nallapati-2016-summarunner-recurrent-neural-network-based/). In AAAI.
33. Ramesh Nallapati, Bowen Zhou, Cı́cero Nogueira dos Santos, Caglar Gülehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In CoNLL. [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Bowen+Zhou%2C+C%C4%B1%CC%81cero+Nogueira+dos+Santos%2C+Caglar+Gu%CC%88lehre%2C+and+Bing+Xiang%2E+2016%2E+Abstractive+text+summarization+using+sequence%2Dto%2Dsequence+RNNs+and+beyond%2E+In+CoNLL%2E)
34. Jekaterina Novikova, Ondrej Dusek, Amanda Cercas Curry, and Verena Rieser. 2017. [Why we need new evaluation metrics for NLG](https://md.research.elamur.ai/free/en/papers/novikova-2017-why-we-need-new-evaluation/). In EMNLP.
35. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In ACL.
36. Ramakanth Pasunuru and Mohit Bansal. 2017. Reinforced video captioning with entailment rewards. In EMNLP. [find](https://scholar.google.com/scholar?q=Ramakanth+Pasunuru+and+Mohit+Bansal%2E+2017%2E+Reinforced+video+captioning+with+entailment+rewards%2E+In+EMNLP%2E)
37. Romain Paulus, Caiming Xiong, and Richard Socher. 2018. [A deep reinforced model for abstractive summarization](https://md.research.elamur.ai/free/en/papers/paulus-2017-deep-reinforced-model-abstractive-summarization/). In ICLR.
38. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. [Glove: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In EMNLP.
39. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). In NAACL.
40. Christopher Purdy, Xinyu Wang, Larry He, and Mark O. Riedl. 2018. Predicting generated story quality with quantitative measures. In AIIDE. [find](https://scholar.google.com/scholar?q=Christopher+Purdy%2C+Xinyu+Wang%2C+Larry+He%2C+and+Mark+O%2E+Riedl%2E+2018%2E+Predicting+generated+story+quality+with+quantitative+measures%2E+In+AIIDE%2E)
41. MarcAurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. [Sequence level training with recurrent neural networks](https://md.research.elamur.ai/free/en/papers/ranzato-2015-sequence-level-training-recurrent-neural/). In ICLR.
42. Ehud Reiter. 2011. Task-based evaluation of NLG systems: Control vs real-world context. In UC-NLG+Eval. [find](https://scholar.google.com/scholar?q=Ehud+Reiter%2E+2011%2E+Task%2Dbased+evaluation+of+NLG+systems%3A+Control+vs+real%2Dworld+context%2E+In+UC%2DNLG%2BEval%2E)
43. Ehud Reiter and Anja Belz. 2006. GENEVAL: A proposal for shared-task evaluation in NLG. In INLG. [find](https://scholar.google.com/scholar?q=Ehud+Reiter+and+Anja+Belz%2E+2006%2E+GENEVAL%3A+A+proposal+for+shared%2Dtask+evaluation+in+NLG%2E+In+INLG%2E)
44. Ehud Reiter and Anja Belz. 2009. An investigation into the validity of some metrics for automatically evaluating natural language generation systems. In CVPR. [find](https://scholar.google.com/scholar?q=Ehud+Reiter+and+Anja+Belz%2E+2009%2E+An+investigation+into+the+validity+of+some+metrics+for+automatically+evaluating+natural+language+generation+systems%2E+In+CVPR%2E)
45. Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. 2016. [Self-critical sequence training for image captioning](https://md.research.elamur.ai/free/en/papers/rennie-2016-self-critical-sequence-training-image/). In CVPR.
46. Tomasi C. Rubner, Y. and L. J. Guibas. 1998. A metric for distributions with applications to image databases. In IEEE. [find](https://scholar.google.com/scholar?q=Tomasi+C%2E+Rubner%2C+Y%2E+and+L%2E+J%2E+Guibas%2E+1998%2E+A+metric+for+distributions+with+applications+to+image+databases%2E+In+IEEE%2E)
47. Manasvi Sagarkar, John Wieting, Lifu Tu, and Kevin Gimpel. 2018. Quality signals in generated stories. In *SEM 2018. [find](https://scholar.google.com/scholar?q=Manasvi+Sagarkar%2C+John+Wieting%2C+Lifu+Tu%2C+and+Kevin+Gimpel%2E+2018%2E+Quality+signals+in+generated+stories%2E+In+%2ASEM+2018%2E)
48. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In ACL. [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Peter+J%2E+Liu%2C+and+Christopher+D%2E+Manning%2E+2017%2E+Get+to+the+point%3A+Summarization+with+pointer%2Dgenerator+networks%2E+In+ACL%2E)
49. Octavia-Maria Sulea. 2017. [Recognizing textual entailment in Twitter using word embeddings](https://md.research.elamur.ai/free/en/papers/sulea-2017-recognizing-textual-entailment-twitter-word/). In 2nd Workshop on Evaluating Vector-Space Representations for NLP.
50. Tsegaye Misikir Tashu and Tomás Horváth. 2018. Pair-Wise: Automatic essay evaluation using word mover’s distance. In CSEDU. [find](https://scholar.google.com/scholar?q=Tsegaye+Misikir+Tashu+and+Toma%CC%81s+Horva%CC%81th%2E+2018%2E+Pair%2DWise%3A+Automatic+essay+evaluation+using+word+mover%E2%80%99s+distance%2E+In+CSEDU%2E)
51. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. [CIDEr: Consensus-based image description evaluation](https://md.research.elamur.ai/free/en/papers/vedantam-2014-cider-consensus-based-image-description/). In CVPR.
52. Evan J. Williams. 1959. ume 14. Wiley. [find](https://scholar.google.com/scholar?q=Evan+J%2E+Williams%2E+1959%2E+ume+14%2E+Wiley%2E)
53. Regression Analysis, vol-Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. In Machine Learning. [find](https://scholar.google.com/scholar?q=Regression+Analysis%2C+vol%2DRonald+J%2E+Williams%2E+1992%2E+Simple+statistical+gradient%2Dfollowing+algorithms+for+connectionist+reinforcement+learning%2E+In+Machine+Learning%2E)
54. Lingfei Wu, Ian En-Hsu Yen, Kun Xu, Fangli Xu, Avinash Balakrishnan, Pin-Yu Chen, Pradeep Ravikumar, and Michael J. Witbrock. 2018. Word movers embedding: From word2vec to document embedding. In EMNLP. [find](https://scholar.google.com/scholar?q=Lingfei+Wu%2C+Ian+En%2DHsu+Yen%2C+Kun+Xu%2C+Fangli+Xu%2C+Avinash+Balakrishnan%2C+Pin%2DYu+Chen%2C+Pradeep+Ravikumar%2C+and+Michael+J%2E+Witbrock%2E+2018%2E+Word+movers+embedding%3A+From+word2vec+to+document+embedding%2E+In+EMNLP)
55. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. ArXiv:1609.08144. [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+et+al%2E+2016%2E+Google%E2%80%99s+neural+machine+translation+system%3A+Bridg)
56. Hao Zhang and Jie Wang. 2018. Semantic WordRank: Generating finer single-document summarizations. In IDEAL. [find](https://scholar.google.com/scholar?q=Hao+Zhang+and+Jie+Wang%2E+2018%2E+Semantic+WordRank%3A+Generating+finer+single%2Ddocument+summarizations%2E+In+IDEAL%2E)
57. Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017. Earth mover’s distance minimization for unsupervised bilingual lexicon induction. In EMNLP. [find](https://scholar.google.com/scholar?q=Meng+Zhang%2C+Yang+Liu%2C+Huanbo+Luan%2C+and+Maosong+Sun%2E+2017%2E+Earth+mover%E2%80%99s+distance+minimization+for+unsupervised+bilingual+lexicon+induction%2E+In+EMNLP%2E)
58. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. BERTScore: Evaluating text generation with BERT. CoRR, abs/1904.09675. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2019%2E+BERTScore%3A+Evaluating+text+generation+with+BERT%2E+CoRR%2C+abs%2F1904%2E09675%2E)

*Updated 24 October 2026*