---
title: "Evaluating Style Transfer for Text — Mir 2019"
description: "Progress in text style transfer is hampered by the lack of shared evaluation standards."
authors: ["Remi Mir", "Bjarke Felbo", "Nick Obradovich", "Iyad Rahwan"]
year: 2019
venue: "Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)"
source: "https://aclanthology.org/N19-1049/"
doi: "10.18653/v1/n19-1049"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mir-2019-evaluating-style-transfer-text/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mir-2019-evaluating-style-transfer-text/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/2/) / Evaluating Style Transfer for Text

# Evaluating Style Transfer for Text

**Authors:** Remi Mir, [Bjarke Felbo](https://md.research.elamur.ai/free/en/authors/bjarke-felbo/), Nick Obradovich, [Iyad Rahwan](https://md.research.elamur.ai/free/en/authors/iyad-rahwan/) · **Year:** 2019 · **Venue:** Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/N19-1049/) · [PDF](https://aclanthology.org/N19-1049.pdf) · [DOI 10.18653/v1/n19-1049](https://doi.org/10.18653/v1/n19-1049) · [OpenAlex](https://openalex.org/W2927085091)

**Citations:** 72 citations (openalex-snapshot · 2026-10-14) · [82](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Progress in text style transfer is hampered by the lack of shared evaluation standards. Papers use different human rating schemes and automated metrics, so results are hard to compare. Mir, Felbo, Obradovich and Rahwan (MIT) define three aspects to measure: style transfer intensity, content preservation and naturalness. They then test evaluation protocols on a Yelp sentiment dataset, using outputs from three models: CAAE, ARAE and DAR.

For human evaluation, they propose comparative judgments instead of absolute ones. Raters see the original and output side by side, style words are masked out for content ratings, and raters pick which of two texts looks more natural. For automated evaluation, they suggest three metrics:
- direction-corrected Earth Mover's Distance between style distributions,
- Word Mover's Distance on style-masked texts,
- adversarial classifiers that try to separate inputs from outputs.

These metrics agree with human judgment better than older measures such as BLEU, target-style accuracy and perplexity. The models also show tradeoffs between the three aspects, so they should be compared at specific points on tradeoff plots. The released code helps make evaluations reproducible.

## Key points
- Three evaluation aspects are defined: style transfer intensity, content preservation and naturalness. Experiments use Yelp sentiment data and outputs of CAAE, ARAE and DAR.
- A style lexicon is built automatically for each dataset. It takes the features with the largest absolute weights from a logistic regression style classifier and favors precision over recall to avoid masking content words.
- Masking style words makes human content-preservation ratings more reliable. Relative scoring of style intensity, however, did not improve inter-rater reliability over absolute scoring.
- Sentence-level perplexity of the output alone is not a meaningful naturalness measure. BLEU unfairly penalizes the word changes that style transfer requires.
- The evaluation software is available at github.com/passeul/style-transfer-model-evaluation.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

82 papers of our library cite it (of the 140884 whose bibliographies we read); 8 of them have a page here, 74 not yet.

- [Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer](https://md.research.elamur.ai/free/en/papers/briakou-2021-evaluating-evaluation-metrics-style-transfer/)
- [RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/deng-2022-rlprompt-optimizing-discrete-text-prompts/)
- [Reformulating Unsupervised Style Transfer as Paraphrase Generation](https://md.research.elamur.ai/free/en/papers/krishna-2020-reformulating-unsupervised-style-transfer-as/)
- [ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfer](https://md.research.elamur.ai/free/en/papers/liu-2023-chatgpt-vs-human-authored-text/)
- [LENS: A Learnable Evaluation Metric for Text Simplification](https://md.research.elamur.ai/free/en/papers/maddela-2023-lens-learnable-evaluation-metric-text/)
- [A Recipe For Arbitrary Text Style Transfer with Large Language Models](https://md.research.elamur.ai/free/en/papers/reif-2021-recipe-arbitrary-text-style-transfer/)
- [AMR-TST: Abstract Meaning Representation-based Text Style Transfer](https://md.research.elamur.ai/free/en/papers/shi-2023-amr-tst-abstract-meaning-representation/)
- [“Transforming” Delete, Retrieve, Generate Approach for Controlled Text Style Transfer](https://md.research.elamur.ai/free/en/papers/sudhakar-2019-transforming-delete-retrieve-generate-approach/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mir2019evaluating,
  title = {{Evaluating Style Transfer for Text}},
  author = {Mir, Remi and Felbo, Bjarke and Obradovich, Nick and Rahwan, Iyad},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1049},
  url = {https://doi.org/10.18653/v1/n19-1049}
}
```

- [← previous Evaluating Gender Bias in Machine Translation](https://md.research.elamur.ai/free/en/papers/stanovsky-2019-evaluating-gender-bias-machine-translation/)
- [next → Evaluating the Underlying Gender Bias in Contextualized Word Embeddings](https://md.research.elamur.ai/free/en/papers/basta-2019-evaluating-underlying-gender-bias-contextualized/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/2/)

## References

16 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Pro-Neil Stewart, Gordon DA Brown, and Nick Chater. 2005. Absolute identification by relative judgment. Psychological review, 112(4):881–911. [find](https://scholar.google.com/scholar?q=Satanjeev+Banerjee+and+Alon+Lavie%2E+2005%2E+Meteor%3A+An+automatic+metric+for+mt+evaluation+with+improved+correlation+with+human+judgments%2E+In+Pro%2DNeil+Stewart%2C+Gordon+DA+Brown%2C+and+Nick+Chater%2E+2005%2E+Abso) [our card](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/)
2. Dongjun Lee. 2018. text-cnn. [https://github.com/DongjunLee/text-cnn-tensorflow](https://github.com/DongjunLee/text-cnn-tensorflow). [↗](https://github.com/DongjunLee/text-cnn-tensorflow)
3. Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1865–1874. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Juncen+Li%2C+Robin+Jia%2C+He+He%2C+and+Percy+Liang%2E+2018%2E+Delete%2C+retrieve%2C+generate%3A+a+simple+approach+to+sentiment+and+style+transfer%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+o) [our card](https://md.research.elamur.ai/free/en/papers/li-2018-delete-retrieve-generate-simple-approach/)
4. Carlo Strapparava and Alessandro Valitutti. 2004. Wordnet affect: an affective extension of wordnet. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04). European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Carlo+Strapparava+and+Alessandro+Valitutti%2E+2004%2E+Wordnet+affect%3A+an+affective+extension+of+wordnet%2E+In+Proceedings+of+the+Fourth+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%E2%80%9904) [our card](https://md.research.elamur.ai/free/en/papers/strapparava-2004-wordnet-affect-affective-extension-wordnet/)
5. Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th International Conference on Machine Learning, ICML ’08, pages 1096–1103. ACM. [find](https://scholar.google.com/scholar?q=Pascal+Vincent%2C+Hugo+Larochelle%2C+Yoshua+Bengio%2C+and+Pierre%2DAntoine+Manzagol%2E+2008%2E+Extracting+and+composing+robust+features+with+denoising+autoencoders%2E+In+Proceedings+of+the+25th+International+Confer)
6. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems -Volume 2, NIPS’13, pages 3111–3119, USA. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Ilya+Sutskever%2C+Kai+Chen%2C+Greg+Corrado%2C+and+Jeffrey+Dean%2E+2013%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+In+Proceedings+of+the+26th+International+Con)
7. Steve J. Young, D. Kershaw, J. Odell, D. Ollason, V. Valtchev, and P. Woodland. 2006. The HTK Book Version 3.4. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Steve+J%2E+Young%2C+D%2E+Kershaw%2C+J%2E+Odell%2C+D%2E+Ollason%2C+V%2E+Valtchev%2C+and+P%2E+Woodland%2E+2006%2E+The+HTK+Book+Version+3%2E4%2E+Cambridge+University+Press%2E)
8. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 311–318. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+Bleu%3A+A+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+Annual+Meeting+on+Association+for+Computat) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
9. Junbo Zhao, Yoon Kim, Kelly Zhang, Alexander Rush, and Yann LeCun. 2018. Adversarially regularized autoencoders. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 5902–5911. [find](https://scholar.google.com/scholar?q=Junbo+Zhao%2C+Yoon+Kim%2C+Kelly+Zhang%2C+Alexander+Rush%2C+and+Yann+LeCun%2E+2018%2E+Adversarially+regularized+autoencoders%2E+In+Proceedings+of+the+35th+International+Conference+on+Machine+Learning%2C+volume+80+of+P)
10. Ofir Pele and Michael Werman. 2009. Fast and robust earth mover’s distances. pages 460–467. IEEE. [find](https://scholar.google.com/scholar?q=Ofir+Pele+and+Michael+Werman%2E+2009%2E+Fast+and+robust+earth+mover%E2%80%99s+distances%2E+pages+460%E2%80%93467%2E+IEEE%2E)
11. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+Manning%2E+2014%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+Proceedings+of+the+2014+Conference+on+Empirical+Methods+in+Natural+Language+Process)
12. Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. 2018. Style transfer through back-translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 866–876. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shrimai+Prabhumoye%2C+Yulia+Tsvetkov%2C+Ruslan+Salakhutdinov%2C+and+Alan+W+Black%2E+2018%2E+Style+transfer+through+back%2Dtranslation%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computationa) [our card](https://md.research.elamur.ai/free/en/papers/prabhumoye-2018-style-transfer-through-back-translation/)
13. Yossi Rubner, Carlo Tomasi, and Leonidas J. Guibas. 1998. A metric for distributions with applications to image databases. In Sixth International Conference on Computer Vision, pages 59–66. IEEE. [find](https://scholar.google.com/scholar?q=Yossi+Rubner%2C+Carlo+Tomasi%2C+and+Leonidas+J%2E+Guibas%2E+1998%2E+A+metric+for+distributions+with+applications+to+image+databases%2E+In+Sixth+International+Conference+on+Computer+Vision%2C+pages+59%E2%80%9366%2E+IEEE%2E)
14. Vasile Rus and Mihai Lintean. 2012. A comparison of greedy and optimal assessment of natural language student input using word-to-word similarity metrics. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pages 157– 162. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Vasile+Rus+and+Mihai+Lintean%2E+2012%2E+A+comparison+of+greedy+and+optimal+assessment+of+natural+language+student+input+using+word%2Dto%2Dword+similarity+metrics%2E+In+Proceedings+of+the+Seventh+Workshop+on+Bui)
15. Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017. Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation. CoRR, abs/1706.09799. [↗](https://arxiv.org/abs/1706.09799)
16. Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In Advances in Neural Information Processing Systems 30, pages 6830–6841. [find](https://scholar.google.com/scholar?q=Tianxiao+Shen%2C+Tao+Lei%2C+Regina+Barzilay%2C+and+Tommi+Jaakkola%2E+2017%2E+Style+transfer+from+non%2Dparallel+text+by+cross%2Dalignment%2E+In+Advances+in+Neural+Information+Processing+Systems+30%2C+pages+6830%E2%80%936841%2E) [our card](https://md.research.elamur.ai/free/en/papers/shen-2017-style-transfer-non-parallel-text/)