---
title: "Reinforced Rewards Framework for Text Style Transfer (2020)"
description: "The paper tackles parallel text style transfer: rewriting a sentence into another style while keeping its… 23 references, cited by 16 papers here."
authors: ["Abhilasha Sancheti", "Kundan Krishna", "Balaji Vasan Srinivasan", "Anandhavelu Natarajan"]
year: 2020
venue: "arXiv:2005.05256"
source: "https://arxiv.org/abs/2005.05256"
doi: "10.48550/arxiv.2005.05256"
arxiv: "2005.05256"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sancheti-2020-reinforced-rewards-framework-text-style/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sancheti-2020-reinforced-rewards-framework-text-style/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/7/) / Reinforced Rewards Framework for Text Style Transfer

# Reinforced Rewards Framework for Text Style Transfer

**Authors:** Abhilasha Sancheti, Kundan Krishna, Balaji Vasan Srinivasan and Anandhavelu Natarajan · **Year:** 2020 · **Venue:** arXiv:2005.05256 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2005.05256) · [PDF](https://arxiv.org/pdf/2005.05256) · [DOI 10.48550/arxiv.2005.05256](https://doi.org/10.48550/arxiv.2005.05256) · [arXiv 2005.05256](https://arxiv.org/abs/2005.05256) · [OpenAlex](https://openalex.org/W3022906046)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [16](https://md.research.elamur.ai/free/en/papers/sancheti-2020-reinforced-rewards-framework-text-style/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper tackles parallel text style transfer: rewriting a sentence into another style while keeping its meaning. The authors note a mismatch. Systems are judged on sentence-level, non-differentiable measures (content preservation and transfer strength), yet they are trained with token-level cross-entropy. To close this gap they add reinforcement-learning rewards to a copy-enriched sequence-to-sequence model, an LSTM encoder-decoder with attention and a pointer mechanism.

Two reward sources are added:
- **Content reward:** a BLEU score fed through Self-Critical Sequence Training (SCST).
- **Style reward:** a CNN style classifier whose target-style probability acts as a cooperative discriminator.

The experiments cover three transfer tasks, each run in both directions:
- formal ↔ informal, on an existing parallel corpus
- high ↔ low excitement, on a corpus the authors curated themselves
- modern ↔ Shakespearean English

Under both automatic and human evaluation, the reinforced model (TS→CP) beats a Transformer, Cross-Aligned and CopyNMT. The work matters because it shows that metrics which cannot be differentiated can still steer generation directly, without explicitly separating style from content.

## Key points
- The base model is copy-enriched seq2seq. Its output distribution mixes an RNN vocabulary distribution with a pointer distribution over source words, weighted by a learned gate δ.
- Content preservation is rewarded via SCST:
  - A multinomially sampled output is compared with a greedy baseline output, both scored by BLEU.
  - The reward gap weights the log-likelihood of the sample, so the metric never needs a gradient.
- Transfer strength is rewarded through a Kim-style CNN classifier trained on the training data. A direction-dependent term such as −log(1 − s(y′)) for high→low is added to the cross-entropy loss.
- The tasks are formality, excitement (a self-curated corpus) and the roughly 30K-pair Shakespeare–modern parallel corpus. Baselines are Transformer, Cross-Aligned and CopyNMT.
- Qualitative outputs show the model learns both lexical and structural shifts:
  - "would" and "inform" when moving to formal style
  - "u", "gonna" and "mama" when moving to informal style
  - "amazing" vs "okay" for excitement
  - "thou" and "hither" for Shakespearean English

## Interesting ideas
- Using a greedy decode as the baseline for a sampled decode yields a cheap, variance-reduced policy gradient for any sentence-level score, however discrete.
- When a property has no formal metric, a separately trained classifier's probability can serve as a stand-in reward that cooperates with, rather than fights, the generator.
- Rewards are kept as multiplicative weights on log-likelihood next to the ordinary cross-entropy, so the model stays anchored to references while optimizing the target metrics.
- A copy/pointer gate lets the model reuse factual source words untouched and edit only the style-bearing ones.
- Evaluating one framework on several unrelated style axes in both directions tests whether the method generalizes beyond a single task.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Parser or generator training could be rewarded on whole-tree scores like LAS, or on downstream answer correctness, with the greedy output as the self-critical baseline instead of per-token loss.
- **I2** (back): Generated text could be scored by a register or style classifier so the graph→text step produces wording in a requested style, while explainability is kept by logging the classifier's score.
- **I3** (learn): Mixing a supervised loss on silver trees with a reward term from witness-parser agreement could stop reward-only training from drifting away from valid UD structure.
- **I4** (back): In the text → graph → text compressor, entity names and numbers from the world model could be copied verbatim while only function words and phrasing are regenerated, which keeps facts faithful.
- **I5** (answer): Reader components could be checked symmetrically, for example on paraphrased modern vs archaic fable texts, to see whether answers survive changes of register.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 365 works · 158,048 citations
- [University of Maryland, College Park](https://md.research.elamur.ai/free/en/universities/university-of-maryland-college-park/) — University, College Park, United States · 83 works · 23,784 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

## Cite

Cite as: Abhilasha Sancheti, Kundan Krishna, Balaji Vasan Srinivasan and Anandhavelu Natarajan. 2020. Reinforced Rewards Framework for Text Style Transfer. arXiv:2005.05256.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{sancheti2020reinforced,
  title = {{Reinforced Rewards Framework for Text Style Transfer}},
  author = {Abhilasha Sancheti, Kundan Krishna, Balaji Vasan Srinivasan and Anandhavelu Natarajan},
  year = {2020},
  doi = {10.48550/arxiv.2005.05256},
  eprint = {2005.05256},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2005.05256}
}
```

- [← previous MASRI-HEADSET: A Maltese Corpus for Speech Recognition](https://md.research.elamur.ai/free/en/papers/mena-2020-masri-headset-maltese-corpus-speech/)
- [next → A Lexicon-Based Approach for Detecting Hedges in Informal Text](https://md.research.elamur.ai/free/en/papers/islam-2020-lexicon-based-approach-detecting-hedges/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/7/)

## References

23 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bahdanau, D., Cho, K., Bengio, Y.: [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). arXiv preprint arXiv:1409.0473 (2014) [↗](https://arxiv.org/abs/1409.0473)
2. Bertram, D.: Likert scales. Retrieved November 2, 2013 (2007) [find](https://scholar.google.com/scholar?q=Bertram%2C+D%2E%3A+Likert+scales%2E+Retrieved+November+2%2C+2013+%282007%29)
3. Dai, N., Liang, J., Qiu, X., Huang, X.: [Style transformer: Unpaired text style transfer without disentangled latent representation](https://md.research.elamur.ai/free/en/papers/dai-2019-style-transformer-unpaired-text-style/). arXiv preprint arXiv:1905.05621 (2019) [↗](https://arxiv.org/abs/1905.05621)
4. Fu, Z., Tan, X., Peng, N., Zhao, D., Yan, R.: [Style transfer in text: Exploration and evaluation](https://md.research.elamur.ai/free/en/papers/fu-2017-style-transfer-text-exploration-evaluation/). arXiv preprint arXiv:1711.06861 (2017) [↗](https://arxiv.org/abs/1711.06861)
5. Gong, H., Bhat, S., Wu, L., Xiong, J., Hwu, W.m.: Reinforcement learning based text style transfer without parallel training corpus. arXiv preprint arXiv:1903.10671 (2019) [↗](https://arxiv.org/abs/1903.10671)
6. Holtzman, A., Buys, J., Forbes, M., Bosselut, A., Golub, D., Choi, Y.: [Learning to write with cooperative discriminators](https://md.research.elamur.ai/free/en/papers/holtzman-2018-learning-write-cooperative-discriminators/). In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1638–1649. Association for Computational Linguistics (2018), http://aclweb.org/anthology/P18-1152 [↗](https://aclanthology.org/P18-1152/)
7. Jhamtani, H., Gangal, V., Hovy, E., Nyberg, E.: [Shakespearizing modern language using copy-enriched sequence to sequence models](https://md.research.elamur.ai/free/en/papers/jhamtani-2017-shakespearizing-modern-language-copy-enriched/). In: Proceedings of the Workshop on Stylistic Variation. pp. 10–19. Association for Computational Linguistics (2017). https://doi.org/10.18653/v1/W17-4902, http://aclweb.org/anthology/W17-4902 [↗](https://doi.org/10.18653/v1/w17-4902)
8. Keneshloo, Y., Shi, T., Reddy, C.K., Ramakrishnan, N.: Deep reinforcement learning for sequence to sequence models. arXiv preprint arXiv:1805.09461 (2018) [↗](https://arxiv.org/abs/1805.09461)
9. Kim, Y.: [Convolutional neural networks for sentence classification](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/). arXiv preprint arXiv:1408.5882 (2014) [↗](https://arxiv.org/abs/1408.5882)
10. Li, J., Jia, R., He, H., Liang, P.: [Delete, retrieve, generate: A simple approach to sentiment and style transfer](https://md.research.elamur.ai/free/en/papers/li-2018-delete-retrieve-generate-simple-approach/). arXiv preprint arXiv:1804.06437 (2018) [↗](https://arxiv.org/abs/1804.06437)
11. Lin, C.Y.: [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). Text Summarization Branches Out (2004)
12. Luong, M.T., Pham, H., Manning, C.D.: [Effective approaches to attention-based neural machine translation](https://md.research.elamur.ai/free/en/papers/luong-2015-effective-approaches-attention-based-neural/). arXiv preprint arXiv:1508.04025 (2015) [↗](https://arxiv.org/abs/1508.04025)
13. Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In: Proceedings of the 40th annual meeting on association for computational linguistics. pp. 311–318. Association for Computational Linguistics (2002)
14. Paulus, R., Xiong, C., Socher, R.: [A deep reinforced model for abstractive summarization](https://md.research.elamur.ai/free/en/papers/paulus-2017-deep-reinforced-model-abstractive-summarization/). arXiv preprint arXiv:1705.04304 (2017) [↗](https://arxiv.org/abs/1705.04304)
15. Prabhumoye, S., Tsvetkov, Y., Salakhutdinov, R., Black, A.W.: [Style transfer through back-translation](https://md.research.elamur.ai/free/en/papers/prabhumoye-2018-style-transfer-through-back-translation/). In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 866–876. Association for Computational Linguistics (2018), http://aclweb.org/anthology/P18-1080 [↗](https://aclanthology.org/P18-1080/)
16. Rao, S., Tetreault, J.: [Dear sir or madam, may i introduce the gyafc dataset: Corpus, benchmarks and metrics for formality style transfer](https://md.research.elamur.ai/free/en/papers/rao-2018-dear-sir-or-madam-may/). In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). vol. 1, pp. 129–140 (2018)
17. Rennie, S.J., Marcheret, E., Mroueh, Y., Ross, J., Goel, V.: [Self-critical sequence training for image captioning](https://md.research.elamur.ai/free/en/papers/rennie-2016-self-critical-sequence-training-image/). In: CVPR. vol. 1, p. 3 (2017)
18. Shen, T., Lei, T., Barzilay, R., Jaakkola, T.: [Style transfer from non-parallel text by cross-alignment](https://md.research.elamur.ai/free/en/papers/shen-2017-style-transfer-non-parallel-text/). In: Advances in Neural Information Processing Systems. pp. 6830–6841 (2017)
19. Subramanian, S., Lample, G., Smith, E.M., Denoyer, L., Ranzato, M., Boureau, Y.L.: Multiple-attribute text style transfer. arXiv preprint arXiv:1811.00552 (2018) Sancheti et al. [↗](https://arxiv.org/abs/1811.00552)
20. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In: Advances in Neural Information Processing Systems. pp. 5998–6008 (2017)
21. Wu, L., Tian, F., Qin, T., Lai, J., Liu, T.Y.: [A study of reinforcement learning for neural machine translation](https://md.research.elamur.ai/free/en/papers/wu-2018-study-reinforcement-learning-neural-machine/). In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 3612–3621. Association for Computational Linguistics (2018), http://aclweb.org/anthology/D18-1397 [↗](https://aclanthology.org/D18-1397/)
22. Xu, J., Sun, X., Zeng, Q., Ren, X., Zhang, X., Wang, H., Li, W.: [Unpaired sentiment-to-sentiment translation: A cycled reinforcement learning approach](https://md.research.elamur.ai/free/en/papers/xu-2018-unpaired-sentiment-sentiment-translation-cycled/). arXiv preprint arXiv:1805.05181 (2018) [↗](https://arxiv.org/abs/1805.05181)
23. Xu, W., Ritter, A., Dolan, B., Grishman, R., Cherry, C.: [Paraphrasing for style](https://md.research.elamur.ai/free/en/papers/xu-2012-paraphrasing-style/). Proceedings of COLING 2012 pp. 2899–2914 (2012)

*Updated 24 October 2026*