---
title: "Integrating Transformer and Paraphrase Rules for — Zhao 2018"
description: "The paper addresses sentence simplification, which means rewriting a sentence so it is easier to read without changing its meaning."
authors: ["Sanqiang Zhao", "Rui Meng", "Daqing He", "Andi Saptono", "Bambang Parmanto"]
year: 2018
venue: "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D18-1355/"
doi: "10.18653/v1/d18-1355"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/zhao-2018-integrating-transformer-paraphrase-rules-sentence/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/zhao-2018-integrating-transformer-paraphrase-rules-sentence/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2018](https://md.research.elamur.ai/free/papers/2018/page/2/) / Integrating Transformer and Paraphrase Rules for Sentence Simplification

# Integrating Transformer and Paraphrase Rules for Sentence Simplification

**Authors:** Sanqiang Zhao, Rui Meng, Daqing He, Andi Saptono, Bambang Parmanto · **Year:** 2018 · **Venue:** Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D18-1355/) · [PDF](https://aclanthology.org/D18-1355.pdf) · [DOI 10.18653/v1/d18-1355](https://doi.org/10.18653/v1/d18-1355) · [OpenAlex](https://openalex.org/W2963658612)

**Citations:** 105 citations (openalex-snapshot · 2026-10-14) · [73](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
The paper addresses sentence simplification, which means rewriting a sentence so it is easier to read without changing its meaning. Neural encoder-decoder systems trained on aligned normal–simple sentence pairs had trailed statistical systems on this task. The authors see two causes: training corpora are too small, and neural models absorb frequent rewrites while dismissing rare ones as noise.

They make two changes. First, they replace RNN/LSTM with a Transformer (multi-layer, multi-head attention). Second, they inject knowledge from Simple PPDB, a large set of weighted normal-to-simple paraphrase rules, in two ways:
- **DCSS** adds a critic-style loss. Whenever a rule applies, it raises the probability of the simpler phrase and lowers that of the original, scaled by the rule's weight.
- **DMASS** adds an external memory that keeps each rule's context and output separately, so rare rules are not overwritten in shared parameters.

The combined DMASS+DCSS achieves the best SARI among the compared systems, including SBMT-SARI and DRESS. It also applies rules with both good recall and good precision. This shows that curated lexical resources can be fused effectively into neural text generation.

## Key points
- Simple PPDB, derived from PPDB, holds about 4.5 million rules. Each rule has a normal phrase, a simpler phrase, a syntactic type (e.g. [NN], [VP]) and a simplification weight.
- The baselines include PBMT-R, Hybrid, SBMT-SARI, DRESS and DRESS-LS. Evaluation uses SARI (with separate add/keep/delete F1), FKGL, average word length (WLen), sentence length (SLen) and rule-utilization precision/recall.
- DRESS-style models simplify mainly by shortening sentences. SBMT-SARI and the proposed models simplify mainly by choosing shorter words.
- SBMT-SARI simplifies aggressively, which gives high recall but weaker precision. DMASS beats DCSS, and DMASS+DCSS is best overall.
- Larger beam sizes correlate strongly with rule utilization but only loosely with SARI. Code and models are released at github.com/Sanqiang/text_simplification.

## Interesting ideas
- **Contrastive critic loss.** The loss rewards the target-side replacement and penalizes the source-side word, weighted by an external confidence score. This is a cheap way to bias any seq2seq model toward a lexicon, for example for terminology or dialect normalization.
- **External per-rule memory.** Storing each rule in its own memory slot, rather than only in shared weights, protects long-tail knowledge from being treated as noise. This is useful wherever data is scarce and rare mappings matter, as in low-resource languages.
- **Combining internal and external memory.** The authors find that the two storage strategies complement each other rather than compete, so it is worth trying both together.
- **Rule utilization as a metric.** Scoring how often applicable rules are used (recall) and how often the applied ones are correct (precision) gives a more interpretable check than a single overall score.
- **Decomposing readability scores.** Splitting FKGL into word length and sentence length reveals *how* a system simplifies. It also flags cases where a good score may come from dropping information.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/)

## Cited by

Cited by

73 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 64 not yet.

- [EditNTS: An Neural Programmer-Interpreter Model for Sentence Simplification through Explicit Editing](https://md.research.elamur.ai/free/papers/dong-2019-editnts-neural-programmer-interpreter-model/)
- [Neural CRF Model for Sentence Alignment in Text Simplification](https://md.research.elamur.ai/free/papers/jiang-2020-neural-crf-model-sentence-alignment/)
- [Controllable Text Simplification with Explicit Paraphrasing](https://md.research.elamur.ai/free/papers/maddela-2021-controllable-text-simplification-explicit-paraphrasing/)
- [EASSE: Easier Automatic Sentence Simplification Evaluation](https://md.research.elamur.ai/free/papers/manchego-2019-easse-easier-automatic-sentence-simplification/)
- [ASSET: A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformations](https://md.research.elamur.ai/free/papers/manchego-2020-asset-dataset-tuning-evaluation-sentence/)
- [Data-Driven Sentence Simplification: Survey and Benchmark](https://md.research.elamur.ai/free/papers/manchego-2020-data-driven-sentence-simplification-survey/)
- [The (Un)Suitability of Automatic Evaluation Metrics for Text Simplification](https://md.research.elamur.ai/free/papers/manchego-2021-un-suitability-automatic-evaluation-metrics/)
- [MUSS: Multilingual Unsupervised Sentence Simplification by Mining Paraphrases](https://md.research.elamur.ai/free/papers/martin-2022-muss-multilingual-unsupervised-sentence-simplification/)
- [Simplify-Pro: A Two-level and Progressive LLM-based Framework for Auto Long Text Simplification](https://md.research.elamur.ai/free/papers/zhou-2026-simplify-pro-two-level-progressive/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhao2018integrating,
  title = {{Integrating Transformer and Paraphrase Rules for Sentence Simplification}},
  author = {Zhao, Sanqiang and Meng, Rui and He, Daqing and Saptono, Andi and Parmanto, Bambang},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1355},
  url = {https://doi.org/10.18653/v1/d18-1355}
}
```

- [← previous Integrating Stance Detection and Fact Checking in a Unified Corpus](https://md.research.elamur.ai/free/papers/baly-2018-integrating-stance-detection-fact-checking/)
- [next → Investigating Backtranslation in Neural Machine Translation](https://md.research.elamur.ai/free/papers/poncelas-2018-investigating-backtranslation-neural-machine-translation/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/papers/2018/page/2/)

## References

25 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. John A Carroll, Guido Minnen, Darren Pearce, Yvonne Canning, Siobhan Devlin, and John Tait. 1999. Simplifying text for language-impaired readers. In EACL, pages 269–270. [find](https://scholar.google.com/scholar?q=John+A+Carroll%2C+Guido+Minnen%2C+Darren+Pearce%2C+Yvonne+Canning%2C+Siobhan+Devlin%2C+and+John+Tait%2E+1999%2E+Simplifying+text+for+language%2Dimpaired+readers%2E+In+EACL%2C+pages+269%E2%80%93270%2E) [our card](https://md.research.elamur.ai/free/papers/carroll-1999-simplifying-text-language-impaired-readers/)
2. Michał Daniluk, Tim Rocktäschel, Johannes Welbl, and Sebastian Riedel. 2017. Frustratingly short attention spans in neural language modeling. arXiv preprint arXiv:1702.04521. [↗](https://arxiv.org/abs/1702.04521)
3. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121–2159. [find](https://scholar.google.com/scholar?q=John+Duchi%2C+Elad+Hazan%2C+and+Yoram+Singer%2E+2011%2E+Adaptive+subgradient+methods+for+online+learning+and+stochastic+optimization%2E+Journal+of+Machine+Learning+Research%2C+12%28Jul%29%3A2121%E2%80%932159%2E)
4. Yang Feng, Shiyue Zhang, Andi Zhang, Dong Wang, and Andrew Abel. 2017. Memory-augmented neural machine translation. arXiv preprint arXiv:1708.02005. [↗](https://arxiv.org/abs/1708.02005)
5. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. Ppdb: The paraphrase database. In HLT-NAACL, pages 758–764. [find](https://scholar.google.com/scholar?q=Juri+Ganitkevitch%2C+Benjamin+Van+Durme%2C+and+Chris+Callison%2DBurch%2E+2013%2E+Ppdb%3A+The+paraphrase+database%2E+In+HLT%2DNAACL%2C+pages+758%E2%80%93764%2E) [our card](https://md.research.elamur.ai/free/papers/ganitkevitch-2013-ppdb-paraphrase-database/)
6. Edouard Grave, Armand Joulin, and Nicolas Usunier. 2016. Improving neural language models with a continuous cache. arXiv preprint arXiv:1612.04426. [↗](https://arxiv.org/abs/1612.04426)
7. David Kauchak. 2013. Improving text simplification language modeling using unsimplified text data. In ACL (1), pages 1537–1546. [find](https://scholar.google.com/scholar?q=David+Kauchak%2E+2013%2E+Improving+text+simplification+language+modeling+using+unsimplified+text+data%2E+In+ACL+%281%29%2C+pages+1537%E2%80%931546%2E) [our card](https://md.research.elamur.ai/free/papers/kauchak-2013-improving-text-simplification-language-modeling/)
8. J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical report, Naval Technical Training Command Millington TN Research Branch. [↗](https://doi.org/10.21236/ada006655)
9. Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. arXiv preprint arXiv:1606.03126. [↗](https://arxiv.org/abs/1606.03126)
10. Shashi Narayan and Claire Gardent. 2014. Hybrid simplification using deep semantics and machine translation. In the 52nd Annual Meeting of the Association for Computational Linguistics, pages 435–445. [find](https://scholar.google.com/scholar?q=Shashi+Narayan+and+Claire+Gardent%2E+2014%2E+Hybrid+simplification+using+deep+semantics+and+machine+translation%2E+In+the+52nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+435%E2%80%93445%2E)
11. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning, pages 1310–1318. [↗](https://doi.org/10.48550/arxiv.1211.5063)
12. Ellie Pavlick and Chris Callison-Burch. 2016. Simple ppdb: A paraphrase database for simplification. In The 54th Annual Meeting of the Association for Computational Linguistics, page 143. [find](https://scholar.google.com/scholar?q=Ellie+Pavlick+and+Chris+Callison%2DBurch%2E+2016%2E+Simple+ppdb%3A+A+paraphrase+database+for+simplification%2E+In+The+54th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+page+143%2E)
13. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+Manning%2E+2014%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+Proceedings+of+the+2014+conference+on+empirical+methods+in+natural+language+process)
14. Luz Rello, Clara Bayarri, Azuki Gòrriz, Ricardo Baeza-Yates, Saurabh Gupta, Gaurang Kanvinde, Horacio Saggion, Stefan Bott, Roberto Carlini, and Vasile Topac. 2013. Dyswebxia 2.0!: more accessible text for people with dyslexia. In Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility, page 25. ACM. [find](https://scholar.google.com/scholar?q=Luz+Rello%2C+Clara+Bayarri%2C+Azuki+Go%CC%80rriz%2C+Ricardo+Baeza%2DYates%2C+Saurabh+Gupta%2C+Gaurang+Kanvinde%2C+Horacio+Saggion%2C+Stefan+Bott%2C+Roberto+Carlini%2C+and+Vasile+Topac%2E+2013%2E+Dyswebxia+2%2E0%21%3A+more+accessible+te)
15. Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of machine learning research, 15(1):1929–1958. [find](https://scholar.google.com/scholar?q=Nitish+Srivastava%2C+Geoffrey+E+Hinton%2C+Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Ruslan+Salakhutdinov%2E+2014%2E+Dropout%3A+a+simple+way+to+prevent+neural+networks+from+overfitting%2E+Journal+of+machine+learning+re)
16. Zhaopeng Tu, Yang Liu, Shuming Shi, and Tong Zhang. 2017. Learning to remember translation history with a continuous cache. arXiv preprint arXiv:1711.09367. [↗](https://arxiv.org/abs/1711.09367)
17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. arXiv preprint arXiv:1706.03762. [↗](https://arxiv.org/abs/1706.03762)
18. Willian Massami Watanabe, Arnaldo Candido Junior, Vinı́cius Rodriguez Uzêda, Renata Pontin de Mattos Fortes, Thiago Alexandre Salgueiro Pardo, and Sandra Maria Aluı́sio. 2009. Facilita: reading assistance for low-literacy readers. In Proceedings of the 27th ACM international conference on Design of communication, pages 29–36. ACM. [find](https://scholar.google.com/scholar?q=Willian+Massami+Watanabe%2C+Arnaldo+Candido+Junior%2C+Vin%C4%B1%CC%81cius+Rodriguez+Uze%CC%82da%2C+Renata+Pontin+de+Mattos+Fortes%2C+Thiago+Alexandre+Salgueiro+Pardo%2C+and+Sandra+Maria+Alu%C4%B1%CC%81sio%2E+2009%2E+Facilita%3A+reading+assis)
19. Kristian Woodsend and Mirella Lapata. 2011. Learning to simplify sentences with quasi-synchronous grammar and integer programming. In Proceedings of the conference on empirical methods in natural language processing, pages 409–420. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kristian+Woodsend+and+Mirella+Lapata%2E+2011%2E+Learning+to+simplify+sentences+with+quasi%2Dsynchronous+grammar+and+integer+programming%2E+In+Proceedings+of+the+conference+on+empirical+methods+in+natural+lang) [our card](https://md.research.elamur.ai/free/papers/woodsend-2011-learning-simplify-sentences-quasi-synchronous/)
20. Sander Wubben, Antal Van Den Bosch, and Emiel Krahmer. 2012. Sentence simplification by monolingual machine translation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 1015–1024. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sander+Wubben%2C+Antal+Van+Den+Bosch%2C+and+Emiel+Krahmer%2E+2012%2E+Sentence+simplification+by+monolingual+machine+translation%2E+In+Proceedings+of+the+50th+Annual+Meeting+of+the+Association+for+Computational+) [our card](https://md.research.elamur.ai/free/papers/wubben-2012-sentence-simplification-monolingual-machine-translation/)
21. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015a. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning, pages 2048–2057. [↗](https://doi.org/10.48550/arxiv.1502.03044) [our card](https://md.research.elamur.ai/free/papers/xu-2015-show-attend-tell-neural-image/)
22. Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015b. Problems in current text simplification research: New data can help. Transactions of the Association of Computational Linguistics, 3(1):283– 297. [↗](https://doi.org/10.1162/tacl_a_00139) [our card](https://md.research.elamur.ai/free/papers/xu-2015-problems-current-text-simplification-research/)
23. Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401–415. [↗](https://doi.org/10.1162/tacl_a_00107) [our card](https://md.research.elamur.ai/free/papers/xu-2016-optimizing-statistical-machine-translation-text/)
24. Xingxing Zhang and Mirella Lapata. 2017. Sentence simplification with deep reinforcement learning. arXiv preprint arXiv:1703.10931. [↗](https://arxiv.org/abs/1703.10931) [our card](https://md.research.elamur.ai/free/papers/zhang-2017-sentence-simplification-deep-reinforcement-learning/)
25. Zhemin Zhu, Delphine Bernhard, and Iryna Gurevych. 2010. A monolingual tree-based translation model for sentence simplification. In Proceedings of the 23rd international conference on computational linguistics, pages 1353–1361. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zhemin+Zhu%2C+Delphine+Bernhard%2C+and+Iryna+Gurevych%2E+2010%2E+A+monolingual+tree%2Dbased+translation+model+for+sentence+simplification%2E+In+Proceedings+of+the+23rd+international+conference+on+computational+li) [our card](https://md.research.elamur.ai/free/papers/zhu-2010-monolingual-tree-based-translation-model/)