---
title: "Improving Text Simplification Language — Kauchak 2013"
description: "This paper asks whether a language model for simplified English can benefit from ordinary, unsimplified English."
authors: ["David Kauchak"]
year: 2013
venue: "Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/P13-1151/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/kauchak-2013-improving-text-simplification-language-modeling/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/kauchak-2013-improving-text-simplification-language-modeling/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2013](https://md.research.elamur.ai/free/papers/2013/) / Improving Text Simplification Language Modeling Using Unsimplified Text Data

# Improving Text Simplification Language Modeling Using Unsimplified Text Data

**Authors:** [David Kauchak](https://md.research.elamur.ai/free/authors/david-kauchak/) · **Year:** 2013 · **Venue:** Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/P13-1151/) · [PDF](https://aclanthology.org/P13-1151.pdf) · [OpenAlex](https://openalex.org/W2112282043)

**Citations:** 124 citations (openalex-snapshot · 2026-10-14) · [99](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
This paper asks whether a language model for simplified English can benefit from ordinary, unsimplified English. Simplification is a monolingual translation task, so data from the input side is also English, but simple text is scarce. The author built a document-aligned corpus of 60K paired articles, each matching a Simple English Wikipedia entry with its regular English Wikipedia counterpart, and trained trigram models with interpolated Kneser-Ney smoothing in SRILM. The models used simple-only data, normal-only data, or mixtures of the two, combined either by concatenating the corpora or by linear interpolation. They were evaluated by perplexity on held-out simple text and on the SemEval 2012 lexical simplification task. Adding normal data improved results over a model trained on all available simple data: perplexity dropped by 23% and lexical simplification improved by 24%. Further analysis showed that normal text mainly helps by covering rare and unseen n-grams. It remains out-of-domain, however, so simple data is worth much more per sentence. This matters for any low-resource monolingual generation task where in-domain target text is limited.

## Key points
- Corpus: 60K aligned article pairs, with 2K pairs held out for testing. The simple side has 385K sentences, 7.15M words and a 78K vocabulary. The normal side has 2,540K sentences, 64.7M words and a 307K vocabulary.
- The test set has 7,799 simple sentences (179K words). All models share a closed vocabulary taken from the simple training side.
- In a 137K sentence-aligned set, 96% of simple unigrams appear in normal text. For 5-grams the overlap is only about 55% (simple in normal) and 46% (normal in simple).
- Perplexity fell from 129 (simple-only) to 100 (all simple + normal). The normal-only model slightly beat the simple-only model, but only by using about ten times more data.
- Simple data is more valuable per sentence. To match 250K simple sentences, a model needs 200K simple plus 300K normal, or 100K simple plus 850K normal. The best interpolation weight on simple was 0.5 for perplexity and 0.98 for lexical simplification.

## Interesting ideas
- **Train the LM on input-side text too.** In monolingual tasks such as compression, simplification or dialect normalization, abundant source-side text can act as a backoff or smoothing resource for a small target-style corpus.
- **Measure n-gram overlap in both directions.** Asymmetric overlap between two text varieties is a cheap diagnostic. It shows whether one variety is more lexically diverse and whether extra data is likely to fill coverage gaps.
- **Use a fixed shared vocabulary.** Fixing one closed vocabulary across all compared models keeps perplexity comparisons fair when training corpora differ in size and source.
- **Tune the mixing weight per task.** The best interpolation weight differed sharply between intrinsic perplexity and the downstream task. Tuning it on the target task avoids misleading conclusions drawn from perplexity alone.
- **Separate content effects from style effects.** Training on content-matched, sentence-aligned pairs shows that the out-of-domain penalty comes from different language use, not topic. The same control could isolate register effects in other corpora.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/)

## Cited by

Cited by

99 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 89 not yet.

- [EditNTS: An Neural Programmer-Interpreter Model for Sentence Simplification through Explicit Editing](https://md.research.elamur.ai/free/papers/dong-2019-editnts-neural-programmer-interpreter-model/)
- [LENS: A Learnable Evaluation Metric for Text Simplification](https://md.research.elamur.ai/free/papers/maddela-2023-lens-learnable-evaluation-metric-text/)
- [Data-Driven Sentence Simplification: Survey and Benchmark](https://md.research.elamur.ai/free/papers/manchego-2020-data-driven-sentence-simplification-survey/)
- [MUSS: Multilingual Unsupervised Sentence Simplification by Mining Paraphrases](https://md.research.elamur.ai/free/papers/martin-2022-muss-multilingual-unsupervised-sentence-simplification/)
- [Exploring Neural Text Simplification Models](https://md.research.elamur.ai/free/papers/nisioi-2017-exploring-neural-text-simplification-models/)
- [Simple and Effective Text Simplification Using Semantic and Neural Methods](https://md.research.elamur.ai/free/papers/sulem-2018-simple-effective-text-simplification-semantic/)
- [Problems in Current Text Simplification Research: New Data Can Help](https://md.research.elamur.ai/free/papers/xu-2015-problems-current-text-simplification-research/)
- [Optimizing Statistical Machine Translation for Text Simplification](https://md.research.elamur.ai/free/papers/xu-2016-optimizing-statistical-machine-translation-text/)
- [Sentence Simplification with Deep Reinforcement Learning](https://md.research.elamur.ai/free/papers/zhang-2017-sentence-simplification-deep-reinforcement-learning/)
- [Integrating Transformer and Paraphrase Rules for Sentence Simplification](https://md.research.elamur.ai/free/papers/zhao-2018-integrating-transformer-paraphrase-rules-sentence/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kauchak2013improving,
  title = {{Improving Text Simplification Language Modeling Using Unsimplified Text Data}},
  author = {Kauchak, David},
  booktitle = {Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2013},
  url = {https://aclanthology.org/P13-1151/}
}
```

- [← previous Improving pairwise coreference models through feature space hierarchy learning](https://md.research.elamur.ai/free/papers/lassalle-2013-improving-pairwise-coreference-models-through/)
- [next → Interpreting Anaphoric Shell Nouns using Antecedents of Cataphoric Shell Nouns as Training Data](https://md.research.elamur.ai/free/papers/kolhatkar-2013-interpreting-anaphoric-shell-nouns-antecedents/)
- [↑ back to papers of 2013](https://md.research.elamur.ai/free/papers/2013/)

## References

35 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Michiel Bacchiani and Brian Roark. 2003. Unsupervised language model adaptation. In Proceedings of ICASSP. [find](https://scholar.google.com/scholar?q=Michiel+Bacchiani+and+Brian+Roark%2E+2003%2E+Unsupervised+language+model+adaptation%2E+In+Proceedings+of+ICASSP%2E)
2. Michele Banko, Vibhu Mittal, and Michael Witbrock. 2000. Headline generation based on statistical translation. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Michele+Banko%2C+Vibhu+Mittal%2C+and+Michael+Witbrock%2E+2000%2E+Headline+generation+based+on+statistical+translation%2E+In+Proceedings+of+ACL%2E)
3. Jerome R. Bellegarda. 2004. Statistical language model adaptation: Review and perspectives. Speech Communication. [↗](https://doi.org/10.1016/j.specom.2003.08.002)
4. Or Biran, Samuel Brody, and Noeḿie Elhadad. 2011. Putting it simply: A context-aware approach to lexical simplification. In Proceedings of ACL. [↗](https://doi.org/10.7916/d8cv4s20) [our card](https://md.research.elamur.ai/free/papers/biran-2011-putting-it-simply-context-aware/)
5. Thorsten Brants, Ashok C. Popat, Peng Xu, Franz J. Och, and Jeffrey Dean. 2007. Large language models in machine translation. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Thorsten+Brants%2C+Ashok+C%2E+Popat%2C+Peng+Xu%2C+Franz+J%2E+Och%2C+and+Jeffrey+Dean%2E+2007%2E+Large+language+models+in+machine+translation%2E+In+Proceedings+of+EMNLP%2E) [our card](https://md.research.elamur.ai/free/papers/brants-2007-large-language-models-machine-translation/)
6. Raman Chandrasekar and Bangalore Srinivas. 1997. Automatic induction of rules for text simplification. Knowledge Based Systems. [↗](https://doi.org/10.1016/s0950-7051(97)00029-4)
7. Stanley Chen, Douglas Beeferman, and Ronald Rosenfeld. 1998. Evaluation metrics for language models. In DARPA Broadcast News Transcription and Understanding Workshop. [find](https://scholar.google.com/scholar?q=Stanley+Chen%2C+Douglas+Beeferman%2C+and+Ronald+Rosenfeld%2E+1998%2E+Evaluation+metrics+for+language+models%2E+In+DARPA+Broadcast+News+Transcription+and+Understanding+Workshop%2E)
8. Trevor Cohn and Mirella Lapata. 2009. Sentence compression as tree transduction. Journal of Artificial Intelligence Research. [↗](https://doi.org/10.1613/jair.2655)
9. William Coster and David Kauchak. 2011a. Learning to simplify sentences using Wikipedia. In Proceedings of Text-To-Text Generation. [find](https://scholar.google.com/scholar?q=William+Coster+and+David+Kauchak%2E+2011a%2E+Learning+to+simplify+sentences+using+Wikipedia%2E+In+Proceedings+of+Text%2DTo%2DText+Generation%2E) [our card](https://md.research.elamur.ai/free/papers/coster-2011-learning-simplify-sentences-wikipedia/)
10. William Coster and David Kauchak. 2011b. Simple English Wikipedia: A new text simplification task. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=William+Coster+and+David+Kauchak%2E+2011b%2E+Simple+English+Wikipedia%3A+A+new+text+simplification+task%2E+In+Proceedings+of+ACL%2E) [our card](https://md.research.elamur.ai/free/papers/coster-2011-simple-english-wikipedia-new-text/)
11. Hal Daume and Daniel Marcu. 2002. A noisy-channel model for document compression. In Proceedings of ACL. [↗](https://doi.org/10.21236/ada459360)
12. Christopher Cieri David Graff. 2003. English gigaword. [http://www.ldc.upenn.edu/Catalog/CatalogEntry](http://www.ldc.upenn.edu/Catalog/CatalogEntry). jsp?catalogId=LDC2003T05. [↗](http://www.ldc.upenn.edu/Catalog/CatalogEntry)
13. Carsten Eickhoff, Pavel Serdyukov, and Arjen P. de Vries. 2010. Web page classification on child suitability. In Proceedings of CIKM. [find](https://scholar.google.com/scholar?q=Carsten+Eickhoff%2C+Pavel+Serdyukov%2C+and+Arjen+P%2E+de+Vries%2E+2010%2E+Web+page+classification+on+child+suitability%2E+In+Proceedings+of+CIKM%2E)
14. Michel Galley and Kathleen McKeown. 2007. Lexicalized Markov grammars for sentence compression. In Proceedings of HLT-NAACL. [↗](https://doi.org/10.7916/d8057qdg)
15. Le Quan Ha, E. I. Sicilia-Garcia, Ji Ming, and F. J. Smith. 2003. Extension of Zipf’s law to word and character n-grams for English and Chinese. Computational Linguistics and Chinese Language Processing. [find](https://scholar.google.com/scholar?q=Le+Quan+Ha%2C+E%2E+I%2E+Sicilia%2DGarcia%2C+Ji+Ming%2C+and+F%2E+J%2E+Smith%2E+2003%2E+Extension+of+Zipf%E2%80%99s+law+to+word+and+character+n%2Dgrams+for+English+and+Chinese%2E+Computational+Linguistics+and+Chinese+Language+Processi)
16. Bo-June Hsu. 2007. Generalized linear interpolation of language models. In IEEE Workshop on ASRU. [find](https://scholar.google.com/scholar?q=Bo%2DJune+Hsu%2E+2007%2E+Generalized+linear+interpolation+of+language+models%2E+In+IEEE+Workshop+on+ASRU%2E)
17. Frederick Jelinek and Robert Mercer. 1980. Interpolated estimation of markov source parameters from sparse data. In Proceedings of the Workshop on Patter Recognition in Practice. [find](https://scholar.google.com/scholar?q=Frederick+Jelinek+and+Robert+Mercer%2E+1980%2E+Interpolated+estimation+of+markov+source+parameters+from+sparse+data%2E+In+Proceedings+of+the+Workshop+on+Patter+Recognition+in+Practice%2E)
18. Kevin Knight and Daniel Marcu. 2002. Summarization beyond sentence extraction: a probabilistic approach to sentence compression. Artificial Intelligence. [↗](https://doi.org/10.1016/s0004-3702(02)00222-9)
19. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics. [↗](https://doi.org/10.2307/2529310)
20. Gondy Leroy, James E. Endicott, Obay Mouradi, David Kauchak, and Melissa Just. 2012. Improving perceived and actual text difficulty for health information consumers using semi-automated methods. In American Medical Informatics Association (AMIA) Fall Symposium. [find](https://scholar.google.com/scholar?q=Gondy+Leroy%2C+James+E%2E+Endicott%2C+Obay+Mouradi%2C+David+Kauchak%2C+and+Melissa+Just%2E+2012%2E+Improving+perceived+and+actual+text+difficulty+for+health+information+consumers+using+semi%2Dautomated+methods%2E+In+Am)
21. Courtney Napoles and Mark Dredze. 2010. Learning simple Wikipedia: A cogitation in ascertaining abecedarian language. In Proceedings of HLT/NAACL Workshop on Computation Linguistics and Writing. [find](https://scholar.google.com/scholar?q=Courtney+Napoles+and+Mark+Dredze%2E+2010%2E+Learning+simple+Wikipedia%3A+A+cogitation+in+ascertaining+abecedarian+language%2E+In+Proceedings+of+HLT%2FNAACL+Workshop+on+Computation+Linguistics+and+Writing%2E)
22. Tadashi Nomoto. 2009. A comparison of model free versus model intensive approaches to sentence compression. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Tadashi+Nomoto%2E+2009%2E+A+comparison+of+model+free+versus+model+intensive+approaches+to+sentence+compression%2E+In+Proceedings+of+EMNLP%2E)
23. Sinno Jialin Pan and Qiang Yang. 2010. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering. [↗](https://doi.org/10.1109/tkde.2009.191)
24. Ronald Rosenfeld. 1996. A maximum entropy approach to adaptive statistical language modeling. Computer, Speech and Language. [find](https://scholar.google.com/scholar?q=Ronald+Rosenfeld%2E+1996%2E+A+maximum+entropy+approach+to+adaptive+statistical+language+modeling%2E+Computer%2C+Speech+and+Language%2E)
25. Lucia Specia, Sujay Kumar Jauhar, and Rada Mihalcea. 2012. Semeval-2012 task 1: English lexical simplification. In Joint Conference on Lexical and Computerational Semantics (*SEM). [find](https://scholar.google.com/scholar?q=Lucia+Specia%2C+Sujay+Kumar+Jauhar%2C+and+Rada+Mihalcea%2E+2012%2E+Semeval%2D2012+task+1%3A+English+lexical+simplification%2E+In+Joint+Conference+on+Lexical+and+Computerational+Semantics+%28%2ASEM%29%2E) [our card](https://md.research.elamur.ai/free/papers/specia-2012-semeval-2012-task-1-english/)
26. Lucia Specia. 2010. Translating from complex to simplified sentences. In Proceedings of Computational Processing of the Portuguese Language. [find](https://scholar.google.com/scholar?q=Lucia+Specia%2E+2010%2E+Translating+from+complex+to+simplified+sentences%2E+In+Proceedings+of+Computational+Processing+of+the+Portuguese+Language%2E)
27. Andreas Stolcke. 2002. SRILM - An extensible language modeling toolkit. In Proceedings of ICSLP. [find](https://scholar.google.com/scholar?q=Andreas+Stolcke%2E+2002%2E+SRILM+%2D+An+extensible+language+modeling+toolkit%2E+In+Proceedings+of+ICSLP%2E)
28. Hisami Suzuki and Jianfeng Gao. 2005. A comparative study on language model adaptation techniques. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Hisami+Suzuki+and+Jianfeng+Gao%2E+2005%2E+A+comparative+study+on+language+model+adaptation+techniques%2E+In+Proceedings+of+EMNLP%2E)
29. Jenine Turner and Eugene Charniak. 2005. Supervised and unsupervised learning for sentence compression. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Jenine+Turner+and+Eugene+Charniak%2E+2005%2E+Supervised+and+unsupervised+learning+for+sentence+compression%2E+In+Proceedings+of+ACL%2E)
30. Ken Urano. 2000. Lexical simplification and elaboration: Sentence comprehension and incidental vocabulary acquisition. Master’s thesis, University of Hawaii. [find](https://scholar.google.com/scholar?q=Ken+Urano%2E+2000%2E+Lexical+simplification+and+elaboration%3A+Sentence+comprehension+and+incidental+vocabulary+acquisition%2E+Master%E2%80%99s+thesis%2C+University+of+Hawaii%2E)
31. Kristian Woodsend and Mirella Lapata. 2011. Learning to simplify sentences with quasi-synchronous grammar and integer programming. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Kristian+Woodsend+and+Mirella+Lapata%2E+2011%2E+Learning+to+simplify+sentences+with+quasi%2Dsynchronous+grammar+and+integer+programming%2E+In+Proceedings+of+EMNLP%2E) [our card](https://md.research.elamur.ai/free/papers/woodsend-2011-learning-simplify-sentences-quasi-synchronous/)
32. Sander Wubben, Antal van den Bosch, and Emiel Krahmer. 2012. Sentence simplification by monolingual machine translation. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Sander+Wubben%2C+Antal+van+den+Bosch%2C+and+Emiel+Krahmer%2E+2012%2E+Sentence+simplification+by+monolingual+machine+translation%2E+In+Proceedings+of+ACL%2E) [our card](https://md.research.elamur.ai/free/papers/wubben-2012-sentence-simplification-monolingual-machine-translation/)
33. Mark Yatskar, Bo Pang, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2010. For the sake of simplicity: Unsupervised extraction of lexical simplifications from Wikipedia. In Proceedings of NAACL. [↗](https://doi.org/10.48550/arxiv.1008.1986)
34. Bing Zhao, Matthias Eck, and Stephan Vogel. 2004. Language model adaptation for statistical machine translation with structured query models. In Proceedings of COLING. [↗](https://doi.org/10.5445/ir/1000166436)
35. Zhemin Zhu, Delphine Bernhard, and Iryna Gurevych. 2010. A monolingual tree-based translation model for sentence simplification. In Proceedings of ICCL. [find](https://scholar.google.com/scholar?q=Zhemin+Zhu%2C+Delphine+Bernhard%2C+and+Iryna+Gurevych%2E+2010%2E+A+monolingual+tree%2Dbased+translation+model+for+sentence+simplification%2E+In+Proceedings+of+ICCL%2E) [our card](https://md.research.elamur.ai/free/papers/zhu-2010-monolingual-tree-based-translation-model/)