---
title: "Understanding the Lexical Simplification Needs of Non-Native"
description: "The authors ran three crowdsourced user studies, one for each stage of a typical Lexical… 33 references, cited by 11 papers here."
authors: ["Gustavo Paetzold", "Lucia Specia"]
year: 2016
venue: "Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers"
source: "https://aclanthology.org/C16-1069/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/paetzold-2016-understanding-lexical-simplification-needs-non/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/paetzold-2016-understanding-lexical-simplification-needs-non/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/) / Understanding the Lexical Simplification Needs of Non-Native Speakers of English

# Understanding the Lexical Simplification Needs of Non-Native Speakers of English

**Authors:** [Gustavo Paetzold](https://md.research.elamur.ai/free/en/authors/gustavo-paetzold/) and [Lucia Specia](https://md.research.elamur.ai/free/en/authors/lucia-specia/) · **Year:** 2016 · **Venue:** [Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers](https://md.research.elamur.ai/free/en/venues/coling/) ([COLING 2016](https://md.research.elamur.ai/free/en/venues/coling/2016/)) · **Publisher:** [International Committee on Computational Linguistics (ICCL)](https://md.research.elamur.ai/free/en/publishers/international-committee-on-computational-linguistics-iccl/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/C16-1069/) · [PDF](https://aclanthology.org/C16-1069.pdf) · [OpenAlex](https://openalex.org/W2576320825)

**Citations:** 11 citations (OpenAlex, 2026-10-14) · [11](https://md.research.elamur.ai/free/en/papers/paetzold-2016-understanding-lexical-simplification-needs-non/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper asks which words actually cause trouble for non-native readers of English and what a good simpler replacement looks like for them. The authors ran three crowdsourced user studies, one for each stage of a typical Lexical Simplification pipeline: Complex Word Identification (CWI), Substitution Selection (SS) and Substitution Ranking (SR). Altogether, 1,100 non-native volunteers produced 211,564 annotations. The CWI sentences came from the CW Corpus, the LexMTurk Corpus and Simple Wikipedia. Annotations were then compared against 15 common complexity features: morphological, WordNet-based and n-gram language-model probabilities.

The main findings are:
- Word length and syllable count say little about difficulty for this audience, unlike earlier results for dyslexic readers.
- Words that readers find hard tend to have fewer senses and lower corpus probability.
- n-gram probabilities that include the surrounding context predict simplicity better than plain word frequency, which contradicts an assumption common in older work.
- Proficiency predicts how many words a reader struggles with, while native language predicts which ones.

The released datasets support building CWI models, simplifiers and readability tools tuned to non-native readers.

## Key points
- **CWI study:**
  - 9,200 sentences were annotated by 400 volunteers with 45 native languages, giving 158,624 annotations.
  - 200 sentences were each labelled by 20 people.
  - Even Simple Wikipedia sentences contained a word judged complex in 27.3% of cases.
- **CWI agreement:** Kappa was 0.616 overall, far above earlier lexical substitution and simplification tasks (0.277 and 0.398). It rose to 0.718 among annotators sharing a native language.
- **SS study:**
  - 2,554 sentences received 31,940 annotations.
  - Word-vector distance between target and candidate tracked grammaticality and meaning preservation better than language-model probability.
  - The probability of a candidate given the target's grammatical role was a reliable grammaticality signal.
  - Single-word replacements tended to damage multi-word expressions.
- **SR study:**
  - 300 annotators made 21,000 pairwise judgements, with five per instance.
  - These were converted into 901 ranked instances using the Wauthier et al. (2013) algorithm.
  - The best feature was Simple[1,1], with Spearman −0.501 and TRank 0.593. Length reached only 0.172 and syllables 0.097.
- **Release:** All data is available as a single download (User_Studies_NNS.zip).

## Interesting ideas
- Annotate a small overlapping subset with many judges and the large remainder with one judge each, so agreement can be measured cheaply without multiplying labelling cost.
- Break agreement down by annotator subgroup (native language, age, proficiency); this revealed that disagreement is often systematic rather than noise.
- Collect easy binary comparisons ("which is simpler?") and infer full rankings from them algorithmically, instead of asking people to rank whole lists.
- Test widely held assumptions against the data, as the authors did when they showed that length is not a good difficulty cue for this audience and that context does matter.
- Check whether a replacement word is plausible in the grammatical role the original word fills, rather than only checking its overall frequency.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Silver data could be labelled this way, with a small slice checked by all witness parsers and LLM teachers to calibrate trust in the single-witness bulk.
- **I2** (learn): Witness-parser agreement could be reported per genre or source, so that a "disagreement" caused by a dialect or domain is not mistaken for random error.
- **I3** (learn): LLM teacher tiers could judge pairs of candidate parses or answers, with a global ranking inferred afterwards, which is more reliable than asking for absolute scores.
- **I4** (learn): Heuristic features in repair rules could be audited for statistical significance before being trusted, similar to a negative control.
- **I5** (back): When generating or compressing text, a candidate lemma could be accepted only if it is attested in the same UD relation, and multi-word expressions should be protected from word-by-word substitution.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Sheffield](https://md.research.elamur.ai/free/en/universities/university-of-sheffield/) — University, Sheffield, United Kingdom · 76 works · 20,808 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 798 works · [💭 Developmental and Educational Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/developmental-and-educational-psychology/) · 260 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 1,373 works

## Cite

Cite as: Gustavo Paetzold and Lucia Specia. 2016. Understanding the Lexical Simplification Needs of Non-Native Speakers of English. Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{paetzold2016understanding,
  title = {{Understanding the Lexical Simplification Needs of Non-Native Speakers of English}},
  author = {Gustavo Paetzold and Lucia Specia},
  booktitle = {Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers},
  year = {2016},
  doi = {10.5281/zenodo.1040781},
  url = {https://doi.org/10.5281/zenodo.1040781}
}
```

- [← previous BCCWJ-DepPara: A Syntactic Annotation Treebank on the ‘Balanced Corpus of Contemporary Written Japanese'](https://md.research.elamur.ai/free/en/papers/asahara-2016-bccwj-deppara-syntactic-annotation-treebank/)
- [next → Stylometric Analysis of Early Modern Period English Plays](https://md.research.elamur.ai/free/en/papers/eisen-2016-stylometric-analysis-early-modern-period/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/4/)

## References

33 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sandra Aluisio and Caroline Gasperin. 2010. Fostering digital inclusion and accessibility: The porsimples project for simplification of portuguese texts. In Proceedings of the 2010 NAACL Young Investigators Workshop on Computational Approaches to Languages of the Americas, pages 46–53. [find](https://scholar.google.com/scholar?q=Sandra+Aluisio+and+Caroline+Gasperin%2E+2010%2E+Fostering+digital+inclusion+and+accessibility%3A+The+porsimples+project+for+simplification+of+portuguese+texts%2E+In+Proceedings+of+the+2010+NAACL+Young+Investi)
2. Or Biran, Samuel Brody, and Noémie Elhadad. 2011. [Putting it simply: a context-aware approach to lexical simplification](https://md.research.elamur.ai/free/en/papers/biran-2011-putting-it-simply-context-aware/). In Proceedings of the 49th ACL, pages 496–501. [↗](https://doi.org/10.7916/d8cv4s20)
3. Marc Brysbaert and Boris New. 2009. Moving beyond kučera and francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for american english. Behavior research methods, 41:977–990. [↗](https://doi.org/10.3758/brm.41.4.977)
4. Philip R Burns. 2013. Morphadorner v2: A java library for the morphological adornment of english language texts. Northwestern University, Evanston, IL. [find](https://scholar.google.com/scholar?q=Philip+R+Burns%2E+2013%2E+Morphadorner+v2%3A+A+java+library+for+the+morphological+adornment+of+english+language+texts%2E+Northwestern+University%2C+Evanston%2C+IL%2E)
5. Jean Carletta. 1996. [Assessing agreement on classification tasks: The kappa statistic](https://md.research.elamur.ai/free/en/papers/carletta-1996-assessing-agreement-classification-tasks-kappa/). Computational Linguistics, 22:249–254. [↗](https://doi.org/10.48550/arxiv.cmp-lg/9602004)
6. John Carroll, Guido Minnen, Darren Pearce, Yvonne Canning, Siobhan Devlin, and John Tait. 1999. [Simplifying text for language-impaired readers](https://md.research.elamur.ai/free/en/papers/carroll-1999-simplifying-text-language-impaired-readers/). In Proceedings of the 9th EACL, pages 269–270.
7. Raman Chandrasekar, Christine Doran, and Bangalore Srinivas. 1996. [Motivations and methods for text simplification](https://md.research.elamur.ai/free/en/papers/chandrasekar-1996-motivations-methods-text-simplification/). In Proceedings of the 16th Conference on Computational Linguistics.
8. Jan De Belder and Marie-Francine Moens. 2012. A dataset for the evaluation of lexical simplification. In Computational Linguistics and Intelligent Text Processing, pages 426–437. Springer. [find](https://scholar.google.com/scholar?q=Jan+De+Belder+and+Marie%2DFrancine+Moens%2E+2012%2E+A+dataset+for+the+evaluation+of+lexical+simplification%2E+In+Computational+Linguistics+and+Intelligent+Text+Processing%2C+pages+426%E2%80%93437%2E+Springer%2E)
9. Siobhan Devlin and John Tait. 1998. The use of a psycholinguistic database in the simplification of text for aphasic readers. Linguistic Databases, pages 161–173. [find](https://scholar.google.com/scholar?q=Siobhan+Devlin+and+John+Tait%2E+1998%2E+The+use+of+a+psycholinguistic+database+in+the+simplification+of+text+for+aphasic+readers%2E+Linguistic+Databases%2C+pages+161%E2%80%93173%2E)
10. Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. Bradford Books. [find](https://scholar.google.com/scholar?q=Christiane+Fellbaum%2E+1998%2E+WordNet%3A+An+Electronic+Lexical+Database%2E+Bradford+Books%2E)
11. Goran Glavaš and Sanja Štajner. 2015. [Simplifying lexical simplification: Do we need simplified corpora?](https://md.research.elamur.ai/free/en/papers/glavas-2015-simplifying-lexical-simplification-do-we/) In Proceedings of the 53rd ACL, pages 63–69.
12. Colby Horn, Cathryn Manduca, and David Kauchak. 2014. [Learning a lexical simplifier using wikipedia](https://md.research.elamur.ai/free/en/papers/horn-2014-learning-lexical-simplifier-wikipedia/). In Proceedings of the 52nd ACL, pages 458–463.
13. Tomoyuki Kajiwara, Hiroshi Matsumoto, and Kazuhide Yamamoto. 2013. Selecting proper lexical paraphrase for children. In Proceedings of the 25th Rocling, pages 59–73. [find](https://scholar.google.com/scholar?q=Tomoyuki+Kajiwara%2C+Hiroshi+Matsumoto%2C+and+Kazuhide+Yamamoto%2E+2013%2E+Selecting+proper+lexical+paraphrase+for+children%2E+In+Proceedings+of+the+25th+Rocling%2C+pages+59%E2%80%9373%2E)
14. David Kauchak. 2013. [Improving text simplification language modeling using unsimplified text data](https://md.research.elamur.ai/free/en/papers/kauchak-2013-improving-text-simplification-language-modeling/). In Proceedings of the 51st ACL, pages 1537–1546.
15. Dan Klein and Christopher D Manning. 2003. [Accurate unlexicalized parsing](https://md.research.elamur.ai/free/en/papers/klein-2003-accurate-unlexicalized-parsing/). In Proceedings of the 41st ACL, pages 423–430.
16. Diana McCarthy and Roberto Navigli. 2007. [Semeval-2007 task 10: English lexical substitution task](https://md.research.elamur.ai/free/en/papers/mccarthy-2007-semeval-2007-task-10-english/). In Proceedings of the 4th SemEval, pages 48–53.
17. Rada Mihalcea, Ravi Sinha, and Diana McCarthy. 2010. Semeval-2010 task 2: Cross-lingual lexical substitution. In Proceedings of the 5th international workshop on semantic evaluation, pages 9–14. [find](https://scholar.google.com/scholar?q=Rada+Mihalcea%2C+Ravi+Sinha%2C+and+Diana+McCarthy%2E+2010%2E+Semeval%2D2010+task+2%3A+Cross%2Dlingual+lexical+substitution%2E+In+Proceedings+of+the+5th+international+workshop+on+semantic+evaluation%2C+pages+9%E2%80%9314%2E)
18. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. [Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/). arXiv preprint arXiv:1301.3781. [↗](https://arxiv.org/abs/1301.3781)
19. Bernardo Pereira Nunes, Ricardo Kawase, Patrick Siehndel, Marco a. Casanova, and Stefan Dietze. 2013. As simple as it gets - a sentence simplifier for different learning levels and contexts. In Proceedings of the 13th ICALT, pages 128–132. [find](https://scholar.google.com/scholar?q=Bernardo+Pereira+Nunes%2C+Ricardo+Kawase%2C+Patrick+Siehndel%2C+Marco+a%2E+Casanova%2C+and+Stefan+Dietze%2E+2013%2E+As+simple+as+it+gets+%2D+a+sentence+simplifier+for+different+learning+levels+and+contexts%2E+In+Procee)
20. Gustavo H. Paetzold and Lucia Specia. 2013. Text simplification as tree transduction. In Proceedings of the 9th STIL, pages 116–125. [find](https://scholar.google.com/scholar?q=Gustavo+H%2E+Paetzold+and+Lucia+Specia%2E+2013%2E+Text+simplification+as+tree+transduction%2E+In+Proceedings+of+the+9th+STIL%2C+pages+116%E2%80%93125%2E)
21. Gustavo Henrique Paetzold and Lucia Specia. 2015. Lexenstein: A framework for lexical simplification. In Proceedings of The 53rd ACL, pages 85–90. [find](https://scholar.google.com/scholar?q=Gustavo+Henrique+Paetzold+and+Lucia+Specia%2E+2015%2E+Lexenstein%3A+A+framework+for+lexical+simplification%2E+In+Proceedings+of+The+53rd+ACL%2C+pages+85%E2%80%9390%2E)
22. Gustavo Henrique Paetzold and Lucia Specia. 2016. Unsupervised lexical simplification for non-native speakers. In Proceedings of The 30th AAAI. [find](https://scholar.google.com/scholar?q=Gustavo+Henrique+Paetzold+and+Lucia+Specia%2E+2016%2E+Unsupervised+lexical+simplification+for+non%2Dnative+speakers%2E+In+Proceedings+of+The+30th+AAAI%2E)
23. Gustavo Henrique Paetzold. 2015. Reliable lexical simplification for non-native speakers. In Proceedings of the 2015 NAACL Student Research Workshop, pages 9–16. [find](https://scholar.google.com/scholar?q=Gustavo+Henrique+Paetzold%2E+2015%2E+Reliable+lexical+simplification+for+non%2Dnative+speakers%2E+In+Proceedings+of+the+2015+NAACL+Student+Research+Workshop%2C+pages+9%E2%80%9316%2E)
24. Luz Rello, Ricardo Baeza-Yates, Stefan Bott, and Horacio Saggion. 2013a. Simplify or help?: text simplification strategies for people with dyslexia. In Proceedings of the 10th W4A, pages 1–10. [find](https://scholar.google.com/scholar?q=Luz+Rello%2C+Ricardo+Baeza%2DYates%2C+Stefan+Bott%2C+and+Horacio+Saggion%2E+2013a%2E+Simplify+or+help%3F%3A+text+simplification+strategies+for+people+with+dyslexia%2E+In+Proceedings+of+the+10th+W4A%2C+pages+1%E2%80%9310%2E)
25. Luz Rello, Ricardo Baeza-Yates, Laura Dempere-Marco, and Horacio Saggion. 2013b. Frequent words improve readability and short words improve understandability for people with dyslexia. Human-Computer Interaction, pages 203–219. [find](https://scholar.google.com/scholar?q=Luz+Rello%2C+Ricardo+Baeza%2DYates%2C+Laura+Dempere%2DMarco%2C+and+Horacio+Saggion%2E+2013b%2E+Frequent+words+improve+readability+and+short+words+improve+understandability+for+people+with+dyslexia%2E+Human%2DComputer+I)
26. Luz Rello, Susana Bautista, Ricardo Baeza-Yates, Pablo Gervás, Raquel Hervás, and Horacio Saggion. 2013c. One half or 50%? an eye-tracking study of number representation readability. Human-Computer Interaction. [find](https://scholar.google.com/scholar?q=Luz+Rello%2C+Susana+Bautista%2C+Ricardo+Baeza%2DYates%2C+Pablo+Gerva%CC%81s%2C+Raquel+Herva%CC%81s%2C+and+Horacio+Saggion%2E+2013c%2E+One+half+or+50%25%3F+an+eye%2Dtracking+study+of+number+representation+readability%2E+Human%2DComputer+)
27. Thea Reves and Peter Medgyes. 1994. The non-native english speaking efl/esl teacher’s self-image: An international survey. System, 22(3):353–367. [↗](https://doi.org/10.1016/0346-251x(94)90021-3)
28. Matthew Shardlow. 2013a. [A comparison of techniques to automatically identify complex words](https://md.research.elamur.ai/free/en/papers/shardlow-2013-comparison-techniques-automatically-identify-complex/). In Proceedings of the 51st ACL Student Research Workshop, pages 103–109.
29. Matthew Shardlow. 2013b. The cw corpus: A new resource for evaluating the identification of complex words. In Proceedings of the 2nd Workshop on Predicting and Improving Text Readability for Target Reader Populations, pages 69–77. [find](https://scholar.google.com/scholar?q=Matthew+Shardlow%2E+2013b%2E+The+cw+corpus%3A+A+new+resource+for+evaluating+the+identification+of+complex+words%2E+In+Proceedings+of+the+2nd+Workshop+on+Predicting+and+Improving+Text+Readability+for+Target+Re)
30. Matthew Shardlow. 2014. Out in the open: Finding and categorising errors in the lexical simplification pipeline. In Proceedings of the 9th LREC, pages 1583–1590. [find](https://scholar.google.com/scholar?q=Matthew+Shardlow%2E+2014%2E+Out+in+the+open%3A+Finding+and+categorising+errors+in+the+lexical+simplification+pipeline%2E+In+Proceedings+of+the+9th+LREC%2C+pages+1583%E2%80%931590%2E)
31. Lucia Specia, Sujay Kumar Jauhar, and Rada Mihalcea. 2012. [Semeval-2012 task 1: English lexical simplification](https://md.research.elamur.ai/free/en/papers/specia-2012-semeval-2012-task-1-english/). In Proceedings of the 1st SemEval, pages 347–355.
32. Andreas Stolcke. 2002. SRILM-an extensible language modeling toolkit. In Proceedings of the 2002 ICSLP, pages 257–286. [find](https://scholar.google.com/scholar?q=Andreas+Stolcke%2E+2002%2E+SRILM%2Dan+extensible+language+modeling+toolkit%2E+In+Proceedings+of+the+2002+ICSLP%2C+pages+257%E2%80%93286%2E)
33. Fabian Wauthier, Michael Jordan, and Nebojsa Jojic. 2013. Efficient ranking from pairwise comparisons. In Proceedings of the 30th International Conference on Machine Learning, pages 109–117. [find](https://scholar.google.com/scholar?q=Fabian+Wauthier%2C+Michael+Jordan%2C+and+Nebojsa+Jojic%2E+2013%2E+Efficient+ranking+from+pairwise+comparisons%2E+In+Proceedings+of+the+30th+International+Conference+on+Machine+Learning%2C+pages+109%E2%80%93117%2E)

*Updated 24 October 2026*