---
title: "SemEval 2016 Task 11 — Paetzold 2016"
description: "This paper reports on the first shared task on Complex Word Identification (CWI) at SemEval 2016."
authors: ["Gustavo Paetzold", "Lucia Specia"]
year: 2016
venue: "Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)"
source: "https://aclanthology.org/S16-1085/"
doi: "10.18653/v1/s16-1085"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/paetzold-2016-semeval-2016-task-11-complex/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/paetzold-2016-semeval-2016-task-11-complex/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/) / SemEval 2016 Task 11: Complex Word Identification

# SemEval 2016 Task 11: Complex Word Identification

**Authors:** [Gustavo Paetzold](https://md.research.elamur.ai/free/en/authors/gustavo-paetzold/), [Lucia Specia](https://md.research.elamur.ai/free/en/authors/lucia-specia/) · **Year:** 2016 · **Venue:** Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/S16-1085/) · [PDF](https://aclanthology.org/S16-1085.pdf) · [DOI 10.18653/v1/s16-1085](https://doi.org/10.18653/v1/s16-1085) · [OpenAlex](https://openalex.org/W2473989358)

**Citations:** 158 citations (openalex-snapshot · 2026-10-14) · [146](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
This paper reports on the first shared task on Complex Word Identification (CWI) at SemEval 2016. CWI decides which words in a sentence a simplifier should replace. The organizers argue that earlier resources were unreliable, such as the CW corpus built automatically from Simple English Wikipedia edits. To address this, they ran a user study in which 400 non-native English speakers marked the content words they did not understand in 9,200 sentences drawn from LexMTurk, the CW corpus and aligned Simple Wikipedia. A small portion was labelled by 20 annotators each and used for training. The much larger remainder was labelled by a single annotator per sentence and used for testing. This setup mimics predicting an individual reader's needs from a sample of the population. In total, 21 teams submitted 42 systems, which were compared against nine baselines and ranked by F-score and G-score. Decision Trees, Random Forests and ensembles did best, while neural and embedding-based models lagged behind. Plain corpus frequency remained the most dependable signal of complexity. The work matters because it supplies a reader-grounded benchmark and shows that editor-derived simplification data poorly reflects what non-native readers actually struggle with.

## Key points
- **Dataset scale:** The training sets cover 2,237 instances, each judged by 20 people. The test set contains 88,221 single-annotator instances. Training data came in two forms: a "joint" set with one binary label per word (complex if any annotator flagged it) and a "decomposed" set with all 20 individual labels.
- **Low agreement:** Krippendorff's Alpha on the multiply-annotated subsets averaged 0.244 (std 0.1). Cohen's Kappa was avoided because the heavy skew toward simple words inflates chance agreement.
- **Editor judgments vs. readers:** Words simplified by Wikipedia editors were rarely confirmed as complex by the annotators. Only about a third or fewer of the target words in CW, LexMTurk, the SemEval 2012 LS data and LSeval were flagged by anyone. Even 27.3% of the "already simple" Simple Wikipedia sentences contained a word someone found complex.
- **Properties of complex words:** These words had lower probability under a Simple Wikipedia trigram LM, fewer WordNet senses, synonyms, hypernyms and hyponyms, and were on average shorter. All of these differences were significant.
- **Results:** The best systems reached roughly 0.77 G-score. The top F-score came from PLUJAGH-SEWDFF, which simply learns a Wikipedia frequency threshold. Neural and embedding systems placed no better than 20th in G-score.

## Interesting ideas
- **Population-to-individual evaluation split:** The organizers train on densely annotated items and test on items labelled by single, different people. This directly measures how well group-level difficulty predicts one reader's needs, and the design could carry over to personalization tasks such as readability for specific learner groups.
- **Releasing both aggregated and per-annotator labels:** Publishing a "joint" and a "decomposed" version lets participants decide how to handle disagreement themselves. This is valuable whenever labels are inherently subjective.
- **Auditing existing resources against target users:** The authors checked whether widely used LS/CWI datasets matched real reader judgments, and the mismatch exposed hidden assumptions behind editor-made data. A similar audit is worth doing before reusing any proxy-labelled corpus for a new audience.
- **Strong frequency baselines and raw counts:** A single learned frequency threshold beat elaborate models on F-score, and raw counts outperformed LM probabilities. This argues for always including a well-tuned frequency baseline and paying attention to corpus choice.
- **Choosing the agreement metric to fit label imbalance:** The organizers preferred Krippendorff's Alpha over Kappa because of the heavily skewed classes. This is a useful reminder whenever annotating rare phenomena such as errors or complex terms.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

146 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 142 not yet.

- [The SAMER Arabic Text Simplification Corpus](https://md.research.elamur.ai/free/en/papers/alhafni-2024-samer-arabic-text-simplification-corpus/)
- [Towards Arabic Sentence Simplification via Classification and Generative Approaches](https://md.research.elamur.ai/free/en/papers/khallaf-2022-arabic-sentence-simplification-classification-generative/)
- [Learning How to Simplify From Explicit Labeling of Complex-Simplified Text Pairs](https://md.research.elamur.ai/free/en/papers/manchego-2017-learning-how-simplify-explicit-labeling/)
- [ASSET: A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformations](https://md.research.elamur.ai/free/en/papers/manchego-2020-asset-dataset-tuning-evaluation-sentence/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{paetzold2016semeval,
  title = {{SemEval 2016 Task 11: Complex Word Identification}},
  author = {Paetzold, Gustavo and Specia, Lucia},
  booktitle = {Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)},
  year = {2016},
  doi = {10.18653/v1/s16-1085},
  url = {https://doi.org/10.18653/v1/s16-1085}
}
```

- [← previous Semantically Motivated Hebrew Verb-Noun Multi-Word Expressions Identification](https://md.research.elamur.ai/free/en/papers/aclbulk-2016-liebeskind-semantically-motivated-hebrew-verb-noun/)
- [next → SemEval-2016 Task 10: Detecting Minimal Semantic Units and their Meanings (DiMSUM)](https://md.research.elamur.ai/free/en/papers/schneider-2016-semeval-2016-task-10-detecting/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/3/)

## References

15 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. John Carroll, Guido Minnen, Darren Pearce, Yvonne Canning, Siobhan Devlin, and John Tait. 1999. Simplifying text for language-impaired readers. In Proceedings of the 9th EACL, pages 269–270. [find](https://scholar.google.com/scholar?q=John+Carroll%2C+Guido+Minnen%2C+Darren+Pearce%2C+Yvonne+Canning%2C+Siobhan+Devlin%2C+and+John+Tait%2E+1999%2E+Simplifying+text+for+language%2Dimpaired+readers%2E+In+Proceedings+of+the+9th+EACL%2C+pages+269%E2%80%93270%2E) [our card](https://md.research.elamur.ai/free/en/papers/carroll-1999-simplifying-text-language-impaired-readers/)
2. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin, 70:213. [find](https://scholar.google.com/scholar?q=Jacob+Cohen%2E+1968%2E+Weighted+kappa%3A+Nominal+scale+agreement+provision+for+scaled+disagreement+or+partial+credit%2E+Psychological+bulletin%2C+70%3A213%2E)
3. Jan De Belder and Marie-Francine Moens. 2012. A dataset for the evaluation of lexical simplification. In Proceedings of the 13th CICLING. [find](https://scholar.google.com/scholar?q=Jan+De+Belder+and+Marie%2DFrancine+Moens%2E+2012%2E+A+dataset+for+the+evaluation+of+lexical+simplification%2E+In+Proceedings+of+the+13th+CICLING%2E)
4. Siobhan Devlin and John Tait. 1998. The use of a psycholinguistic database in the simplification of text for aphasic readers. Linguistic Databases, pages 161– 173. [find](https://scholar.google.com/scholar?q=Siobhan+Devlin+and+John+Tait%2E+1998%2E+The+use+of+a+psycholinguistic+database+in+the+simplification+of+text+for+aphasic+readers%2E+Linguistic+Databases%2C+pages+161%E2%80%93+173%2E)
5. Christiane Fellbaum. 1998. [find](https://scholar.google.com/scholar?q=Christiane+Fellbaum%2E+1998%2E)
6. WordNet: An Electronic Lexical Database. Bradford Books. Andrew F. Hayes and Klaus Krippendorff. 2007. Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1. [find](https://scholar.google.com/scholar?q=WordNet%3A+An+Electronic+Lexical+Database%2E+Bradford+Books%2E+Andrew+F%2E+Hayes+and+Klaus+Krippendorff%2E+2007%2E+Answering+the+call+for+a+standard+reliability+measure+for+coding+data%2E+Communication+Methods+and+)
7. Colby Horn, Cathryn Manduca, and David Kauchak. 2014. Learning a lexical simplifier using wikipedia. In Proceedings of the 52nd ACL, pages 458–463. [find](https://scholar.google.com/scholar?q=Colby+Horn%2C+Cathryn+Manduca%2C+and+David+Kauchak%2E+2014%2E+Learning+a+lexical+simplifier+using+wikipedia%2E+In+Proceedings+of+the+52nd+ACL%2C+pages+458%E2%80%93463%2E) [our card](https://md.research.elamur.ai/free/en/papers/horn-2014-learning-lexical-simplifier-wikipedia/)
8. David Kauchak. 2013. Improving text simplification language modeling using unsimplified text data. In Proceedings of the 51st ACL, pages 1537–1546. [find](https://scholar.google.com/scholar?q=David+Kauchak%2E+2013%2E+Improving+text+simplification+language+modeling+using+unsimplified+text+data%2E+In+Proceedings+of+the+51st+ACL%2C+pages+1537%E2%80%931546%2E) [our card](https://md.research.elamur.ai/free/en/papers/kauchak-2013-improving-text-simplification-language-modeling/)
9. Llus Padr and Evgeny Stanilovsky. 2012. Freeling 3.0: Towards wider multilinguality. In Proceedings of the 2012 LREC. [find](https://scholar.google.com/scholar?q=Llus+Padr+and+Evgeny+Stanilovsky%2E+2012%2E+Freeling+3%2E0%3A+Towards+wider+multilinguality%2E+In+Proceedings+of+the+2012+LREC%2E) [our card](https://md.research.elamur.ai/free/en/papers/padro-2012-freeling-3-0-wider-multilinguality/)
10. Gustavo H. Paetzold and Lucia Specia. 2013. Text simplification as tree transduction. In Proceedings of the 9th STIL. [find](https://scholar.google.com/scholar?q=Gustavo+H%2E+Paetzold+and+Lucia+Specia%2E+2013%2E+Text+simplification+as+tree+transduction%2E+In+Proceedings+of+the+9th+STIL%2E)
11. Gustavo H. Paetzold. 2013. Um sistema de simplificação automática de textos escritos em inglês por meio de transduçao de árvores. State University of Western Paraná. [find](https://scholar.google.com/scholar?q=Gustavo+H%2E+Paetzold%2E+2013%2E+Um+sistema+de+simplificac%CC%A7a%CC%83o+automa%CC%81tica+de+textos+escritos+em+ingle%CC%82s+por+meio+de+transduc%CC%A7ao+de+a%CC%81rvores%2E+State+University+of+Western+Parana%CC%81%2E)
12. Matthew Shardlow. 2013a. A comparison of techniques to automatically identify complex words. In Proceedings of the 51st ACL Student Research Workshop, pages 103–109. [find](https://scholar.google.com/scholar?q=Matthew+Shardlow%2E+2013a%2E+A+comparison+of+techniques+to+automatically+identify+complex+words%2E+In+Proceedings+of+the+51st+ACL+Student+Research+Workshop%2C+pages+103%E2%80%93109%2E)
13. Matthew Shardlow. 2013b. The cw corpus: A new resource for evaluating the identification of complex words. In Proceedings of the 2nd Workshop on Predicting and Improving Text Readability for Target Reader Populations, pages 69–77. [find](https://scholar.google.com/scholar?q=Matthew+Shardlow%2E+2013b%2E+The+cw+corpus%3A+A+new+resource+for+evaluating+the+identification+of+complex+words%2E+In+Proceedings+of+the+2nd+Workshop+on+Predicting+and+Improving+Text+Readability+for+Target+Re)
14. Matthew Shardlow. 2014. Out in the open: Finding and categorising errors in the lexical simplification pipeline. In Proceedings of the 9th LREC. [find](https://scholar.google.com/scholar?q=Matthew+Shardlow%2E+2014%2E+Out+in+the+open%3A+Finding+and+categorising+errors+in+the+lexical+simplification+pipeline%2E+In+Proceedings+of+the+9th+LREC%2E)
15. Andreas Stolcke. 2002. Srilm-an extensible language modeling toolkit. In Proceedings of the International Conference on Spoken Language Processing, pages 257–286. [find](https://scholar.google.com/scholar?q=Andreas+Stolcke%2E+2002%2E+Srilm%2Dan+extensible+language+modeling+toolkit%2E+In+Proceedings+of+the+International+Conference+on+Spoken+Language+Processing%2C+pages+257%E2%80%93286%2E)