--- title: "SemEval-2021 Task 1: Lexical Complexity Prediction — Shardlow 2021" description: "The organisers therefore replace Complex Word Identification (CWI) with Lexical Complexity… 0.0182 Pearson. 2 references, cited by 96 papers here." authors: ["Matthew Shardlow", "Richard Evans", "Gustavo Henrique Paetzold", "Marcos Zampieri"] year: 2021 venue: "Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)" source: "https://aclanthology.org/2021.semeval-1.1/" doi: "10.18653/v1/2021.semeval-1.1" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/shardlow-2021-semeval-2021-task-1-lexical/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shardlow-2021-semeval-2021-task-1-lexical/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / SemEval-2021 Task 1: Lexical Complexity Prediction # SemEval-2021 Task 1: Lexical Complexity Prediction **Authors:** Matthew Shardlow, [Richard Evans](https://md.research.elamur.ai/free/en/authors/richard-evans/), [Gustavo Henrique Paetzold](https://md.research.elamur.ai/free/en/authors/gustavo-paetzold/) and [Marcos Zampieri](https://md.research.elamur.ai/free/en/authors/marcos-zampieri/) · **Year:** 2021 · **Venue:** Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2021.semeval-1.1/) · [PDF](https://aclanthology.org/2021.semeval-1.1.pdf) · [DOI 10.18653/v1/2021.semeval-1.1](https://doi.org/10.18653/v1/2021.semeval-1.1) · [OpenAlex](https://openalex.org/W3185421637) **Citations:** 59 citations (OpenAlex, 2026-10-14) · 96 in our library, of the 140,884 papers whose bibliographies we read ## Summary This shared-task report argues that a binary "complex vs. simple" label for words loses information, because readers differ in background and in how strictly they define difficulty. The organisers therefore replace Complex Word Identification (CWI) with Lexical Complexity Prediction (LCP), where systems output a graded score between 0 and 1. They extended the CompLex corpus into CompLex 2.0 by collecting about ten extra crowd judgments per item on Amazon Mechanical Turk and merging them with the retained CompLex 1.0 labels. Each Likert answer was mapped to a number and averaged. The resulting 10,800 in-context targets come from three genres and include both nouns and two-word expressions. These targets were split into two Sub-tasks: single words, and MWEs. Many teams took part. The best systems correlated well with human scores and clearly beat a frequency baseline, which earlier CWI tasks often failed to do. The work matters because a graded complexity score can feed lexical simplification, readability assessment and other downstream tasks. When needed, the score can still be turned into a binary label with a threshold. ## Key points - CompLex 2.0 has 10,800 instances drawn from Europarl, the World English Bible and the CRAFT biomedical corpus. Targets are nouns plus Noun-Noun and Adjective-Noun bigrams, identified with the Stanford CoreNLP POS tagger. - The five Likert levels were converted to 0, 0.25, 0.5, 0.75 and 1.0, and the mean of these values became the gold label. CompLex 1.0 had kept only items with at least 4 annotations, averaging about 7 per item; roughly 10 more were added for 2.0. - 198 teams registered. 54 submitted official test runs for Sub-task 1 (single words) and 37 for Sub-task 2 (MWEs), with 55 teams active across both. - Top Pearson correlations were 0.7886 for Sub-task 1 and 0.8612 for Sub-task 2, with R2 scores of 0.6172 and 0.7389. - The spread between rank 1 and rank 10 was tiny: 0.0182 Pearson points for Sub-task 1 and 0.021 for Sub-task 2. Biomedical text and MWEs tended to be rated harder than average. ## Interesting ideas - **Average graded judgments instead of collecting yes/no votes.** Mapping Likert answers to evenly spaced numbers and averaging them preserves annotator disagreement as a mid-range value rather than forcing an arbitrary decision. This suits any subjective labelling task, such as toxicity, formality or text difficulty for Ukrainian learners. - **Annotate the same word in several contexts.** Including up to five sentences per token per genre means a system that ignores context is automatically penalised. This is a simple way to make a dataset measure context sensitivity rather than lexicon lookup. - **Define every scale point in terms of reader experience.** The levels were described concretely, for example "never seen before" or "unclear but inferable from the sentence." Tying labels to behaviour like this can make crowdsourced annotations more consistent across languages and annotator pools. - **Treat small leaderboard gaps with caution.** The organisers note that the top-10 differences are so small that a different random split might reorder the ranking. Reporting such gaps, or testing robustness across splits, is a good habit for any benchmark or shared task. - **Pick a continuous target that can still be thresholded.** Predicting a score allows systems to rank words and still produce binary decisions when a downstream tool needs them. One dataset can therefore serve simplification, readability and feature-engineering uses at once. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Institutions - [University of Wolverhampton](https://md.research.elamur.ai/free/en/universities/university-of-wolverhampton/) — University, Wolverhampton, United Kingdom · 27 works · 5,947 citations - [Rochester Institute of Technology](https://md.research.elamur.ai/free/en/universities/rochester-institute-of-technology/) — University, Rochester, United States · 7 works · 4,596 citations - [Universidade Tecnológica Federal do Paraná](https://md.research.elamur.ai/free/en/universities/universidade-tecnologica-federal-do-parana/) — University, Curitiba, Brazil · 4 works · 442 citations Also on the byline: [Manchester Metropolitan University](https://ror.org/02hstj355). ## Cite Cite as: Matthew Shardlow, Richard Evans, Gustavo Henrique Paetzold and Marcos Zampieri. 2021. SemEval-2021 Task 1: Lexical Complexity Prediction. Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{shardlow2021semeval, title = {{SemEval-2021 Task 1: Lexical Complexity Prediction}}, author = {Matthew Shardlow, Richard Evans, Gustavo Henrique Paetzold and Marcos Zampieri}, booktitle = {Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)}, year = {2021}, doi = {10.18653/v1/2021.semeval-1.1}, url = {https://doi.org/10.18653/v1/2021.semeval-1.1} } ``` - [← previous On Hallucination and Predictive Uncertainty in Conditional Language Generation](https://md.research.elamur.ai/free/en/papers/xiao-2021-hallucination-predictive-uncertainty-conditional-language/) - [next → Pre-Training BERT on Arabic Tweets: Practical Considerations](https://md.research.elamur.ai/free/en/papers/abdelali-2021-pre-training-bert-arabic-tweets/) - [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) ## References 2 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Ahmed AbuRa’ed and Horacio Saggion. 2018. LaS-TUS/TALN at Complex Word Identification (CWI) 2018 Shared Task. In Proceedings of the 13th Workshop on Innovative Use of NLP for Building Educational Applications, New Orleans, United States. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ahmed+AbuRa%E2%80%99ed+and+Horacio+Saggion%2E+2018%2E+LaS%2DTUS%2FTALN+at+Complex+Word+Identification+%28CWI%29+2018+Shared+Task%2E+In+Proceedings+of+the+13th+Workshop+on+Innovative+Use+of+NLP+for+Building+Educational+Appl) 2. David Alfter and Ildikó Pilán. 2018. SB@GU at the Complex Word Identification 2018 Shared Task. In Proceedings of the 13th Workshop on Innovative Use of NLP for Building Educational Applications, New Orleans, United States. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=David+Alfter+and+Ildiko%CC%81+Pila%CC%81n%2E+2018%2E+SB%40GU+at+the+Complex+Word+Identification+2018+Shared+Task%2E+In+Proceedings+of+the+13th+Workshop+on+Innovative+Use+of+NLP+for+Building+Educational+Applications%2C+Ne)