--- title: "SemEval-2022 Task 2 — Madabushi 2022" description: "Idiomatic multiword expressions, such as noun compounds whose meaning cannot be derived from their parts, remain difficult for language models." authors: ["Harish Tayyar Madabushi", "Edward Gow-Smith", "Marcos Garcia", "Carolina Scarton", "Marco Idiart", "Aline Villavicencio"] year: 2022 venue: "Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)" source: "https://aclanthology.org/2022.semeval-1.13/" doi: "10.18653/v1/2022.semeval-1.13" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/madabushi-2022-semeval-2022-task-2-multilingual/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/madabushi-2022-semeval-2022-task-2-multilingual/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2022](https://md.research.elamur.ai/free/papers/2022/page/3/) / SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding # SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding **Authors:** [Harish Tayyar Madabushi](https://md.research.elamur.ai/free/authors/harish-tayyar-madabushi/), [Edward Gow-Smith](https://md.research.elamur.ai/free/authors/edward-gow-smith/), [Marcos Garcia](https://md.research.elamur.ai/free/authors/marcos-garcia/), [Carolina Scarton](https://md.research.elamur.ai/free/authors/carolina-scarton/), [Marco Idiart](https://md.research.elamur.ai/free/authors/marco-idiart/), [Aline Villavicencio](https://md.research.elamur.ai/free/authors/aline-villavicencio/) · **Year:** 2022 · **Venue:** Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2022.semeval-1.13/) · [PDF](https://aclanthology.org/2022.semeval-1.13.pdf) · [DOI 10.18653/v1/2022.semeval-1.13](https://doi.org/10.18653/v1/2022.semeval-1.13) · [OpenAlex](https://openalex.org/W4287854311) **Citations:** 34 citations (openalex-snapshot · 2026-10-14) · [74](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Idiomatic multiword expressions, such as noun compounds whose meaning cannot be derived from their parts, remain difficult for language models. This shared task asks two things: whether systems can tell idiomatic uses from literal ones, and whether their sentence embeddings reflect the intended meaning of such expressions. The organisers extended an earlier English–Portuguese resource with Galician. They chose potentially ambiguous compounds, collected sentences with surrounding context from news, Wikipedia or CC-100, wrote short paraphrases for every sense and had two annotators label each example. Subtask A is binary idiomaticity classification in Zero Shot and One Shot settings. Subtask B is a semantic similarity task scored with Spearman correlation, in Pre Train and Fine Tune settings. About 100 people in 25 teams took part, and the best systems beat the transformer baselines. The analysis shows that generalising to unseen idioms and exploiting a single example call for quite different strategies. The task supplies a multilingual benchmark and baselines for studying non-compositional meaning, including in a low-resource language. ## Key points - The dataset has 8,683 entries: English 5,352, Portuguese 2,555 and Galician 776. Galician had only 63 training items (One Shot data), so the test checks cross-lingual transfer to a low-resource language. - English and Portuguese compounds came from the Noun Compound Senses dataset and were spread across compositionality levels. Galician noun-adjective compounds were mined from Wikipedia and CC-100. 'Proper Noun' and 'Meta Usage' were added as extra labels. - Participation reached roughly 100 registrants in 25 teams. They made over 650 submissions in the practice phase and over 150 in the evaluation phase. The CodaLab competition remains open post-evaluation. - In Subtask A, regularisation, data augmentation and dropout helped most in the Zero Shot setting. Larger, ensembled or adversarially trained models did best with One Shot data. A linguistically motivated span–context similarity method placed third in Zero Shot, about 11 points behind the leader. - In Subtask B, the top Pre Train score was 0.6402, against a baseline of 0.4810. That system learned single-token MWE representations. YNU-HPCC led Fine Tune with 0.6648, against a baseline of 0.5951. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) ## Cited by Cited by 74 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 63 not yet. - [A Survey of Idiom Datasets for Psycholinguistic and Computational Research](https://md.research.elamur.ai/free/papers/flor-2025-survey-idiom-datasets-psycholinguistic-computational/) - [ID10M-JAM: Stress-Testing Idiom Identification Under Challenging Context](https://md.research.elamur.ai/free/papers/hashiloni-2026-id10m-jam-stress-testing-idiom/) - [Comparative Study of Multilingual Idioms and Similes in Large Language Models](https://md.research.elamur.ai/free/papers/khoshtab-2025-comparative-study-multilingual-idioms-similes/) - [Stands to Reason: Investigating the Effect of Reasoning on Idiomaticity Detection](https://md.research.elamur.ai/free/papers/phelps-2026-stands-reason-investigating-effect-reasoning/) - [SemEval-2025 Task 1: AdMIRe - Advancing Multimodal Idiomaticity Representation](https://md.research.elamur.ai/free/papers/pickard-2025-semeval-2025-task-1-admire/) - [PARSEME 2.0 Multilingual Corpus of Multiword Expressions](https://md.research.elamur.ai/free/papers/savary-2026-parseme-2-0-multilingual-corpus/) - [Edition 2.0 of the PARSEME shared task on multilingual identification and paraphrasing of multiword expressions](https://md.research.elamur.ai/free/papers/scholivet-2026-edition-2-0-parseme-shared/) - [A Parallel Cross-Lingual Benchmark for Multimodal Idiomaticity Understanding](https://md.research.elamur.ai/free/papers/selamet-2026-parallel-cross-lingual-benchmark-multimodal/) - [MultiCoPIE: A Multilingual Corpus of Potentially Idiomatic Expressions for Cross-lingual PIE Disambiguation](https://md.research.elamur.ai/free/papers/sentsova-2025-multicopie-multilingual-corpus-potentially-idiomatic/) - [ITUNLP at MWE-2026 AdMIRe 2: A Zero-Shot LLM Pipeline for Multimodal Idiom Understanding and Ranking](https://md.research.elamur.ai/free/papers/site-2026-itunlp-mwe-2026-admire-2/) - [Token-level Identification of Multiword Expressions using Pre-trained Multilingual Language Models](https://md.research.elamur.ai/free/papers/swaminathan-2023-token-level-identification-multiword-expressions/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{madabushi2022semeval, title = {{SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding}}, author = {Madabushi, Harish Tayyar and Gow-Smith, Edward and Garcia, Marcos and Scarton, Carolina and Idiart, Marco and Villavicencio, Aline}, booktitle = {Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)}, year = {2022}, doi = {10.18653/v1/2022.semeval-1.13}, url = {https://doi.org/10.18653/v1/2022.semeval-1.13} } ``` - [← previous Semantic properties of English nominal pluralization: Insights from word embeddings](https://md.research.elamur.ai/free/papers/shafaei-2022-bajestan-semantic-properties-english-nominal-pluralization/) - [next → SemEval-2022 Task 3: PreTENS — Evaluating Neural Networks on Presuppositional Semantic Knowledge](https://md.research.elamur.ai/free/papers/zamparelli-2022-semeval-2022-task-3-pretens/) - [↑ back to papers of 2022](https://md.research.elamur.ai/free/papers/2022/page/3/) ## References 2 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Timothy Baldwin and Su Nam Kim. 2010. Multiword expressions. In Nitin Indurkhya and Fred J. Damerau, editors, Handbook of Natural Language Processing, pages 267–292. CRC Press. [find](https://scholar.google.com/scholar?q=Timothy+Baldwin+and+Su+Nam+Kim%2E+2010%2E+Multiword+expressions%2E+In+Nitin+Indurkhya+and+Fred+J%2E+Damerau%2C+editors%2C+Handbook+of+Natural+Language+Processing%2C+pages+267%E2%80%93292%2E+CRC+Press%2E) 2. Douglas Biber, Stig Johansson, Geoffrey Leech, Susan Conrad, and Edward Finegan. 1999. Longman Grammar of Spoken and Written English. Pearson Education Ltd. [find](https://scholar.google.com/scholar?q=Douglas+Biber%2C+Stig+Johansson%2C+Geoffrey+Leech%2C+Susan+Conrad%2C+and+Edward+Finegan%2E+1999%2E+Longman+Grammar+of+Spoken+and+Written+English%2E+Pearson+Education+Ltd%2E)