--- title: "SarcasmDet at SemEval-2022 Task 6 — Abdullah 2022" description: "The paper describes SarcasmDet, a team entry in SemEval-2022 Task 6 (iSarcasmEval). 17 references, cited by 1 paper here." authors: ["Malak Abdullah", "Dalya Alnore", "Safa Swedat", "Jumana Khrais", "Mahmoud Al-Ayyoub"] year: 2022 venue: "Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)" source: "https://aclanthology.org/2022.semeval-1.144/" doi: "10.18653/v1/2022.semeval-1.144" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/abdullah-2022-sarcasmdet-semeval-2022-task-6/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/abdullah-2022-sarcasmdet-semeval-2022-task-6/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/) / SarcasmDet at SemEval-2022 Task 6: Detecting Sarcasm using Pre-trained Transformers in English and Arabic Languages # SarcasmDet at SemEval-2022 Task 6: Detecting Sarcasm using Pre-trained Transformers in English and Arabic Languages **Authors:** [Malak Abdullah](https://md.research.elamur.ai/free/en/authors/malak-abdullah/), Dalya Alnore, Safa Swedat, Jumana Khrais and [Mahmoud Al-Ayyoub](https://md.research.elamur.ai/free/en/authors/mahmoud-al-ayyoub/) · **Year:** 2022 · **Venue:** Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2022.semeval-1.144/) · [PDF](https://aclanthology.org/2022.semeval-1.144.pdf) · [DOI 10.18653/v1/2022.semeval-1.144](https://doi.org/10.18653/v1/2022.semeval-1.144) · [OpenAlex](https://openalex.org/W4287888532) **Citations:** 1 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 140,884 papers whose bibliographies we read ## Summary The paper describes SarcasmDet, a team entry in SemEval-2022 Task 6 (iSarcasmEval). The team took part only in subtask A, which asks whether a tweet is sarcastic, for both English and Arabic. Both training sets are small and skewed toward non-sarcastic tweets, so the official measure is F1 on the sarcastic class rather than accuracy. For English, the authors fine-tuned BERT and RoBERTa with most lower layers frozen and combined several runs through a weighted sum of predictions. For Arabic, they fine-tuned AraBERT (large v2) and MARBERT and merged them by hard majority voting. They also tried adding tweets from ArSarcasm-v2 as extra training data, but this did not help. The Arabic system placed seventh of 32 teams. The official English submission scored much lower, and a better ensemble was built only after the deadline. The work shows that simple ensembles of pretrained transformers give modest but steady gains on a hard pragmatic task, though absolute scores remain low. ## Key points - Data: the English set has 3467 tweets (867 sarcastic, 2600 not). The Arabic set has 3102 tweets (745 sarcastic, 2357 not), with dialect labels and rephrasings. - English setup and scores: - BERT with only the top 2 of 12 layers trainable (batch 6, 3 epochs) reached F1-sarcastic 0.39. - RoBERTa with the top 3 layers trainable (batch 12, 3 epochs) reached 0.41. - A weighted ensemble of one BERT and four RoBERTa runs, one RoBERTa run at weight 0.5, reached about 0.43 (0.4322). - English ranking: the official English submission placed 17th of 42 teams with 0.3561. The 0.43 result came after the deadline. The paper puts it at seventh in the abstract but at eighth in the results section. The task's top English score was 0.6052. - Arabic setup and scores: - AraBERT alone scored 0.4255. - Hard voting of AraBERT (learning rate 2e-5) and MARBERT (learning rate 1e-5), both trained for 5 epochs with batch 8, scored 0.4305 and placed seventh of 32 teams. - Augmentation: adding ArSarcasm-v2 data lowered results, with the best augmented model (MARBERT) scoring 0.4065. All models were built with HuggingFace and Simple Transformers. ## Interesting ideas - Freeze most lower transformer layers and train only the top two to four, which works on a few thousand examples and reduces overfitting and compute. - Combine runs of different models with explicit per-model weights, including fractional ones, instead of trusting a single best checkpoint. - Test whether more data actually helps: here, related out-of-task sarcasm data made scores worse. - On imbalanced data, score the rare positive class with its own F1 instead of overall accuracy. - Pair a model built for a language variety (MARBERT, focused on dialects) with a general one (AraBERT) and let them vote. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (tree): gparse could be adapted to small treebanks or a new UD dialect by retraining only its upper layers, keeping the general encoder fixed. - **I2** (learn): witness parsers that vote on silver data could carry tuned weights, so a weaker witness still counts but cannot outvote stronger ones. - **I3** (learn): every new silver or LLM-teacher data source should face an ablation check against the baseline before it is merged into training. - **I4** (frames): the prag irony and sarcasm detector should report precision, recall and F1 on ironic cases alone, since a model that always says "literal" would look deceptively good on accuracy. - **I5** (tokens): a general English tagger could vote with one tuned on informal or social-media text, so slang-heavy inputs get a second opinion before parsing. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) Topics: [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · [🐪 Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) ## Institutions - [Jordan University of Science and Technology](https://md.research.elamur.ai/free/en/universities/jordan-university-of-science-and-technology/) — University, Irbid, Jordan · 9 works · 1,785 citations ## Cite Cite as: Malak Abdullah, Dalya Alnore, Safa Swedat, Jumana Khrais and Mahmoud Al-Ayyoub. 2022. SarcasmDet at SemEval-2022 Task 6: Detecting Sarcasm using Pre-trained Transformers in English and Arabic Languages. Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{abdullah2022sarcasmdet, title = {{SarcasmDet at SemEval-2022 Task 6: Detecting Sarcasm using Pre-trained Transformers in English and Arabic Languages}}, author = {Malak Abdullah, Dalya Alnore, Safa Swedat, Jumana Khrais and Mahmoud Al-Ayyoub}, booktitle = {Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)}, year = {2022}, doi = {10.18653/v1/2022.semeval-1.144}, url = {https://doi.org/10.18653/v1/2022.semeval-1.144} } ``` - [← previous Roadblocks in Gender Bias Measurement for Diachronic Corpora](https://md.research.elamur.ai/free/en/papers/alshahrani-2022-roadblocks-gender-bias-measurement-diachronic/) - [next → Serial Verb Constructions in North-West Semitic languages: From a synchronic radiation back to the 'Big Bang'](https://md.research.elamur.ai/free/en/papers/andrason-2022-serial-verb-constructions-north-west/) - [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/) ## References 17 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020. Arbert & marbert: deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785. [↗](https://arxiv.org/abs/2101.01785) [our card](https://md.research.elamur.ai/free/en/papers/abdul-mageed-2021-arbert-marbert-deep-bidirectional-transformers/) 2. Ibrahim Abu Farha, Silviu Oprea, Steven Wilson, and Walid Magdy. 2022. SemEval-2022 Task 6: iSarcasmEval, Intended Sarcasm Detection in English and Arabic. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/farha-2022-semeval-2022-task-6-isarcasmeval/) 3. Dia Abujaber, Ahmed Qarqaz, and Malak A Abdullah. 2021. Lecun at semeval-2021 task 6: Detecting persuasion techniques in text using ensembled pretrained transformers and data augmentation. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 1068–1074. Wissam Antoun, Fady Baly, and Hazem Hajj. Arabert: Transformer-based model for arabic language under-Rank 7th F1-sarcastic 0.4305 F-score 0.6114 Precision 0.6240 Recall 0.7412 Accuracy 0.6957 *Table 8: Official results on subtask 1 (sarcasm Detection) test set.* Experiment 1 2 3 4 5 6 7 8 Model large-arabertv2 large-arabertv2 large-arabertv02 marbert marbert large-arabertv2 marbert SarcasmDet F1-sarcastic 0.3847 0.4255 0.4166 0.3650 0.4015 0.3276 0.4065 0.4305 [find](https://scholar.google.com/scholar?q=Dia+Abujaber%2C+Ahmed+Qarqaz%2C+and+Malak+A+Abdullah%2E+2021%2E+Lecun+at+semeval%2D2021+task+6%3A+Detecting+persuasion+techniques+in+text+using+ensembled+pretrained+transformers+and+data+augmentation%2E+In+Proceedi) 4. Prakash M Nadkarni, Lucila Ohno-Machado, and Wendy W Chapman. 2011. Natural language processing: an introduction. Journal of the American Medical Informatics Association, 18(5):544–551. [find](https://scholar.google.com/scholar?q=Prakash+M+Nadkarni%2C+Lucila+Ohno%2DMachado%2C+and+Wendy+W+Chapman%2E+2011%2E+Natural+language+processing%3A+an+introduction%2E+Journal+of+the+American+Medical+Informatics+Association%2C+18%285%29%3A544%E2%80%93551%2E) 5. Neha Pawar and Sukhada Bhingarkar. 2020. Machine learning based sarcasm detection on twitter data. In 2020 5th International Conference on Communication and Electronics Systems (ICCES), pages 957– 961. IEEE. *Table 9: Results in all experiments, SarcasmDet is an ensemble for experiments 2,3,4,5 and 7* standing. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. [find](https://scholar.google.com/scholar?q=Neha+Pawar+and+Sukhada+Bhingarkar%2E+2020%2E+Machine+learning+based+sarcasm+detection+on+twitter+data%2E+In+2020+5th+International+Conference+on+Communication+and+Electronics+Systems+%28ICCES%29%2C+pages+957%E2%80%93+961) 6. Steven Bird. 2006. Nltk: the natural language toolkit. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pages 69–72. [↗](https://doi.org/10.3115/1225403.1225421) [our card](https://md.research.elamur.ai/free/en/papers/bird-2006-nltk-natural-language-toolkit/) 7. Mondher Bouazizi and Tomoaki Otsuki Ohtsuki. 2016. A pattern-based approach for sarcasm detection on twitter. IEEE Access, 4:5477–5488. [find](https://scholar.google.com/scholar?q=Mondher+Bouazizi+and+Tomoaki+Otsuki+Ohtsuki%2E+2016%2E+A+pattern%2Dbased+approach+for+sarcasm+detection+on+twitter%2E+IEEE+Access%2C+4%3A5477%E2%80%935488%2E) 8. Shelley Channon, Asa Pellijeff, and Andrea Rule. 2005. Social cognition after head injury: Sarcasm and theory of mind. Brain and Language, 93:123–134. [find](https://scholar.google.com/scholar?q=Shelley+Channon%2C+Asa+Pellijeff%2C+and+Andrea+Rule%2E+2005%2E+Social+cognition+after+head+injury%3A+Sarcasm+and+theory+of+mind%2E+Brain+and+Language%2C+93%3A123%E2%80%93134%2E) 9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [↗](https://arxiv.org/abs/1810.04805) [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/) 10. Rolandos Alexandros Potamias, Georgios Siolas, and Andreas-Georgios Stafylopatis. 2020. A transformer-based approach to irony and sarcasm detection. Neural Computing and Applications, 32(23):17309– 17320. [our card](https://md.research.elamur.ai/free/en/papers/potamias-2019-transformer-based-approach-irony-sarcasm/) 11. Ahmed Qarqaz, Dia Abujaber, and Malak Abdullah. 2021. R00 at nlp4if-2021 fighting covid-19 info-demic with transformers and more transformers. In Proceedings of the Fourth Workshop on NLP for Internet Freedom: Censorship, Disinformation, and Propaganda, pages 104–109. [find](https://scholar.google.com/scholar?q=Ahmed+Qarqaz%2C+Dia+Abujaber%2C+and+Malak+Abdullah%2E+2021%2E+R00+at+nlp4if%2D2021+fighting+covid%2D19+info%2Ddemic+with+transformers+and+more+transformers%2E+In+Proceedings+of+the+Fourth+Workshop+on+NLP+for+Internet) 12. Ashwin Rajadesingan, Reza Zafarani, and Huan Liu. 2015. Sarcasm detection on twitter: A behavioral modeling approach. In Proceedings of the eighth ACM international conference on web search and data mining, pages 97–106. [find](https://scholar.google.com/scholar?q=Ashwin+Rajadesingan%2C+Reza+Zafarani%2C+and+Huan+Liu%2E+2015%2E+Sarcasm+detection+on+twitter%3A+A+behavioral+modeling+approach%2E+In+Proceedings+of+the+eighth+ACM+international+conference+on+web+search+and+data+m) 13. Cynthia Van Hee, Gilles Jacobs, Chris Emmery, Bart Desmet, Els Lefever, Ben Verhoeven, Guy De Pauw, Walter Daelemans, and Véronique Hoste. 2018. Automatic detection of cyberbullying in social media text. PloS one, 13(10):e0203794. [find](https://scholar.google.com/scholar?q=Cynthia+Van+Hee%2C+Gilles+Jacobs%2C+Chris+Emmery%2C+Bart+Desmet%2C+Els+Lefever%2C+Ben+Verhoeven%2C+Guy+De+Pauw%2C+Walter+Daelemans%2C+and+V%C3%A9ronique+Hoste%2E+2018%2E+Automatic+detection+of+cyberbullying+in+social+media+te) 14. Dalya Faraj and Malak Abdullah. 2021. Sarcasmdet at sarcasm detection task 2021 in arabic using arabert pretrained model. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 345– 350. [our card](https://md.research.elamur.ai/free/en/papers/faraj-2021-sarcasmdet-sarcasm-detection-task-2021/) 15. Ibrahim Abu Farha and Walid Magdy. 2020. From arabic sentiment analysis to sarcasm detection: The arsarcasm dataset. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 32–39. [our card](https://md.research.elamur.ai/free/en/papers/farha-2020-arabic-sentiment-analysis-sarcasm-detection/) 16. Debanjan Ghosh, Avijit Vajpayee, and Smaranda Muresan. 2020. A report on the 2020 sarcasm detection shared task. arXiv preprint arXiv:2005.05814. [↗](https://arxiv.org/abs/2005.05814) [our card](https://md.research.elamur.ai/free/en/papers/ghosh-2020-report-2020-sarcasm-detection-shared/) 17. Jihen Karoui, Farah Banamara Zitoune, and Veronique Moriceau. 2017. Soukhria: Towards an irony detection system for arabic in social media. Procedia Computer Science, 117:161–168. [↗](https://doi.org/10.1016/j.procs.2017.10.105)