--- title: "CTSys at SemEval-2018 Task 3: Irony in Tweets (2018)" description: "This system paper reports an entry in SemEval-2018 Task 3, which asks whether English tweets are ironic… 0.5089 precision. 8 references." authors: ["Myan Sherif", "Sherine Mamdouh", "Wegdan Ghazi"] year: 2018 venue: "Proceedings of the 12th International Workshop on Semantic Evaluation" source: "https://aclanthology.org/S18-1094/" doi: "10.18653/v1/s18-1094" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/sherif-2018-ctsys-semeval-2018-task-3/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sherif-2018-ctsys-semeval-2018-task-3/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/6/) / CTSys at SemEval-2018 Task 3: Irony in Tweets # CTSys at SemEval-2018 Task 3: Irony in Tweets **Authors:** Myan Sherif, Sherine Mamdouh and Wegdan Ghazi · **Year:** 2018 · **Venue:** [Proceedings of the 12th International Workshop on Semantic Evaluation](https://md.research.elamur.ai/free/en/venues/semeval/) ([SemEval 2018](https://md.research.elamur.ai/free/en/venues/semeval/2018/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/S18-1094/) · [PDF](https://aclanthology.org/S18-1094.pdf) · [DOI 10.18653/v1/s18-1094](https://doi.org/10.18653/v1/s18-1094) · [OpenAlex](https://openalex.org/W2806696949) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary This system paper reports an entry in SemEval-2018 Task 3, which asks whether English tweets are ironic (subtask A, binary) and, if so, which kind of irony they show (subtask B: situational, verbal, other, or none). The team used the organizers' corpus of 3,842 tweets and built three separate feature families. The first was a frequency-based bag of words. The second was a rule-based sentiment-contrast detector, which splits a tweet at a conjunction and checks whether the two halves have opposite polarity. The third was averaged Word2Vec embeddings. Each feature family was fed to four classifiers: Naive Bayes, SVM, Decision Trees and 1-NN. A two-stage voting scheme then combined the predictions, first across classifiers and then across feature families. The final scores were modest: F1 of 0.4675 in subtask A (35th place) and 0.3014 in subtask B (22nd place). The paper is still useful as a clear baseline. Its own error analysis points at what flat features miss, namely dependencies between words and representations of whole sentences. ## Key points - The data is the SemEval-2018 Task 3 corpus: 3,842 English tweets collected through the hashtags #irony, #sarcasm and #not. Subtask B labels are 0 = non-ironic, 1 = situational, 2 = verbal, 3 = other. - There are three feature models: a bag-of-words built from frequency hash-maps of ironic vs non-ironic words, with words frequent in both classes discarded; polarity contrast computed with a Sentiment Intensity Analyzer using a 0.5 threshold; and 100-dimensional Word2Vec vectors averaged per tweet. - The classifiers are Naive Bayes, SVM, Decision Trees and K-NN, where k=1 tuned best. On the training set, the best single pairing was bag-of-words with Naive Bayes. - Official test results: - Subtask A: accuracy 0.5089, precision 0.4102, recall 0.5434, F1 0.4675. - Subtask B: accuracy 0.4923, F1 0.3014. - Preprocessing was adapted to each feature. Negations and conjunctions such as "not" and "but" were kept for the sentiment feature, and lemmatization was skipped there because it can alter polarity. ## Interesting ideas - Verbal irony is treated as a clash between two evaluative halves of an utterance: the tweet is cut at a conjunction and the dominant polarity of each side is compared, which gives a cheap and inspectable irony cue. - Preprocessing is made feature-specific: negations and contrastive conjunctions are not thrown away as stop words, and lemmas are avoided where they would erase sentiment-bearing surface forms. - The paper separates situational irony (an event that defies expectation) from verbal irony (saying the opposite of what is meant), and the four-way task scored far below the binary one with purely lexical features. - A two-level vote over feature families and classifiers still landed near 0.47 F1, and the authors name ignored word dependencies as the core weakness. - The full feature-by-classifier grid was scored on training data, which makes it hard to tell real gains from overfitting, as the 1-NN choice suggests. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (frames): prag could compare the evaluative polarity of the two sides of a `conj`/`advcl`/`cc` structure in the UD tree instead of splitting the flat string, and record the clash as an explicit irony signal with its two evidence spans. - **I2** (tokens): the en crate should keep surface form, lemma and negation marking side by side, so that downstream evaluative and irony logic never loses polarity information to normalization. - **I3** (world): situational irony could be detected as a mismatch between an event in the text's world and the expected outcome drawn from compiled knowledge (verb classes, ontology), something string features cannot see. - **I4** (learn): this is a reminder that agreement among many weak, structure-blind witnesses does not create meaning. Silver data from witness voting should be checked against negative controls rather than trusted for agreement alone. - **I5** (learn): every component comparison should be reported on held-out data alongside a trivial baseline, so that a "best" configuration is not an artifact of memorization. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Alexandria University](https://md.research.elamur.ai/free/en/universities/alexandria-university/) — University, Alexandria, Egypt · 9 works · 34 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works 🧭 Topics (2): [English](https://md.research.elamur.ai/free/en/topics/english/) · 1,373 works · [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 608 works ## Cite Cite as: Myan Sherif, Sherine Mamdouh and Wegdan Ghazi. 2018. CTSys at SemEval-2018 Task 3: Irony in Tweets. Proceedings of the 12th International Workshop on Semantic Evaluation. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{sherif2018ctsys, title = {{CTSys at SemEval-2018 Task 3: Irony in Tweets}}, author = {Myan Sherif, Sherine Mamdouh and Wegdan Ghazi}, booktitle = {Proceedings of the 12th International Workshop on Semantic Evaluation}, year = {2018}, doi = {10.18653/v1/s18-1094}, url = {https://doi.org/10.18653/v1/s18-1094} } ``` - [← previous An OpenNMT Model to Arabic Broken Plurals](https://md.research.elamur.ai/free/en/papers/issa-2018-opennmt-model-arabic-broken-plurals/) - [next → Deletion phenomena in comparative constructions: English comparatives in a cross-linguistic perspective](https://md.research.elamur.ai/free/en/papers/atkari-2018-deletion-phenomena-comparative-constructions/) - [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/6/) ## References 8 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Harith Alani, Miriam Fernández, Yulan He and Hassan Saif. 2014. On stopwords, filtering and data sparsity for sentiment analysis of Twitter. In proceedings of LREC 2014, 9 th International Conference on Language Resources and Evaluation:810–817. [find](https://scholar.google.com/scholar?q=Harith+Alani%2C+Miriam+Fern%C3%A1ndez%2C+Yulan+He+and+Hassan+Saif%2E+2014%2E+On+stopwords%2C+filtering+and+data+sparsity+for+sentiment+analysis+of+Twitter%2E+In+proceedings+of+LREC+2014%2C+9+th+International+Conference+) 2. Mathieu Cliche. 2014. The sarcasm detector. URL: [http://www.thesarcasmdetector.com/about/](http://www.thesarcasmdetector.com/about/). [↗](http://www.thesarcasmdetector.com/about/) 3. Aditya Joshi, Mark James Carman and Pushpak Bhattacharyya. 2017. [Automatic Sarcasm Detection: A Survey](https://md.research.elamur.ai/free/en/papers/joshi-2016-automatic-sarcasm-detection-survey/). ACM Computing Surveys 50(5) Article 73, 2017. https://doi.org/10.1145/3124420 [↗](https://doi.org/10.1145/3124420) 4. Chun-Che Peng, Jan Wei Pan and Mohammad Lakis. 2015. Detecting Sarcasm in Text: An Obvious Solution to a Trivial Problem. Stanford CS 229 Machine Learning Final Project. [find](https://scholar.google.com/scholar?q=Chun%2DChe+Peng%2C+Jan+Wei+Pan+and+Mohammad+Lakis%2E+2015%2E+Detecting+Sarcasm+in+Text%3A+An+Obvious+Solution+to+a+Trivial+Problem%2E+Stanford+CS+229+Machine+Learning+Final+Project%2E) 5. Antonio Reyes, Paolo Rosso and Tony Veale. 2012. A multidimensional approach for detecting irony in Twitter. [↗](https://doi.org/10.1007/s10579-012-9196-x) 6. Language Resources & Evaluation, March 2013, 47(1) :239–268. https://doi.org/10.1007/s10579-012-9196-x [↗](https://doi.org/10.1007/s10579-012-9196-x) 7. Cameron Shelley. 2001. The bicoherence theory of situational irony. Cognitive Science 25:775-818. https://doi.org/10.1016/S0364-0213(01)00053-2 [↗](https://doi.org/10.1016/s0364-0213(01)00053-2) 8. Cynthia Van Hee, Els Lefever and Veronique Hoste. 2016. Guidelines for Annotating Irony in Social Media Text. LT3, Department of Translation, Interpreting and Communication, Faculty of Arts, Humanities and Law - Ghent University, Belgium. [find](https://scholar.google.com/scholar?q=Cynthia+Van+Hee%2C+Els+Lefever+and+Veronique+Hoste%2E+2016%2E+Guidelines+for+Annotating+Irony+in+Social+Media+Text%2E+LT3%2C+Department+of+Translation%2C+Interpreting+and+Communication%2C+Faculty+of+Arts%2C+Humanitie) *Updated 24 October 2026*