--- title: "PunFields at SemEval-2018 Task 3 — paper by Mikhalkova et al. (2018)" description: "They test whether PunFields, their SemEval-2017 pun classifier, transfers to irony detection in English tweets… 0.6495 F1. 1 reference." authors: ["Elena Mikhalkova", "Yuri Karyakin", "Alexander Voronov", "Dmitry Grigoriev", "Artem Leoznov"] year: 2018 venue: "Proceedings of the 12th International Workshop on Semantic Evaluation" source: "https://aclanthology.org/S18-1088/" doi: "10.18653/v1/s18-1088" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/mikhalkova-2018-punfields-semeval-2018-task-3/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mikhalkova-2018-punfields-semeval-2018-task-3/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/6/) / PunFields at SemEval-2018 Task 3: Detecting Irony by Tools of Humor Analysis # PunFields at SemEval-2018 Task 3: Detecting Irony by Tools of Humor Analysis **Authors:** [Elena Mikhalkova](https://md.research.elamur.ai/free/en/authors/elena-mikhalkova/), [Yuri Karyakin](https://md.research.elamur.ai/free/en/authors/yuri-karyakin/), Alexander Voronov, Dmitry Grigoriev and Artem Leoznov · **Year:** 2018 · **Venue:** [Proceedings of the 12th International Workshop on Semantic Evaluation](https://md.research.elamur.ai/free/en/venues/semeval/) ([SemEval 2018](https://md.research.elamur.ai/free/en/venues/semeval/2018/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/S18-1088/) · [PDF](https://aclanthology.org/S18-1088.pdf) · [DOI 10.18653/v1/s18-1088](https://doi.org/10.18653/v1/s18-1088) · [OpenAlex](https://openalex.org/W2805617069) **Citations:** 1 citations (OpenAlex, 2026-10-14) ## Summary The authors look for a single mechanism that detects intentional lexical ambiguity across kinds of creative language. They test whether PunFields, their SemEval-2017 pun classifier, transfers to irony detection in English tweets (SemEval-2018 Task 3). Their argument is that both puns and irony set two competing meanings, or "scripts", inside one utterance. Irony is harder to spot because its opposition is a binary flip (fun vs. not fun) rather than a clash between two domains, so it often depends on markers such as hashtags and emoji. The method has three stages: - A heavy normalization pipeline cleans the tweets. - Each tweet becomes a 39-dimensional count vector over the Sections of Roget's Thesaurus. - A linear SVM classifies the vectors. On the official test set the constrained run scored F1=0.549, against 0.596 in preliminary cross-validation. It beat the unconstrained bag-of-words run on recall and F1 but lost on accuracy and precision. The work matters because it shows the limits of transferring a topic-clash detector from puns to irony. ## Key points - PunFields maps each non-stopword word or collocation to its Roget's Thesaurus Sections, removes duplicates, and counts Section hits into a vector of length 39. - Constrained submission (Gold training data only): Acc 0.5765, P 0.4753, R 0.6495, F1 0.5489. - Unconstrained submission (bag-of-words SVM trained with extra tweets from other SemEval tasks): Acc 0.6033, P 0.5000, R 0.5563, F1 0.5266. - For comparison, PunFields on SemEval-2017 puns reached F1 0.7651 on homographic puns and 0.6661 on heterographic puns. - Preprocessing works as follows: - Emoji are rewritten as their names, and words with stretched letters are shortened. - Hashtags and mentions are split on capitals and digits. - Glued names are split by checking the NLTK names list before calling Wordninja. - Slang abbreviations are expanded. - Alternatives the authors tried: - Keras networks peaked at F1 about 0.56. - Summed 300-dimensional word2vec vectors from Gensim gave no meaningful gain. - Merging Sections into fewer classes did not help, and performance dropped sharply at around 4 classes. ## Interesting ideas - Puns and irony can both be framed as two semantic groups coexisting in one utterance, so a detector of competing topical clusters is a candidate shared cue for several figurative phenomena. - A small, fixed, top-down set of thesaurus classes gives a sentence profile in which each dimension can be read and inspected, unlike opaque embeddings. - Irony markers such as #not or an unamused emoji tend to come at the end of the utterance, much as a pun's double-sense word usually does, so position is a useful cue. - Splitting glued tokens is safer when a names dictionary runs before a generic word segmenter, because segmenters mangle proper names. - The authors saw an optimistic cross-validation score on a small subset drop once the full, noisier set was used. This shows that a good internal score can hide poor generalization. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (frames): the irony module in prag could flag sentences whose content words split into two strongly represented semantic domains, then pass that clash as an explainable irony or wordplay hypothesis. - **I2** (world): coarse class labels derived from the WordNet/FrameNet ontology could give each text a readable "topic histogram" that later stages can cite when explaining why two parts of a sentence pull apart. - **I3** (frames): speech-act and irony rules could weight utterance-final hashtags, emoji names and evaluative words more heavily when deciding whether a statement's literal polarity should be inverted. - **I4** (tokens): the en tokenizer could check gazetteer or name lists first for hashtags and camel-case or glued tokens, then fall back to segmentation, and also normalize stretched letters before tagging. - **I5** (learn): evaluation should always report a gap measure between internal cross-validation and a truly held-out split, so that apparent gains on small silver subsets are not trusted blindly. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions Also on the byline: [University of Tyumen](https://ror.org/05vehv290). 📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 798 works · [💭 Social Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/social-psychology/) · 129 works 🧭 Topics (2): [English](https://md.research.elamur.ai/free/en/topics/english/) · 1,373 works · [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 608 works ## Cite Cite as: Elena Mikhalkova, Yuri Karyakin, Alexander Voronov, Dmitry Grigoriev and Artem Leoznov. 2018. PunFields at SemEval-2018 Task 3: Detecting Irony by Tools of Humor Analysis. Proceedings of the 12th International Workshop on Semantic Evaluation. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{mikhalkova2018punfields, title = {{PunFields at SemEval-2018 Task 3: Detecting Irony by Tools of Humor Analysis}}, author = {Elena Mikhalkova, Yuri Karyakin, Alexander Voronov, Dmitry Grigoriev and Artem Leoznov}, booktitle = {Proceedings of the 12th International Workshop on Semantic Evaluation}, year = {2018}, doi = {10.18653/v1/s18-1088}, url = {https://doi.org/10.18653/v1/s18-1088} } ``` - [← previous Lancaster at SemEval-2018 Task 3: Investigating Ironic Features in English Tweets](https://md.research.elamur.ai/free/en/papers/dearden-2018-lancaster-semeval-2018-task-3/) - [next → Role of Intonation in Scoring Spoken English](https://md.research.elamur.ai/free/en/papers/nigam-2018-role-intonation-scoring-spoken-english/) - [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/6/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work. 1. Francesco Barbieri and Horacio Saggion. 2014. Modelling irony in Twitter. In Proceedings of the Student Research Workshop at the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 56–64. [our card](https://md.research.elamur.ai/free/en/papers/barbieri-2014-modelling-irony-twitter/) *Updated 24 October 2026*