---
title: "Disambiguating False-Alarm Hashtag Usages in Tweets for Irony"
description: "Irony datasets built from Twitter usually treat any tweet carrying #not, #sarcasm or #irony as ironic. 23 references, cited by 2 papers here."
authors: ["Hen-Hsen Huang", "Chiao-Chen Chen", "Hsin-Hsi Chen"]
year: 2018
venue: "Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)"
source: "https://aclanthology.org/P18-2122/"
doi: "10.18653/v1/p18-2122"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/huang-2018-disambiguating-false-alarm-hashtag-usages/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/huang-2018-disambiguating-false-alarm-hashtag-usages/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/5/) / Disambiguating False-Alarm Hashtag Usages in Tweets for Irony Detection

# Disambiguating False-Alarm Hashtag Usages in Tweets for Irony Detection

**Authors:** [Hen-Hsen Huang](https://md.research.elamur.ai/free/en/authors/hen-hsen-huang/), Chiao-Chen Chen and [Hsin-Hsi Chen](https://md.research.elamur.ai/free/en/authors/hsin-hsi-chen/) · **Year:** 2018 · **Venue:** [Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2018](https://md.research.elamur.ai/free/en/venues/acl/2018/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P18-2122/) · [PDF](https://aclanthology.org/P18-2122.pdf) · [DOI 10.18653/v1/p18-2122](https://doi.org/10.18653/v1/p18-2122) · [OpenAlex](https://openalex.org/W2798322299)

**Citations:** 9 citations (OpenAlex, 2026-10-14) · 2 in our library, of the 141,840 papers whose bibliographies we read

## Summary
Irony datasets built from Twitter usually treat any tweet carrying #not, #sarcasm or #irony as ironic. This self-labeling is noisy in two ways. Some writers use the tag for tweets that are not ironic at all. In other tweets the hashtag is part of the sentence itself, as in "the #irony of…". The authors recovered the original hashtags for SemEval 2018 Task 3 tweets through Twitter search, giving 1,359 human-verified examples. More than 20% of these turned out to be false alarms.

They built a neural classifier that combines several signals:
- separate encoders for the text before and after the hashtag;
- handcrafted position and count features;
- the probabilities a GRU language model over POS tags assigns to three versions of the tweet.

This classifier reaches an F-score of 88.49%. They then used it to filter about 14k self-labeled positive tweets. An attentive-RNN irony detector trained on the smaller, cleaner set beat one trained on all the data. It also beat one trained only on the human-verified tweets. The approach is a cheap way to clean up any self-labeled corpus.

## Key points
- Ground truth: 1,359 tweets (1,083 ironic, 276 false-alarm) from the SemEval 2018 Task 3 data, with hashtags restored via Twitter search; 1,072 further verified ironic tweets were held out for irony-detection testing.
- POS-sequence GRU language model trained on 2 million unrelated English tweets tagged with Stanford CoreNLP; tokenization used the NLTK Penn Treebank Tokenizer; embeddings were 300-d word2vec vectors trained on Google News.
- Hashtag disambiguation F-scores:
  - LR 82.89%, CNN 69.52%, GRU 83.32%, Att.GRU 83.62%;
  - full model with Att.GRU encoder 88.49%;
  - dropping the POS language model lowers it to 84.17% (significant, p \< 0.05).
- Pruning shrank 14,055 hashtag tweets to 4,617 positives. Each set was balanced with an equal number of negatives (training sets of 28,110 vs 9,234), and irony-detection F rose from 81.46% to 83.03%.
- Training only on human-verified data (2,166 instances) gave just 75.26% F. A threshold sweep over the classifier's sigmoid confidence found the default 0.5 cutoff worked best.

## Interesting ideas
- Decide whether a marked token is metadata or a real word by scoring how grammatical the sentence looks in three versions: token deleted, only the marker symbol stripped, or left as is.
- A small gold set can train a filter that cleans a large noisy self-labeled corpus, and the filtered corpus outperforms both the raw noisy data and the gold data alone.
- Treat a calibrated classifier confidence as a knob for instance reliability, then sweep the threshold to trade training-set size against quality.
- Encode the context to the left and right of a target marker separately, and add explicit position features such as first/last token or first/last hashtag.
- Self-applied labels like a trailing #irony are evidence to weigh, not ground truth, because users misunderstand or misuse the label.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tree): Before parsing, the same POS-LM comparison could choose whether a hashtag, @mention or emoji-word enters the UD tree as a content word or is set aside as a discourse/meta attachment.
- **I2** (learn): Silver data agreed by witness parsers could go through a similar learned reliability filter trained on a small gold slice, keeping fewer but trustworthy trees.
- **I3** (learn): Plotting downstream score against filter threshold gives a principled way to set the cutoff for LLM-teacher or silver labels instead of guessing.
- **I4** (frames): The irony module in prag could check where a cue like "#not" sits and whether it is detachable before using it to flip a statement's polarity.
- **I5** (frames): Speech-act and irony detection should treat explicit author tags as one defeasible signal, and confirm them against an actual polarity contrast in the text.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [National Taiwan University](https://md.research.elamur.ai/free/en/universities/national-taiwan-university/) — University, Taipei, Taiwan · 13 works · 1,886 citations

Also on the byline: [National Science and Technology Council](https://ror.org/02kv4zf79).

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 609 works

## Cite

Cite as: Hen-Hsen Huang, Chiao-Chen Chen and Hsin-Hsi Chen. 2018. Disambiguating False-Alarm Hashtag Usages in Tweets for Irony Detection. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{huang2018disambiguating,
  title = {{Disambiguating False-Alarm Hashtag Usages in Tweets for Irony Detection}},
  author = {Hen-Hsen Huang, Chiao-Chen Chen and Hsin-Hsi Chen},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-2122},
  url = {https://doi.org/10.18653/v1/p18-2122}
}
```

- [← previous A POS Tagging Model Designed for Learner English](https://md.research.elamur.ai/free/en/papers/nagata-2018-pos-tagging-model-adapted-learner/)
- [next → Foreign English Accent Adjustment by Learning Phonetic Patterns](https://md.research.elamur.ai/free/en/papers/kitashov-2018-foreign-english-accent-adjustment-learning/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/5/)

## References

23 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751, Doha, Qatar. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=2014+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+pages+1746%E2%80%931751%2C+Doha%2C+Qatar%2E+Association+for+Computational+Linguistics%2E)
2. Steven Bird, Edward Loper, and Ewan Klein. 2009. Natural Language Processing with Python. O’Reilly Media Inc. [find](https://scholar.google.com/scholar?q=Steven+Bird%2C+Edward+Loper%2C+and+Ewan+Klein%2E+2009%2E+Natural+Language+Processing+with+Python%2E+O%E2%80%99Reilly+Media+Inc%2E)
3. Diederik P. Kingma and Jimmy Ba. 2014. A method for stochastic optimization. abs/1412.6980. [↗](https://arxiv.org/abs/1412.6980)
4. Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. CoRR, abs/1406.1078. Adam: CoRR, Hassan H. Malik and Vikas S. Bhardwaj. 2011. Automatic training data cleaning for text classification. In Proceedings of the 2011 IEEE 11th International Conference on Data Mining Workshops, ICDMW ’11, pages 442–449, Washington, DC, USA. IEEE Computer Society. [↗](https://arxiv.org/abs/1406.1078)
5. Keith Cortis, André Freitas, Tobias Daudert, Manuela Huerlimann, Manel Zarrouk, Siegfried Handschuh, and Brian Davis. 2017. Semeval-2017 task 5: Fine-grained sentiment analysis on financial microblogs and news. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 519–535, Vancouver, Canada. Association for Computational Linguistics. [↗](https://doi.org/10.5281/zenodo.1228821)
6. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. [The Stanford CoreNLP natural language processing toolkit](https://md.research.elamur.ai/free/en/papers/manning-2014-stanford-corenlp-natural-language-processing/). In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60.
7. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. [Semi-supervised recognition of sarcasm in twitter and amazon](https://md.research.elamur.ai/free/en/papers/davidov-2010-semi-supervised-recognition-sarcasm-twitter/). In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pages 107–116, Uppsala, Sweden. Association for Computational Linguistics.
8. Tomas Mikolov, Stefan Kombrink, Anoop Deoras, Lukar Burget, and Jan Cernocky. 2011. Rnnlm-recurrent neural network language modeling toolkit. In Proc. of the 2011 ASRU Workshop, pages 196– 201. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Stefan+Kombrink%2C+Anoop+Deoras%2C+Lukar+Burget%2C+and+Jan+Cernocky%2E+2011%2E+Rnnlm%2Drecurrent+neural+network+language+modeling+toolkit%2E+In+Proc%2E+of+the+2011+ASRU+Workshop%2C+pages+196%E2%80%93+201%2E)
9. Andrea Esuli and Fabrizio Sebastiani. 2013. Improving text classification accuracy by training label cleaning. ACM Trans. Inf. Syst., 31(4):19:1–19:28. [↗](https://doi.org/10.1145/2516889)
10. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. [Semeval-2016 task 6: Detecting stance in tweets](https://md.research.elamur.ai/free/en/papers/mohammad-2016-semeval-2016-task-6-detecting/). In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 31– 41, San Diego, California. Association for Computational Linguistics.
11. Fumiyo Fukumoto and Yoshimi Suzuki. 2004. Correcting category errors in text classification. In Proceedings of Coling 2004, pages 868–874, Geneva, Switzerland. COLING. [find](https://scholar.google.com/scholar?q=Fumiyo+Fukumoto+and+Yoshimi+Suzuki%2E+2004%2E+Correcting+category+errors+in+text+classification%2E+In+Proceedings+of+Coling+2004%2C+pages+868%E2%80%93874%2C+Geneva%2C+Switzerland%2E+COLING%2E)
12. Debanjan Ghosh, Weiwei Guo, and Smaranda Muresan. 2015. [Sarcastic or not: Word embeddings to predict the literal or sarcastic meaning of words](https://md.research.elamur.ai/free/en/papers/ghosh-2015-sarcastic-or-not-word-embeddings/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1003–1012, Lisbon, Portugal. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d15-1116)
13. Alexandru Niculescu-Mizil and Rich Caruana. 2005. Predicting good probabilities with supervised learning. In Proceedings of the 22Nd International Conference on Machine Learning, ICML ’05, pages 625–632, New York, NY, USA. ACM. [find](https://scholar.google.com/scholar?q=Alexandru+Niculescu%2DMizil+and+Rich+Caruana%2E+2005%2E+Predicting+good+probabilities+with+supervised+learning%2E+In+Proceedings+of+the+22Nd+International+Conference+on+Machine+Learning%2C+ICML+%E2%80%9905%2C+pages+625%E2%80%936)
14. Lotem Peled and Roi Reichart. 2017. [Sarcasm sign: Interpreting sarcasm with sentiment based monolingual machine translation](https://md.research.elamur.ai/free/en/papers/peled-2017-sarcasm-sign-interpreting-sarcasm-sentiment/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1690–1700, Vancouver, Canada. Association for Computational Linguistics.
15. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1321–1330, International Convention Centre, Sydney, Australia. PMLR. [↗](https://doi.org/10.48550/arxiv.1706.04599)
16. Ashequl Qadir and Ellen Riloff. 2014. Learning emotion indicators from tweets: Hashtags, hashtag patterns, and phrases. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1203–1209, Doha, Qatar. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ashequl+Qadir+and+Ellen+Riloff%2E+2014%2E+Learning+emotion+indicators+from+tweets%3A+Hashtags%2C+hashtag+patterns%2C+and+phrases%2E+In+Proceedings+of+the+2014+Conference+on+Empirical+Methods+in+Natural+Language+P)
17. Cynthia Van Hee, Els Lefever, and Vronique Hoste. 2018. [Semeval-2018 task 3: Irony detection in english tweets](https://md.research.elamur.ai/free/en/papers/hee-2018-semeval-2018-task-3-irony/). In Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), New Orleans, LA, USA.
18. Parinaz Sobhani, Diana Inkpen, and Xiaodan Zhu. 2017. [A dataset for multi-target stance detection](https://md.research.elamur.ai/free/en/papers/sobhani-2017-dataset-multi-target-stance-detection/). In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 551–557, Valencia, Spain. Association for Computational Linguistics.
19. Yu-Hsiang Huang, Hen-Hsen Huang, and Hsin-Hsi Chen. 2017. Irony detection with attentive recurrent neural networks. In European Conference on Information Retrieval, pages 534–540. Springer. [find](https://scholar.google.com/scholar?q=Yu%2DHsiang+Huang%2C+Hen%2DHsen+Huang%2C+and+Hsin%2DHsi+Chen%2E+2017%2E+Irony+detection+with+attentive+recurrent+neural+networks%2E+In+European+Conference+on+Information+Retrieval%2C+pages+534%E2%80%93540%2E+Springer%2E)
20. Yoon Kim. 2014. [Convolutional neural networks for sentence classification](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/). In Proceedings of the Emilio Sulis, Delia Irazú Hernández Farı́as, Paolo Rosso, Viviana Patti, and Giancarlo Ruffo. 2016. Figurative messages and affect in twitter. Know.-Based Syst., 108(C):132–143.
21. Cynthia Van Hee, Els Lefever, and Veronique Hoste. 2016a. [Exploring the realization of irony in twitter data](https://md.research.elamur.ai/free/en/papers/hee-2016-exploring-realization-irony-twitter-data/). In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC ’16), pages 1795–1799. European Language Resources Association (ELRA).
22. Cynthia Van Hee, Els Lefever, and Veronique Hoste. 2016b. [Monday mornings are my fave :) #not exploring the automatic recognition of irony in english tweets](https://md.research.elamur.ai/free/en/papers/hee-2016-monday-mornings-are-my-fave/). In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2730–2739, Osaka, Japan.
23. The COLING 2016 Organizing Committee. [find](https://scholar.google.com/scholar?q=The+COLING+2016+Organizing+Committee%2E)

*Updated 24 October 2026*