---
title: "The Iron(ic) Melting Pot — Loakman 2023"
description: "Human judgement is usually treated as the most trustworthy way to assess generated text."
authors: ["Tyler Loakman", "Aaron Maladry", "Chenghua Lin"]
year: 2023
venue: "Findings of the Association for Computational Linguistics: EMNLP 2023"
source: "https://aclanthology.org/2023.findings-emnlp.444/"
doi: "10.18653/v1/2023.findings-emnlp.444"
arxiv: "2311.05552"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/loakman-2023-iron-ic-melting-pot-reviewing/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/loakman-2023-iron-ic-melting-pot-reviewing/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2023](https://md.research.elamur.ai/free/papers/2023/page/6/) / The Iron(ic) Melting Pot: Reviewing Human Evaluation in Humour, Irony and Sarcasm Generation

# The Iron(ic) Melting Pot: Reviewing Human Evaluation in Humour, Irony and Sarcasm Generation

**Authors:** Tyler Loakman, Aaron Maladry, [Chenghua Lin](https://md.research.elamur.ai/free/authors/chenghua-lin/) · **Year:** 2023 · **Venue:** Findings of the Association for Computational Linguistics: EMNLP 2023 · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.findings-emnlp.444/) · [PDF](https://aclanthology.org/2023.findings-emnlp.444.pdf) · [DOI 10.18653/v1/2023.findings-emnlp.444](https://doi.org/10.18653/v1/2023.findings-emnlp.444) · [arXiv 2311.05552](https://arxiv.org/abs/2311.05552) · [OpenAlex](https://openalex.org/W4389518843)

**Citations:** 6 citations (openalex-snapshot · 2026-10-14) · 12 in our library, of the 140673 papers whose bibliographies we read

## Summary
Human judgement is usually treated as the most trustworthy way to assess generated text. This position paper argues that this trust is especially fragile for humour, irony and sarcasm. How people read these forms depends heavily on who they are: their politics, gender, culture, region, background knowledge and language proficiency.

The authors first review each phenomenon. They use example jokes and ironic statements to show how readers with different backgrounds may reach different interpretations. They then survey 17 ACL Anthology papers (2018–2023) on generating these language forms and check how openly each describes its human evaluation. The survey finds that evaluator demographics are rarely disclosed and that crowdsourcing is the dominant way of recruiting evaluators. Compensation, agreement statistics and annotator training are also often left out.

In response, the authors propose an "evaluation statement", modelled on data statements. They also recommend stratified sampling to build more diverse evaluator panels. They frame disagreement between evaluators as useful information rather than noise. The work matters because unreported panel composition makes results on subjective generation tasks hard to interpret or replicate.

## Key points
- The survey covers 17 NLG papers on humour, irony and sarcasm generation, drawn from the ACL Anthology (2018–2023).
- Reporting rates in the surveyed papers are roughly as follows:
  - Recruitment method: about 71%.
  - Compensation: about 30%.
  - Inter-annotator agreement (IAA): about 53%.
  - Training materials: about 24%, i.e. only 4 of 17 papers.
- The proposed evaluation statement, inspired by Bender and Friedman (2018), has four parts:
  - Logistics: number of evaluators, sample split, recruitment and pay.
  - Demographics: age, L1 and proficiency, gender, cultural background and expertise.
  - Process: operationalised definitions, pilot-study changes and training.
  - An agreement study.
- Illustrative examples include political lightbulb jokes, a gender-targeted joke, and Chomsky's "Colourless green ideas" sentence as a case of incongruity without humour.
- The authors support perspectivist approaches. They also stress that L1 speakers should not automatically be treated as the gold standard.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{loakman2023iron,
  title = {{The Iron(ic) Melting Pot: Reviewing Human Evaluation in Humour, Irony and Sarcasm Generation}},
  author = {Loakman, Tyler and Maladry, Aaron and Lin, Chenghua},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.444},
  eprint = {2311.05552},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.444}
}
```

- [← previous The intersection of syntax and poetry in the first Hebrew acrostic poem of Lamentations 1](https://md.research.elamur.ai/free/papers/hbo-2023-naude-intersection-syntax-poetry-first-hebrew/)
- [next → The Linear Representation Hypothesis and the Geometry of Large Language Models](https://md.research.elamur.ai/free/papers/park-2023-linear-representation-hypothesis-geometry-large/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/papers/2023/page/6/)

## References

86 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ibrahim Abu Farha, Silviu Vlad Oprea, Steven Wilson, and Walid Magdy. 2022. SemEval-2022 task 6: iSarcasmEval, intended sarcasm detection in English and Arabic. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 802–814, Seattle, United States. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ibrahim+Abu+Farha%2C+Silviu+Vlad+Oprea%2C+Steven+Wilson%2C+and+Walid+Magdy%2E+2022%2E+SemEval%2D2022+task+6%3A+iSarcasmEval%2C+intended+sarcasm+detection+in+English+and+Arabic%2E+In+Proceedings+of+the+16th+Internationa) [our card](https://md.research.elamur.ai/free/papers/farha-2022-semeval-2022-task-6-isarcasmeval/)
2. Ibrahim Abu Farha, Wajdi Zaghouani, and Walid Magdy. 2021. Overview of the WANLP 2021 shared task on sarcasm and sentiment detection in Arabic. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 296–305, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ibrahim+Abu+Farha%2C+Wajdi+Zaghouani%2C+and+Walid+Magdy%2E+2021%2E+Overview+of+the+WANLP+2021+shared+task+on+sarcasm+and+sentiment+detection+in+Arabic%2E+In+Proceedings+of+the+Sixth+Arabic+Natural+Language+Proc) [our card](https://md.research.elamur.ai/free/papers/farha-2021-overview-wanlp-2021-shared-task/)
3. Fernando Alva-Manchego, Carolina Scarton, and Lucia Specia. 2021. The (un)suitability of automatic evaluation metrics for text simplification. Computational Linguistics, 47(4):861–889. [↗](https://doi.org/10.1162/coli_a_00418) [our card](https://md.research.elamur.ai/free/papers/manchego-2021-un-suitability-automatic-evaluation-metrics/)
4. Jacopo Amidei, Paul Piwek, and Alistair Willis. 2019. The use of rating and Likert scales in natural language generation human evaluation tasks: A review and some recommendations. In Proceedings of the 12th International Conference on Natural Language Generation, pages 397–402, Tokyo, Japan. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jacopo+Amidei%2C+Paul+Piwek%2C+and+Alistair+Willis%2E+2019%2E+The+use+of+rating+and+Likert+scales+in+natural+language+generation+human+evaluation+tasks%3A+A+review+and+some+recommendations%2E+In+Proceedings+of+th)
5. Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604. [↗](https://doi.org/10.1162/tacl_a_00041)
6. Dawn G. Blasko, Victoria A. Kazmerski, and Sharif-fah Sheik Dawood. 2021. Saying what you don’t mean: A cross-cultural study of perceptions of sarcasm. Canadian Journal of Experimental Psychology / Revue canadienne de psychologie expérimentale, 75(2):114–119. [↗](https://doi.org/10.1037/cep0000258)
7. Aurélie Breidenbach, Caroline Mahlow, and Andreas Schreiber. 2021. Implicit gender bias in computer science – a qualitative study. [↗](https://doi.org/10.48550/arxiv.2107.01624)
8. Gregory A Bryant and Jean E Fox Tree. 2002. Recognizing verbal irony in spontaneous speech. Metaphor and symbol, 17(2):99–119. [↗](https://doi.org/10.1207/s15327868ms1702_2)
9. C. Burgers. 2010. Verbal irony: Use and effects in written discourse. Ph.D. thesis, UB Nijmegen. [find](https://scholar.google.com/scholar?q=C%2E+Burgers%2E+2010%2E+Verbal+irony%3A+Use+and+effects+in+written+discourse%2E+Ph%2ED%2E+thesis%2C+UB+Nijmegen%2E)
10. Chris Callison-Burch. 2009. Fast, cheap, and creative: Evaluating translation quality using Amazon’s Mechanical Turk. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 286–295, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chris+Callison%2DBurch%2E+2009%2E+Fast%2C+cheap%2C+and+creative%3A+Evaluating+translation+quality+using+Amazon%E2%80%99s+Mechanical+Turk%2E+In+Proceedings+of+the+2009+Conference+on+Empirical+Methods+in+Natural+Language+Pro) [our card](https://md.research.elamur.ai/free/papers/burch-2009-fast-cheap-creative-evaluating-translation/)
11. E. Camp. 2012. Sarcasm, pretense, and the semantics/pragmatics distinction. Nous, 46:587–634. [↗](https://doi.org/10.1111/j.1468-0068.2010.00822.x)
12. Tuhin Chakrabarty, Debanjan Ghosh, Smaranda Muresan, and Nanyun Peng. 2020. Rˆ3: Reverse, retrieve, and rank for sarcasm generation with commonsense knowledge. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7976–7986, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tuhin+Chakrabarty%2C+Debanjan+Ghosh%2C+Smaranda+Muresan%2C+and+Nanyun+Peng%2E+2020%2E+R%CB%863%3A+Reverse%2C+retrieve%2C+and+rank+for+sarcasm+generation+with+commonsense+knowledge%2E+In+Proceedings+of+the+58th+Annual+Meetin) [our card](https://md.research.elamur.ai/free/papers/chakrabarty-2020-r-textasciicircum3-reverse-retrieve-rank/)
13. Arjun Chandrasekaran, Devi Parikh, and Mohit Bansal. 2018. Punny captions: Witty wordplay in image descriptions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 770–775, New Orleans, Louisiana. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Arjun+Chandrasekaran%2C+Devi+Parikh%2C+and+Mohit+Bansal%2E+2018%2E+Punny+captions%3A+Witty+wordplay+in+image+descriptions%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Association+)
14. Wenhu Chen, Yu Su, Xifeng Yan, and William Yang Wang. 2020. KGPT: Knowledge-grounded pre-training for data-to-text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8635–8648, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wenhu+Chen%2C+Yu+Su%2C+Xifeng+Yan%2C+and+William+Yang+Wang%2E+2020%2E+KGPT%3A+Knowledge%2Dgrounded+pre%2Dtraining+for+data%2Dto%2Dtext+generation%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Lan)
15. Noam Chomsky. 1957. Syntactic structures. Syntactic structures. Mouton, Oxford, England. [find](https://scholar.google.com/scholar?q=Noam+Chomsky%2E+1957%2E+Syntactic+structures%2E+Syntactic+structures%2E+Mouton%2C+Oxford%2C+England%2E)
16. Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A. Smith. 2021. All that’s ‘human’ is not gold: Evaluating human evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7282–7296, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Elizabeth+Clark%2C+Tal+August%2C+Sofia+Serrano%2C+Nikita+Haduong%2C+Suchin+Gururangan%2C+and+Noah+A%2E+Smith%2E+2021%2E+All+that%E2%80%99s+%E2%80%98human%E2%80%99+is+not+gold%3A+Evaluating+human+evaluation+of+generated+text%2E+In+Proceedings+of)
17. Jennifer Coates. 2016. Women, men and language: A sociolinguistic account of gender differences in language. Routledge. [find](https://scholar.google.com/scholar?q=Jennifer+Coates%2E+2016%2E+Women%2C+men+and+language%3A+A+sociolinguistic+account+of+gender+differences+in+language%2E+Routledge%2E)
18. Iria de Dios-Flores, Carmen Magariños, Adina Ioana Vladu, John E. Ortega, José Ramom Pichel, Marcos García, Pablo Gamallo, Elisa Fernández Rei, Alberto Bugarín-Diz, Manuel González González, Senén Barro, and Xosé Luis Regueira. 2022. The nós project: Opening routes for the Galician language in the field of language technologies. In Proceedings of the Workshop Towards Digital Language Equality within the 13th Language Resources and Evaluation Conference, pages 52–61, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Iria+de+Dios%2DFlores%2C+Carmen+Magari%C3%B1os%2C+Adina+Ioana+Vladu%2C+John+E%2E+Ortega%2C+Jos%C3%A9+Ramom+Pichel%2C+Marcos+Garc%C3%ADa%2C+Pablo+Gamallo%2C+Elisa+Fern%C3%A1ndez+Rei%2C+Alberto+Bugar%C3%ADn%2DDiz%2C+Manuel+Gonz%C3%A1lez+Gonz%C3%A1lez%2C+Sen%C3%A9n+Bar)
19. European Commission and Directorate-General for Research and Innovation. 2019. She figures 2018. Technical report, European Commission. [find](https://scholar.google.com/scholar?q=European+Commission+and+Directorate%2DGeneral+for+Research+and+Innovation%2E+2019%2E+She+figures+2018%2E+Technical+report%2C+European+Commission%2E)
20. Dalya Faraj and Malak Abdullah. 2021. SarcasmDet at SemEval-2021 task 7: Detect humor and offensive based on demographic factors using RoBERTa pre-trained model. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 527–533, Online. Association for Computational Linguistics. [↗](https://doi.org/10.60692/wy73n-d7955)
21. Elena Filatova. 2012. Irony and sarcasm: Corpus generation and analysis using crowdsourcing. In Lrec, pages 392–398. Citeseer. [find](https://scholar.google.com/scholar?q=Elena+Filatova%2E+2012%2E+Irony+and+sarcasm%3A+Corpus+generation+and+analysis+using+crowdsourcing%2E+In+Lrec%2C+pages+392%E2%80%93398%2E+Citeseer%2E) [our card](https://md.research.elamur.ai/free/papers/filatova-2012-irony-sarcasm-corpus-generation-analysis/)
22. Ruth Filik, Alexandra Ţurcan, Christina Ralph-Nearman, and Alain Pitiot. 2019. What is the difference between irony and sarcasm? an fmri study. Cortex, 115:112–122. [↗](https://doi.org/10.1016/j.cortex.2019.01.025)
23. Simona Frenda, Alessandro Pedrani, Valerio Basile, Soda Marem Lo, Alessandra Teresa Cignarella, Raffaella Panizzon, Cristina Marco, Bianca Scarlini, Viviana Patti, Cristina Bosco, and Davide Bernardi. 2023. Epic: Multi-perspective annotation of a corpus of irony. In ACL 2023. [find](https://scholar.google.com/scholar?q=Simona+Frenda%2C+Alessandro+Pedrani%2C+Valerio+Basile%2C+Soda+Marem+Lo%2C+Alessandra+Teresa+Cignarella%2C+Raffaella+Panizzon%2C+Cristina+Marco%2C+Bianca+Scarlini%2C+Viviana+Patti%2C+Cristina+Bosco%2C+and+Davide+Bernardi%2E) [our card](https://md.research.elamur.ai/free/papers/frenda-2023-epic-multi-perspective-annotation-corpus/)
24. Judit García-González, Patricia Forcén, and Maria Jimenez-Sanchez. 2019. Men and women differ in their perception of gender bias in research institutions. PloS one, 14(12):e0225763–e0225763. [↗](https://doi.org/10.1371/journal.pone.0225763)
25. Aparna Garimella, Carmen Banea, Nabil Hossain, and Rada Mihalcea. 2020. “judge me by my size (noun), do you?” YodaLib: A demographic-aware humor generation framework. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2814–2825, Barcelona, Spain (Online). International Committee on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Aparna+Garimella%2C+Carmen+Banea%2C+Nabil+Hossain%2C+and+Rada+Mihalcea%2E+2020%2E+%E2%80%9Cjudge+me+by+my+size+%28noun%29%2C+do+you%3F%E2%80%9D+YodaLib%3A+A+demographic%2Daware+humor+generation+framework%2E+In+Proceedings+of+the+28th+Intern)
26. Ricky Gervais. 2011. Ricky gervais: The difference between american and british humour. [find](https://scholar.google.com/scholar?q=Ricky+Gervais%2E+2011%2E+Ricky+gervais%3A+The+difference+between+american+and+british+humour%2E)
27. Debanjan Ghosh, Avijit Vajpayee, and Smaranda Muresan. 2020. A report on the 2020 sarcasm detection shared task. In Proceedings of the Second Workshop on Figurative Language Processing, pages 1–11, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Debanjan+Ghosh%2C+Avijit+Vajpayee%2C+and+Smaranda+Muresan%2E+2020%2E+A+report+on+the+2020+sarcasm+detection+shared+task%2E+In+Proceedings+of+the+Second+Workshop+on+Figurative+Language+Processing%2C+pages+1%E2%80%9311%2C+On) [our card](https://md.research.elamur.ai/free/papers/ghosh-2020-report-2020-sarcasm-detection-shared/)
28. Raymond W Gibbs. 1986. On the psycholinguistics of sarcasm. Journal of experimental psychology: general, 115(1):3. [↗](https://doi.org/10.1037//0096-3445.115.1.3)
29. Seraphina Goldfarb-Tarrant, Tuhin Chakrabarty, Ralph Weischedel, and Nanyun Peng. 2020. Content planning for neural story generation with aristotelian rescoring. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4319–4338, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Seraphina+Goldfarb%2DTarrant%2C+Tuhin+Chakrabarty%2C+Ralph+Weischedel%2C+and+Nanyun+Peng%2E+2020%2E+Content+planning+for+neural+story+generation+with+aristotelian+rescoring%2E+In+Proceedings+of+the+2020+Conference+)
30. Tomas Goldsack, Zhihao Zhang, Chenghua Lin, and Carolina Scarton. 2022. Making science simple: Corpora for the lay summarisation of scientific literature. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10589–10604, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2022.emnlp-main.724)
31. Tomas Goldsack, Zhihao Zhang, Chenghua Lin, and Carolina Scarton. 2023. Domain-driven and discourse-guided scientific summarisation. In European Conference on Information Retrieval, pages 361–376. [find](https://scholar.google.com/scholar?q=Tomas+Goldsack%2C+Zhihao+Zhang%2C+Chenghua+Lin%2C+and+Carolina+Scarton%2E+2023%2E+Domain%2Ddriven+and+discourse%2Dguided+scientific+summarisation%2E+In+European+Conference+on+Information+Retrieval%2C+pages+361%E2%80%93376%2E)
32. P.H. Grice. 1978. Further notes on logic and conversation, volume 9, pages 113–127. P. Cole, Syntax and Semantics, New York: Academic Press. [find](https://scholar.google.com/scholar?q=P%2EH%2E+Grice%2E+1978%2E+Further+notes+on+logic+and+conversation%2C+volume+9%2C+pages+113%E2%80%93127%2E+P%2E+Cole%2C+Syntax+and+Semantics%2C+New+York%3A+Academic+Press%2E)
33. David Gros, Yu Li, and Zhou Yu. 2022. Robots-dont-cry: Understanding falsely anthropomorphic utterances in dialog systems. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3266–3284, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=David+Gros%2C+Yu+Li%2C+and+Zhou+Yu%2E+2022%2E+Robots%2Ddont%2Dcry%3A+Understanding+falsely+anthropomorphic+utterances+in+dialog+systems%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+Languag)
34. He He, Nanyun Peng, and Percy Liang. 2019. Pun generation with surprise. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1734–1744, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1904.06828)
35. Joseph Henrich, Steven J. Heine, and Ara Norenzayan. 2010. The weirdest people in the world? Behavioral and Brain Sciences, 33(2-3):61–83. [↗](https://doi.org/10.1017/s0140525x0999152x)
36. Anita Holdcroft. 2007. Gender bias in research: how does it affect evidence based medicine? Journal of the Royal Society of Medicine, 100(1):2–3. PMID: 17197669. [↗](https://doi.org/10.1177/014107680710000102)
37. Nabil Hossain, John Krumm, Michael Gamon, and Henry Kautz. 2020. SemEval-2020 task 7: Assessing humor in edited news headlines. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 746–758, Barcelona (online). International Committee for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2008.00304)
38. David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020. Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions. In Proceedings of the 13th International Conference on Natural Language Generation, pages 169–182, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=David+M%2E+Howcroft%2C+Anya+Belz%2C+Miruna%2DAdriana+Clinciu%2C+Dimitra+Gkatzia%2C+Sadid+A%2E+Hasan%2C+Saad+Mahamood%2C+Simon+Mille%2C+Emiel+van+Miltenburg%2C+Sashank+Santhanam%2C+and+Verena+Rieser%2E+2020%2E+Twenty+years+of+con)
39. Jessica Huynh, Jeffrey Bigham, and Maxine Eskenazi. 2021. A survey of nlp-related crowdsourcing hits: what works and what does not. [↗](https://doi.org/10.48550/arxiv.2111.05241)
40. Neslihan Iskender, Tim Polzehl, and Sebastian Möller. 2021. Reliability of human evaluation for text summarization: Lessons learned and challenges ahead. In Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval), pages 86–96. [find](https://scholar.google.com/scholar?q=Neslihan+Iskender%2C+Tim+Polzehl%2C+and+Sebastian+M%C3%B6ller%2E+2021%2E+Reliability+of+human+evaluation+for+text+summarization%3A+Lessons+learned+and+challenges+ahead%2E+In+Proceedings+of+the+Workshop+on+Human+Evalua)
41. Tonglin Jiang, Hao Li, and Yubo Hou. 2019. Cultural differences in humor perception, usage, and implications. Frontiers in Psychology, 10. [↗](https://doi.org/10.3389/fpsyg.2019.00123)
42. Valentin Jijkoun and Katja Hofmann. 2009. Generating a non-english subjectivity lexicon: Relations that matter. In Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), pages 398–405. [find](https://scholar.google.com/scholar?q=Valentin+Jijkoun+and+Katja+Hofmann%2E+2009%2E+Generating+a+non%2Denglish+subjectivity+lexicon%3A+Relations+that+matter%2E+In+Proceedings+of+the+12th+Conference+of+the+European+Chapter+of+the+ACL+%28EACL+2009%29%2C+pa)
43. Faria Binte Kader, Nafisa Hossain Nujat, Tasmia Binte Sogir, Mohsinul Kabir, Hasan Mahmud, and Md Kam-rul Hasan. 2023. “when words fail, emojis prevail”: A novel architecture for generating sarcastic sentences with emoji using valence reversal and semantic incongruity. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 334–351, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Faria+Binte+Kader%2C+Nafisa+Hossain+Nujat%2C+Tasmia+Binte+Sogir%2C+Mohsinul+Kabir%2C+Hasan+Mahmud%2C+and+Md+Kam%2Drul+Hasan%2E+2023%2E+%E2%80%9Cwhen+words+fail%2C+emojis+prevail%E2%80%9D%3A+A+novel+architecture+for+generating+sarcastic+)
44. Mihir Kale and Abhinav Rastogi. 2020. Text-to-text pre-training for data-to-text tasks. In Proceedings of the 13th International Conference on Natural Language Generation, pages 97–102, Dublin, Ireland. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2005.10433)
45. Tiffany J. Lawless, Conor J. O’Dea, Stuart S. Miller, and Donald A. Saucier. 2020. Is it really just a joke? gender differences in perceptions of sexist humor. Humor (Berlin, Germany), 33(2):291–315. [↗](https://doi.org/10.1515/humor-2019-0033)
46. Elena Lloret, Laura Plaza, and Ahmet Aker. 2018. The challenging task of summary evaluation: an overview. Language Resources and Evaluation, 52:101–148. [↗](https://doi.org/10.1007/s10579-017-9399-2)
47. Tyler Loakman, Chen Tang, and Chenghua Lin. 2023. TwistList: Resources and baselines for tongue twister generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 579–589, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tyler+Loakman%2C+Chen+Tang%2C+and+Chenghua+Lin%2E+2023%2E+TwistList%3A+Resources+and+baselines+for+tongue+twister+generation%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Lingu)
48. Fuli Luo, Shunyao Li, Pengcheng Yang, Lei Li, Baobao Chang, Zhifang Sui, and Xu Sun. 2019. Pun-GAN: Generative adversarial network for pun generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3388–3393, Hong Kong, China. Association for Computational Linguistics. [↗](https://doi.org/10.60692/87m32-70t86)
49. Aaron Maladry, Els Lefever, Cynthia Van Hee, and Véronique Hoste. 2022. The limitations of irony detection in dutch social media. Language Resources and Evaluation. [↗](https://doi.org/10.1007/s10579-023-09656-1)
50. J. A. Meaney. 2020. Crossing the line: Where do demographic variables fit into humor detection? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 176–181, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=J%2E+A%2E+Meaney%2E+2020%2E+Crossing+the+line%3A+Where+do+demographic+variables+fit+into+humor+detection%3F+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+Student+Rese)
51. Abhijit Mishra, Tarun Tater, and Karthik Sankaranarayanan. 2019. A modular architecture for unsupervised sarcasm generation. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 6144–6154. [find](https://scholar.google.com/scholar?q=Abhijit+Mishra%2C+Tarun+Tater%2C+and+Karthik+Sankaranarayanan%2E+2019%2E+A+modular+architecture+for+unsupervised+sarcasm+generation%2E+In+Proceedings+of+the+2019+conference+on+empirical+methods+in+natural+langu) [our card](https://md.research.elamur.ai/free/papers/mishra-2019-modular-architecture-unsupervised-sarcasm-generation/)
52. Anirudh Mittal, Yufei Tian, and Nanyun Peng. 2022. AmbiPun: Generating humorous puns with ambiguous context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1053–1062, Seattle, United States. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Anirudh+Mittal%2C+Yufei+Tian%2C+and+Nanyun+Peng%2E+2022%2E+AmbiPun%3A+Generating+humorous+puns+with+ambiguous+context%2E+In+Proceedings+of+the+2022+Conference+of+the+North+American+Chapter+of+the+Association+for+)
53. Silviu Oprea, Steven Wilson, and Walid Magdy. 2021. Chandler: An explainable sarcastic response generator. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 339–349, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2021.emnlp-demo.38) [our card](https://md.research.elamur.ai/free/papers/oprea-2021-chandler-explainable-sarcastic-response-generator/)
54. Silviu Vlad Oprea, Steven Wilson, and Walid Magdy. 2022. Should a chatbot be sarcastic? understanding user preferences towards sarcasm generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7686–7700, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Silviu+Vlad+Oprea%2C+Steven+Wilson%2C+and+Walid+Magdy%2E+2022%2E+Should+a+chatbot+be+sarcastic%3F+understanding+user+preferences+towards+sarcasm+generation%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Asso)
55. Vaishnavi Pamulapati, Gayatri Purigilla, and Radhika Mamidi. 2020. A novel annotation schema for conversational humor: Capturing the cultural nuances in kanyasulkam. In Proceedings of the 14th Linguistic Annotation Workshop, pages 34–47, Barcelona, Spain. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Vaishnavi+Pamulapati%2C+Gayatri+Purigilla%2C+and+Radhika+Mamidi%2E+2020%2E+A+novel+annotation+schema+for+conversational+humor%3A+Capturing+the+cultural+nuances+in+kanyasulkam%2E+In+Proceedings+of+the+14th+Linguis)
56. Mihir Parmar, Swaroop Mishra, Mor Geva, and Chitta Baral. 2023. Don’t blame the annotator: Bias already starts in the annotation instructions. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1779–1789, Dubrovnik, Croatia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mihir+Parmar%2C+Swaroop+Mishra%2C+Mor+Geva%2C+and+Chitta+Baral%2E+2023%2E+Don%E2%80%99t+blame+the+annotator%3A+Bias+already+starts+in+the+annotation+instructions%2E+In+Proceedings+of+the+17th+Conference+of+the+European+Cha)
57. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent Disagreements in Human Textual Inferences. Transactions of the Association for Computational Linguistics, 7:677–694. [↗](https://doi.org/10.1162/tacl_a_00293) [our card](https://md.research.elamur.ai/free/papers/pavlick-2019-inherent-disagreements-human-textual-inferences/)
58. Louise H. Phillips, Roy Allen, Rebecca Bull, Alexandra Hering, Matthias Kliegel, and Shelley Channon. 2015. Older adults have difficulty in decoding sarcasm. Developmental Psychology, 51(12):1840–1852. [↗](https://doi.org/10.1037/dev0000063)
59. Barbara Plank. 2022. The “problem” of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2022.emnlp-main.731) [our card](https://md.research.elamur.ai/free/papers/plank-2022-problem-human-label-variation-ground/)
60. Ella Rabinovich, Hila Gonen, and Suzanne Stevenson. 2020. Pick a fight or bite your tongue: Investigation of gender differences in idiomatic language usage. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5181–5192, Barcelona, Spain (Online). International Committee on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ella+Rabinovich%2C+Hila+Gonen%2C+and+Suzanne+Stevenson%2E+2020%2E+Pick+a+fight+or+bite+your+tongue%3A+Investigation+of+gender+differences+in+idiomatic+language+usage%2E+In+Proceedings+of+the+28th+International+Co)
61. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. [↗](https://doi.org/10.48550/arxiv.1910.10683)
62. Victor Raskin. 1985. Semantic mechanisms of humor. Synthese language library ; 24. Reidel, Dordrecht. [↗](https://doi.org/10.1007/978-94-009-6472-3)
63. Ehud Reiter. 2018. A structured review of the validity of BLEU. Computational Linguistics, 44(3):393–401. [↗](https://doi.org/10.1162/coli_a_00322) [our card](https://md.research.elamur.ai/free/papers/reiter-2018-structured-review-validity-bleu/)
64. Patricia Rockwell and Evelyn M. Theriot. 2001. Culture, gender, and gender mix in encoders of sarcasm: A self-assessment analysis. Communication Research Reports, 18(1):44–52. [↗](https://doi.org/10.1080/08824090109384781)
65. Stephanie Schoch, Diyi Yang, and Yangfeng Ji. 2020. “this is a problem, don’t you agree?” framing and bias in human evaluation for natural language generation. In Proceedings of the 1st Workshop on Evaluating NLG Evaluation, pages 10–16, Online (Dublin, Ireland). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Stephanie+Schoch%2C+Diyi+Yang%2C+and+Yangfeng+Ji%2E+2020%2E+%E2%80%9Cthis+is+a+problem%2C+don%E2%80%99t+you+agree%3F%E2%80%9D+framing+and+bias+in+human+evaluation+for+natural+language+generation%2E+In+Proceedings+of+the+1st+Workshop+on+Ev)
66. Chen Shani, Alexander Libov, Sofia Tolmach, Liane Lewin-Eytan, Yoelle Maarek, and Dafna Shahaf. 2022. “alexa, do you want to build a snowman?” characterizing playful requests to conversational agents. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, CHI EA ’22, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Chen+Shani%2C+Alexander+Libov%2C+Sofia+Tolmach%2C+Liane+Lewin%2DEytan%2C+Yoelle+Maarek%2C+and+Dafna+Shahaf%2E+2022%2E+%E2%80%9Calexa%2C+do+you+want+to+build+a+snowman%3F%E2%80%9D+characterizing+playful+requests+to+conversational+agents%2E)
67. Natalie Shapira, Oren Kalinsky, Alexander Libov, Chen Shani, and Sofia Tolmach. 2023. Evaluating humorous response generation to playful shopping requests. In ECIR 2023. [find](https://scholar.google.com/scholar?q=Natalie+Shapira%2C+Oren+Kalinsky%2C+Alexander+Libov%2C+Chen+Shani%2C+and+Sofia+Tolmach%2E+2023%2E+Evaluating+humorous+response+generation+to+playful+shopping+requests%2E+In+ECIR+2023%2E)
68. Boaz Shmueli, Jan Fell, Soumya Ray, and Lun-Wei Ku. 2021. Beyond fair pay: Ethical implications of NLP crowdsourcing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3758–3769, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Boaz+Shmueli%2C+Jan+Fell%2C+Soumya+Ray%2C+and+Lun%2DWei+Ku%2E+2021%2E+Beyond+fair+pay%3A+Ethical+implications+of+NLP+crowdsourcing%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Associa)
69. Gary F. Simons, Abbey L. L. Thomas, and Chad K. K. White. 2022. Assessing digital language support on a global scale. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4299–4305, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2209.13515)
70. Anders Søgaard. 2022. Should we ban English NLP for a year? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5254–5260, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Anders+S%C3%B8gaard%2E+2022%2E+Should+we+ban+English+NLP+for+a+year%3F+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+5254%E2%80%935260%2C+Abu+Dhabi%2C+United+Arab+Emirates%2E)
71. Jiao Sun, Anjali Narayan-Chen, Shereen Oraby, Alessandra Cervone, Tagyoung Chung, Jing Huang, Yang Liu, and Nanyun Peng. 2022a. ExPUNations: Augmenting puns with keywords and explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4590–4605, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2210.13513)
72. Jiao Sun, Anjali Narayan-Chen, Shereen Oraby, Shuyang Gao, Tagyoung Chung, Jing Huang, Yang Liu, and Nanyun Peng. 2022b. Context-situated pun generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4635– 4648, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2210.13522)
73. Chen Tang, Hongbo Zhang, Tyler Loakman, Chenghua Lin, and Frank Guerin. 2023. Terminology-aware medical dialogue generation. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. [find](https://scholar.google.com/scholar?q=Chen+Tang%2C+Hongbo+Zhang%2C+Tyler+Loakman%2C+Chenghua+Lin%2C+and+Frank+Guerin%2E+2023%2E+Terminology%2Daware+medical+dialogue+generation%2E+In+ICASSP+2023+%2D+2023+IEEE+International+Conference+on+Acoustics%2C+Speech+an)
74. Chen Tang, Zhihao Zhang, Tyler Loakman, Chenghua Lin, and Frank Guerin. 2022. NGEP: A graph-based event planning framework for story generation. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 186–193, Online only. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2210.10602)
75. Yufei Tian, Divyanshu Sheth, and Nanyun Peng. 2022. A unified framework for pun generation with humor principles. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3253–3261, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yufei+Tian%2C+Divyanshu+Sheth%2C+and+Nanyun+Peng%2E+2022%2E+A+unified+framework+for+pun+generation+with+humor+principles%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2022%2C+pages+3253%E2%80%933)
76. Alexandra N. Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2022. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385–1470. [↗](https://doi.org/10.1613/jair.1.12752)
77. Tim Van de Cruys. 2020. Automatic poetry generation from prosaic text. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2471–2480, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.acl-main.223)
78. Chris van der Lee, Albert Gatt, Emiel van Miltenburg, Sander Wubben, and Emiel Krahmer. 2019. Best practices for the human evaluation of automatically generated text. In Proceedings of the 12th International Conference on Natural Language Generation, pages 355–368, Tokyo, Japan. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chris+van+der+Lee%2C+Albert+Gatt%2C+Emiel+van+Miltenburg%2C+Sander+Wubben%2C+and+Emiel+Krahmer%2E+2019%2E+Best+practices+for+the+human+evaluation+of+automatically+generated+text%2E+In+Proceedings+of+the+12th+Intern)
79. Cynthia Van Hee. 2017. Can machines sense irony? : exploring automatic irony detection on social media. Ph.D. thesis, Ghent University. [find](https://scholar.google.com/scholar?q=Cynthia+Van+Hee%2E+2017%2E+Can+machines+sense+irony%3F+%3A+exploring+automatic+irony+detection+on+social+media%2E+Ph%2ED%2E+thesis%2C+Ghent+University%2E)
80. Byron C Wallace, Laura Kertz, Eugene Charniak, et al. 2014. Humans require context to infer ironic intent (so computers probably do, too). In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 512–516. [find](https://scholar.google.com/scholar?q=Byron+C+Wallace%2C+Laura+Kertz%2C+Eugene+Charniak%2C+et+al%2E+2014%2E+Humans+require+context+to+infer+ironic+intent+%28so+computers+probably+do%2C+too%29%2E+In+Proceedings+of+the+52nd+Annual+Meeting+of+the+Association+)
81. Zhiwei Yu, Hongyu Zang, and Xiaojun Wan. 2020. Homophonic pun generation with lexically constrained rewriting. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2870–2876, Online. Association for Computational Linguistics. [↗](https://doi.org/10.60692/gfh5q-qqr15)
82. Mike Zajko. 2021. Conservative AI and social inequality: conceptualizing alternatives to bias through social theory. AI & SOCIETY, 36(3):1047–1056. [↗](https://doi.org/10.1007/s00146-021-01153-9)
83. Qingcheng Zeng and An-Ran Li. 2022. A survey in automatic irony processing: Linguistic, cognitive, and multi-X perspectives. In Proceedings of the 29th International Conference on Computational Linguistics, pages 824–836, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2209.04712) [our card](https://md.research.elamur.ai/free/papers/zeng-2022-survey-automatic-irony-processing-linguistic/)
84. Yue Zhang, Hongliang Fei, Dingcheng Li, and Ping Li. 2022. PromptGen: Automatically generate prompts using generative models. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 30–37, Seattle, United States. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yue+Zhang%2C+Hongliang+Fei%2C+Dingcheng+Li%2C+and+Ping+Li%2E+2022%2E+PromptGen%3A+Automatically+generate+prompts+using+generative+models%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+NAACL+2022%2C+)
85. Kun Zhao, Bohao Yang, Chenghua Lin, Wenge Rong, Aline Villavicencio, and Xiaohui Cui. 2023a. Evaluating open-domain dialogues in latent space with next sentence prediction and mutual information. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 562–574. [find](https://scholar.google.com/scholar?q=Kun+Zhao%2C+Bohao+Yang%2C+Chenghua+Lin%2C+Wenge+Rong%2C+Aline+Villavicencio%2C+and+Xiaohui+Cui%2E+2023a%2E+Evaluating+open%2Ddomain+dialogues+in+latent+space+with+next+sentence+prediction+and+mutual+information%2E+In+P)
86. Wenye Zhao, Qingbao Huang, Dongsheng Xu, and Peizhi Zhao. 2023b. Multi-modal sarcasm generation: Dataset and solution. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5601– 5613, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wenye+Zhao%2C+Qingbao+Huang%2C+Dongsheng+Xu%2C+and+Peizhi+Zhao%2E+2023b%2E+Multi%2Dmodal+sarcasm+generation%3A+Dataset+and+solution%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2023%2C+pages+560)