---
title: "Is ChatGPT better than Human Annotators? Potential and"
description: "The authors ask whether ChatGPT can handle a subjective task that depends on social and cultural… 20 references, cited by 94 papers here."
authors: ["Fan Huang", "Haewoon Kwak", "Jisun An"]
year: 2023
venue: "arXiv:2302.07736"
source: "https://arxiv.org/abs/2302.07736"
doi: "10.1145/3543873.3587368"
arxiv: "2302.07736"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/huang-2023-chatgpt-better-than-human-annotators/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/huang-2023-chatgpt-better-than-human-annotators/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/) / Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech

# Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech

**Authors:** Fan Huang, [Haewoon Kwak](https://md.research.elamur.ai/free/en/authors/haewoon-kwak/) and [Jisun An](https://md.research.elamur.ai/free/en/authors/jisun-an/) · **Year:** 2023 · **Venue:** [arXiv:2302.07736](https://md.research.elamur.ai/free/en/venues/acm-web-conference-www/) ([WWW 2023](https://md.research.elamur.ai/free/en/venues/acm-web-conference-www/2023/)) · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://doi.org/10.1145/3543873.3587368) · [PDF](https://arxiv.org/pdf/2302.07736) · [DOI 10.1145/3543873.3587368](https://doi.org/10.1145/3543873.3587368) · [arXiv 2302.07736](https://arxiv.org/abs/2302.07736) · [OpenAlex](https://openalex.org/W4327811957)

**Citations:** 224 citations (OpenAlex, 2026-10-14) · [94](https://md.research.elamur.ai/free/en/papers/huang-2023-chatgpt-better-than-human-annotators/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The authors ask whether ChatGPT can handle a subjective task that depends on social and cultural context: recognizing implicit hate speech and explaining in natural language (NLE) why a post is hateful. They randomly sampled 795 tweets from the LatentHatred dataset, all labeled implicitly hateful. Each tweet went to ChatGPT (Jan 9, 2023 version) in a fresh session, three times, using a prompt that asks for a yes/no verdict plus a one-sentence justification. The three answers were combined into a single score per tweet. Amazon Mechanical Turk workers then re-judged the cases where ChatGPT disagreed with the dataset labels and rated the quality of explanations. ChatGPT agreed with the original labels about 80% of the time. On the disputed cases, lay annotators mostly sided with ChatGPT, and ChatGPT's explanations pushed their judgments even further toward its verdict. This shows the model's promise for annotation, but also a risk: persuasive explanations could spread wrong labels. The supplied extract stops before the RQ2 results on explanation quality, so those findings are not covered here.

## Key points
- **Data:** 795 tweets (12.5%, the same size as the original test split) were drawn from the 6,358 implicit hate tweets in LatentHatred. The human-written "implied statements" in that dataset serve as the reference NLEs.
- **Aggregation:** each answer was scored +1 (Yes), −1 (No) or 0 (other, e.g. "It's unclear"). The mean of the three answers decided whether a tweet was labeled Hateful, Non-Hateful or Uncertain.
- **Classification results:** ChatGPT labeled 636 tweets hateful (80%), 146 non-hateful (18.4%) and 13 uncertain (1.6%).
- **Human re-evaluation of disputed cases:** average hatefulness scores were:
  - −0.41 when workers saw only the post;
  - −0.52 when they also saw ChatGPT's explanation (significantly lower, t(316) = −13.75, p\<0.001);
  - +0.29 when they saw the human-written NLE.
- **Explanation-quality setup:** 100 random tweets were paired with either human or ChatGPT NLEs and rated on 7-point Likert scales for Informativeness and Clarity. Three MTurk Masters rated each pair, and research assistants reviewed the ratings.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Indiana University Bloomington](https://md.research.elamur.ai/free/en/universities/indiana-university-bloomington/) — University, Bloomington, United States · 40 works · 4,003 citations
- [Indiana University](https://md.research.elamur.ai/free/en/universities/indiana-university/) — University, Bloomington, United States · 42 works · 3,981 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 626 works

🧭 Topics (2): [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · 1,035 works · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · 587 works

## Cite

Cite as: Fan Huang, Haewoon Kwak and Jisun An. 2023. Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech. arXiv:2302.07736.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{huang2023is,
  title = {{Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech}},
  author = {Fan Huang, Haewoon Kwak and Jisun An},
  year = {2023},
  doi = {10.1145/3543873.3587368},
  eprint = {2302.07736},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3543873.3587368}
}
```

- [← previous Self-collaboration Code Generation via ChatGPT](https://md.research.elamur.ai/free/en/papers/dong-2023-self-collaboration-code-generation/)
- [next → How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://md.research.elamur.ai/free/en/papers/wang-2023-far-camels-go-exploring-state/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/)

## References

20 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jisun An, Haewoon Kwak, Claire Seungeun Lee, Bogang Jun, and Yong-Yeol Ahn. 2021. Predicting Anti-Asian Hateful Users on Twitter during COVID-19. In EMNLP Findings. [find](https://scholar.google.com/scholar?q=Jisun+An%2C+Haewoon+Kwak%2C+Claire+Seungeun+Lee%2C+Bogang+Jun%2C+and+Yong%2DYeol+Ahn%2E+2021%2E+Predicting+Anti%2DAsian+Hateful+Users+on+Twitter+during+COVID%2D19%2E+In+EMNLP+Findings%2E)
2. Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. [SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter](https://md.research.elamur.ai/free/en/papers/basile-2019-semeval-2019-task-5-multilingual/). In 13th International Workshop on Semantic E…
3. Anja Belz and Eric Kow. 2009. System Building Cost vs. Output Quality in Data-to-Text Generation. In Proceedings of European Workshop on Natural Language Generation. [find](https://scholar.google.com/scholar?q=Anja+Belz+and+Eric+Kow%2E+2009%2E+System+Building+Cost+vs%2E+Output+Quality+in+Data%2Dto%2DText+Generation%2E+In+Proceedings+of+European+Workshop+on+Natural+Language+Generation%2E)
4. Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. [e-SNLI: Natural language inference with natural language explanations](https://md.research.elamur.ai/free/en/papers/camburu-2018-e-snli-natural-language-inference/). In NeurIPS.
5. Ona de Gibert, Naiara Perez, Aitor García-Pablos, and Montse Cuadros. 2018. [Hate Speech Dataset from a White Supremacy Forum](https://md.research.elamur.ai/free/en/papers/gibert-2018-hate-speech-dataset-white-supremacy/). In Proceedings of the 2nd Workshop on Abusive Language Online.
6. Ondřej Dušek, Jekaterina Novikova, and Verena Rieser. 2020. Evaluating the state-of-the-art of end-to-end natural language generation: The E2E NLG challenge. Computer Speech & Language (2020). [find](https://scholar.google.com/scholar?q=Ond%C5%99ej+Du%C5%A1ek%2C+Jekaterina+Novikova%2C+and+Verena+Rieser%2E+2020%2E+Evaluating+the+state%2Dof%2Dthe%2Dart+of+end%2Dto%2Dend+natural+language+generation%3A+The+E2E+NLG+challenge%2E+Computer+Speech+%26+Language+%282020%29%2E)
7. Mai ElSherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Munmun De Choudhury, and Diyi Yang. 2021. [Latent Hatred: A Benchmark for Understanding Implicit Hate Speech](https://md.research.elamur.ai/free/en/papers/elsherief-2021-latent-hatred-benchmark-understanding-implicit/). In EMNLP.
8. Engadget. 2023. ChatGPT reportedly reached 100 million users in January. [https://www.engadget.com/chatgpt-100-million-users-january-130619073.html](https://www.engadget.com/chatgpt-100-million-users-january-130619073.html). [↗](https://www.engadget.com/chatgpt-100-million-users-january-130619073.html)
9. Aidan Gilson, Conrad Safranek, Thomas Huang, Vimig Socrates, Ling Chi, Richard Andrew Taylor, and David Chartash. 2022. How Well Does ChatGPT Do When Taking the Medical Licensing Exams? The Implications of Large Language Models for Medical Education and Knowledge Assessment. medRxiv (2022), 2022–12. [find](https://scholar.google.com/scholar?q=Aidan+Gilson%2C+Conrad+Safranek%2C+Thomas+Huang%2C+Vimig+Socrates%2C+Ling+Chi%2C+Richard+Andrew+Taylor%2C+and+David+Chartash%2E+2022%2E+How+Well+Does+ChatGPT+Do+When+Taking+the+Medical+Licensing+Exams%3F+The+Implicatio)
10. Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. [How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection](https://md.research.elamur.ai/free/en/papers/guo-2023-close-chatgpt-human-experts-comparison/). arXiv:2301.07597 (2023).
11. Gabriel Emile Hine, Jeremiah Onaolapo, Emiliano De Cristofaro, Nicolas Kourtellis, Ilias Leontiadis, Riginos Samaras, Gianluca Stringhini, and Jeremy Blackburn. 2017. Kek, cucks, and god emperor trump: A measurement study of 4chan’s politically incorrect forum and its effects on the web. In ICWSM. [find](https://scholar.google.com/scholar?q=Gabriel+Emile+Hine%2C+Jeremiah+Onaolapo%2C+Emiliano+De+Cristofaro%2C+Nicolas+Kourtellis%2C+Ilias+Leontiadis%2C+Riginos+Samaras%2C+Gianluca+Stringhini%2C+and+Jeremy+Blackburn%2E+2017%2E+Kek%2C+cucks%2C+and+god+emperor+trump)
12. Fan Huang, Haewoon Kwak, and Jisun An. 2023. Chain of Explanation: New Prompting Method to Generate Higher Quality Natural Language Explanation for Implicit Hate Speech. In Companion Proceedings of the ACM Web Conference. [find](https://scholar.google.com/scholar?q=Fan+Huang%2C+Haewoon+Kwak%2C+and+Jisun+An%2E+2023%2E+Chain+of+Explanation%3A+New+Prompting+Method+to+Generate+Higher+Quality+Natural+Language+Explanation+for+Implicit+Hate+Speech%2E+In+Companion+Proceedings+of+th)
13. Katharina Jeblick, Balthasar Schachtner, Jakob Dexl, Andreas Mittermeier, Anna Theresa Stüber, Johanna Topalis, Tobias Weber, Philipp Wesp, Bastian Sabel, Jens Ricke, et al. 2022. ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reports. arXiv:2212.14882 (202… — [arXiv:2212.14882](https://arxiv.org/abs/2212.14882) [find](https://scholar.google.com/scholar?q=Katharina+Jeblick%2C+Balthasar+Schachtner%2C+Jakob+Dexl%2C+Andreas+Mittermeier%2C+Anna+Theresa+St%C3%BCber%2C+Johanna+Topalis%2C+Tobias+Weber%2C+Philipp+Wesp%2C+Bastian+Sabel%2C+Jens+Ricke%2C+et+al%2E+2022%2E+ChatGPT+Makes+Medici)
14. Haewoon Kwak, Jeremy Blackburn, and Seungyeop Han. 2015. Exploring cyberbullying and other toxic behavior in team competition online games. In Proceedings of the 33rd annual ACM conference on human factors in computing systems. [find](https://scholar.google.com/scholar?q=Haewoon+Kwak%2C+Jeremy+Blackburn%2C+and+Seungyeop+Han%2E+2015%2E+Exploring+cyberbullying+and+other+toxic+behavior+in+team+competition+online+games%2E+In+Proceedings+of+the+33rd+annual+ACM+conference+on+human+fa)
15. OpenAI. 2022. ChatGPT: Optimizing Language Model for Dialogue. OpenAI blog (2022). [find](https://scholar.google.com/scholar?q=OpenAI%2E+2022%2E+ChatGPT%3A+Optimizing+Language+Model+for+Dialogue%2E+OpenAI+blog+%282022%29%2E)
16. Joni Salminen, Hind Almerekhi, Milica Milenković, Soon-gyo Jung, Jisun An, Haewoon Kwak, and Bernard J Jansen. 2018. Anatomy of online hate: developing a taxonomy and machine learning models for identifying and classifying hate in online news media. In ICWSM. [find](https://scholar.google.com/scholar?q=Joni+Salminen%2C+Hind+Almerekhi%2C+Milica+Milenkovi%C4%87%2C+Soon%2Dgyo+Jung%2C+Jisun+An%2C+Haewoon+Kwak%2C+and+Bernard+J+Jansen%2E+2018%2E+Anatomy+of+online+hate%3A+developing+a+taxonomy+and+machine+learning+models+for+ident)
17. Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://md.research.elamur.ai/free/en/papers/sap-2020-social-bias-frames-reasoning-about/). In ACL.
18. Dominik Sobania, Martin Briesch, Carol Hanna, and Justyna Petke. 2023. An Analysis of the Automatic Bug Fixing Performance of ChatGPT. arXiv:2301.08653 (2023). — [arXiv:2301.08653](https://arxiv.org/abs/2301.08653) [find](https://scholar.google.com/scholar?q=Dominik+Sobania%2C+Martin+Briesch%2C+Carol+Hanna%2C+and+Justyna+Petke%2E+2023%2E+An+Analysis+of+the+Automatic+Bug+Fixing+Performance+of+ChatGPT%2E+arXiv%3A2301%2E08653+%282023%29%2E+%E2%80%94+%5BarXiv%3A2301%2E08653%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
19. Rohit Sridhar and Diyi Yang. 2022. Explaining Toxic Text via Knowledge Enhanced Text Generation. In NAACL. [find](https://scholar.google.com/scholar?q=Rohit+Sridhar+and+Diyi+Yang%2E+2022%2E+Explaining+Toxic+Text+via+Knowledge+Enhanced+Text+Generation%2E+In+NAACL%2E)
20. Bowen Zhang, Daijun Ding, and Liwen Jing. 2022. [How would Stance Detection Techniques Evolve after the Launch of ChatGPT?](https://md.research.elamur.ai/free/en/papers/zhang-2022-would-stance-detection-techniques-evolve/) arXiv:2212.14548 (2022).

*Updated 2 November 2026*