---
title: "ChatGPT Outperforms Crowd-Workers for — Gilardi 2023"
description: "Research projects in NLP and computational social science need labeled texts to train classifiers and to evaluate models."
authors: ["Fabrizio Gilardi", "Meysam Alizadeh", "Maël Kubli"]
year: 2023
venue: "arXiv:2303.15056"
source: "https://arxiv.org/abs/2303.15056"
doi: "10.1073/pnas.2305016120"
arxiv: "2303.15056"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gilardi-2023-chatgpt-outperforms-crowd-workers-text/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gilardi-2023-chatgpt-outperforms-crowd-workers-text/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/) / ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks

# ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks

**Authors:** Fabrizio Gilardi, Meysam Alizadeh, Maël Kubli · **Institutions:** [Harvard University](https://md.research.elamur.ai/free/en/universities/harvard-university/) · [University of Zurich](https://md.research.elamur.ai/free/en/universities/university-of-zurich/) · **Year:** 2023 · **Venue:** arXiv:2303.15056 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://doi.org/10.1073/pnas.2305016120) · [PDF](https://arxiv.org/pdf/2303.15056) · [DOI 10.1073/pnas.2305016120](https://doi.org/10.1073/pnas.2305016120) · [arXiv 2303.15056](https://arxiv.org/abs/2303.15056) · [OpenAlex](https://openalex.org/W4384662964)

**Citations:** 1162 citations (OpenAlex, 2026-10-14) · [370](https://md.research.elamur.ai/free/en/papers/gilardi-2023-chatgpt-outperforms-crowd-workers-text/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
Research projects in NLP and computational social science need labeled texts to train classifiers and to evaluate models. Typically these labels come from trained research assistants, who are costly, or MTurk crowd-workers, who are cheaper but less reliable. The authors tested whether ChatGPT (gpt-3.5-turbo, zero-shot) can replace crowd-workers. They used 6,183 English tweets and news articles about content moderation and US politics. These texts had already been labeled by trained political science students for a previous study, which served as the gold standard. The tasks were relevance, stance, topic and frame detection. The same codebooks were given unchanged to ChatGPT and to filtered MTurk workers. ChatGPT was run twice at each of two temperature settings. Its accuracy was on average about 25 percentage points higher than MTurk's, and its run-to-run consistency was higher than that of both human groups. It also cost roughly thirty times less. These results suggest that LLMs could make large-scale annotation far cheaper and might undermine the business model of crowdsourcing platforms.

## Key points
- Data: four samples were used. These were 2,382 content-moderation tweets (2020–2021), 1,856 US Congress tweets (2017–2022), 1,606 LexisNexis news articles and 339 English tweets from January 2023. The 2023 sample was added to check for memorization.
- Tasks: relevance (two variants), stance on Section 230 (three classes), topics (six classes), problem/solution frames and 14 policy frames from Card et al. (2015).
- Intercoder agreement averaged about 56% for MTurk and 79% for trained annotators. For ChatGPT it was 91% at temperature 1 and 97% at temperature 0.2. The lower temperature improved consistency without hurting accuracy.
- Zero-shot relevance accuracy was 70% for content-moderation tweets, 81% for news articles, 83% for Congress tweets and 59% for the 2023 tweets. The weaker 2023 result came from mislabeling tweets about user suspensions, which shows how much prompt quality matters.
- ChatGPT cost less than $0.003 per annotation. Replication materials are available on Harvard Dataverse (doi:10.7910/DVN/PQYF6M).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{gilardi2023chatgpt,
  title = {{ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks}},
  author = {Gilardi, Fabrizio and Alizadeh, Meysam and Kubli, Maël},
  year = {2023},
  doi = {10.1073/pnas.2305016120},
  eprint = {2303.15056},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1073/pnas.2305016120}
}
```

- [← previous Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://md.research.elamur.ai/free/en/papers/pan-2023-unifying-large-language-models-knowledge/)
- [next → Sigmoid Loss for Language Image Pre-Training](https://md.research.elamur.ai/free/en/papers/zhai-2023-sigmoid-loss-language-image-pre/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/)

## References

14 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alizadeh, Meysam, Fabrizio Gilardi, Emma Hoes, K Jonathan Klüser, Mael Kubli and Nahema Marchal. 2022. “Content Moderation As a Political Issue: The Twitter Discourse Around Trump’s Ban.” Journal of Quantitative Description: Digital Media 2. [find](https://scholar.google.com/scholar?q=Alizadeh%2C+Meysam%2C+Fabrizio+Gilardi%2C+Emma+Hoes%2C+K+Jonathan+Kl%C3%BCser%2C+Mael+Kubli+and+Nahema+Marchal%2E+2022%2E+%E2%80%9CContent+Moderation+As+a+Political+Issue%3A+The+Twitter+Discourse+Around+Trump%E2%80%99s+Ban%2E%E2%80%9D+Journal+of+Q)
2. Argyle, Lisa P., Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting and David Wingate. 2023. “Out of One, Many: Using Language Models to Simulate Human Samples.” Political Analysis pp. 1–15. [find](https://scholar.google.com/scholar?q=Argyle%2C+Lisa+P%2E%2C+Ethan+C%2E+Busby%2C+Nancy+Fulda%2C+Joshua+R%2E+Gubler%2C+Christopher+Rytting+and+David+Wingate%2E+2023%2E+%E2%80%9COut+of+One%2C+Many%3A+Using+Language+Models+to+Simulate+Human+Samples%2E%E2%80%9D+Political+Analysis+pp%2E) [our card](https://md.research.elamur.ai/free/en/papers/argyle-2022-out-one-many-using-language/)
3. Bayerl, Petra Saskia and Karsten Ingmar Paul. 2011. “What Determines Inter-Coder Agreement in Manual Annotations? A Meta-Analytic Investigation.” Computational Linguistics 37(4):699–725. [find](https://scholar.google.com/scholar?q=Bayerl%2C+Petra+Saskia+and+Karsten+Ingmar+Paul%2E+2011%2E+%E2%80%9CWhat+Determines+Inter%2DCoder+Agreement+in+Manual+Annotations%3F+A+Meta%2DAnalytic+Investigation%2E%E2%80%9D+Computational+Linguistics+37%284%29%3A699%E2%80%93725%2E)
4. Benoit, Kenneth, Drew Conway, Benjamin E. Lauderdale, Michael Laver and Slava Mikhaylov. 2016. “Crowd-Sourced Text Analysis: Reproducible and Agile Production of Political Data.” American Political Science Review 116(2):278–295. [find](https://scholar.google.com/scholar?q=Benoit%2C+Kenneth%2C+Drew+Conway%2C+Benjamin+E%2E+Lauderdale%2C+Michael+Laver+and+Slava+Mikhaylov%2E+2016%2E+%E2%80%9CCrowd%2DSourced+Text+Analysis%3A+Reproducible+and+Agile+Production+of+Political+Data%2E%E2%80%9D+American+Political+Sc)
5. Binz, Marcel and Eric Schulz. 2023. “Using cognitive psychology to understand GPT-3.” Proceedings of the National Academy of Sciences 120(6):e2218523120. [find](https://scholar.google.com/scholar?q=Binz%2C+Marcel+and+Eric+Schulz%2E+2023%2E+%E2%80%9CUsing+cognitive+psychology+to+understand+GPT%2D3%2E%E2%80%9D+Proceedings+of+the+National+Academy+of+Sciences+120%286%29%3Ae2218523120%2E)
6. Card, Dallas, Amber Boydstun, Justin H Gross, Philip Resnik and Noah A Smith. 2015. The media frames corpus: Annotations of frames across issues. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language… [find](https://scholar.google.com/scholar?q=Card%2C+Dallas%2C+Amber+Boydstun%2C+Justin+H+Gross%2C+Philip+Resnik+and+Noah+A+Smith%2E+2015%2E+The+media+frames+corpus%3A+Annotations+of+frames+across+issues%2E+In+Proceedings+of+the+53rd+Annual+Meeting+of+the+Assoc) [our card](https://md.research.elamur.ai/free/en/papers/card-2015-media-frames-corpus-annotations-frames/)
7. Chmielewski, Michael and Sarah C. Kucker. 2020. “An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results.” Social Psychological and Personality Science 11(4):464–473. [find](https://scholar.google.com/scholar?q=Chmielewski%2C+Michael+and+Sarah+C%2E+Kucker%2E+2020%2E+%E2%80%9CAn+MTurk+Crisis%3F+Shifts+in+Data+Quality+and+the+Impact+on+Study+Results%2E%E2%80%9D+Social+Psychological+and+Personality+Science+11%284%29%3A464%E2%80%93473%2E)
8. Desmond, Michael, Evelyn Duesterwald, Kristina Brimijoin, Michelle Brachman and Qian Pan. 2021. Semi-automated data labeling. In NeurIPS 2020 Competition and Demonstration Track. PMLR pp. 156–169. [find](https://scholar.google.com/scholar?q=Desmond%2C+Michael%2C+Evelyn+Duesterwald%2C+Kristina+Brimijoin%2C+Michelle+Brachman+and+Qian+Pan%2E+2021%2E+Semi%2Dautomated+data+labeling%2E+In+NeurIPS+2020+Competition+and+Demonstration+Track%2E+PMLR+pp%2E+156%E2%80%93169%2E)
9. Emerson, Guy, Natalie Schluter, Gabriel Stanovsky, Ritesh Kumar, Alexis Palmer, Nathan Schneider, Siddharth Singh and Shyam Ratan, eds. 2022. Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). Seattle: Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Emerson%2C+Guy%2C+Natalie+Schluter%2C+Gabriel+Stanovsky%2C+Ritesh+Kumar%2C+Alexis+Palmer%2C+Nathan+Schneider%2C+Siddharth+Singh+and+Shyam+Ratan%2C+eds%2E+2022%2E+Proceedings+of+the+16th+International+Workshop+on+Semantic)
10. Huang, Fan, Haewoon Kwak and Jisun An. 2023. “Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech.” arXiv preprint arXiv:2302.07736 . [find](https://scholar.google.com/scholar?q=Huang%2C+Fan%2C+Haewoon+Kwak+and+Jisun+An%2E+2023%2E+%E2%80%9CIs+ChatGPT+better+than+Human+Annotators%3F+Potential+and+Limitations+of+ChatGPT+in+Explaining+Implicit+Hate+Speech%2E%E2%80%9D+arXiv+preprint+arXiv%3A2302%2E07736+%2E)
11. Kojima, Takeshi, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo and Yusuke Iwasawa. 2022. “Large language models are zero-shot reasoners.” arXiv preprint arXiv:2205.11916 . [find](https://scholar.google.com/scholar?q=Kojima%2C+Takeshi%2C+Shixiang+Shane+Gu%2C+Machel+Reid%2C+Yutaka+Matsuo+and+Yusuke+Iwasawa%2E+2022%2E+%E2%80%9CLarge+language+models+are+zero%2Dshot+reasoners%2E%E2%80%9D+arXiv+preprint+arXiv%3A2205%2E11916+%2E) [our card](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/)
12. Kuzman, Taja, Igor Mozetič and Nikola Ljubešić. 2023. “ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification.” arXiv e-prints pp. arXiv–2303. [find](https://scholar.google.com/scholar?q=Kuzman%2C+Taja%2C+Igor+Mozetic%CC%8C+and+Nikola+Ljube%C5%A1ic%CC%81%2E+2023%2E+%E2%80%9CChatGPT%3A+Beginning+of+an+End+of+Manual+Linguistic+Data+Annotation%3F+Use+Case+of+Automatic+Genre+Identification%2E%E2%80%9D+arXiv+e%2Dprints+pp%2E+arXiv%E2%80%932303%2E)
13. Nay, John J. 2023. “Large Language Models as Corporate Lobbyists.”. [find](https://scholar.google.com/scholar?q=Nay%2C+John+J%2E+2023%2E+%E2%80%9CLarge+Language+Models+as+Corporate+Lobbyists%2E%E2%80%9D%2E)
14. Wu, Patrick Y., Joshua A. Tucker, Jonathan Nagler and Solomon Messing. 2023. “Large Language Models Can Be Used to Estimate the Ideologies of Politicians in a Zero-Shot Learning Setting.”. [find](https://scholar.google.com/scholar?q=Wu%2C+Patrick+Y%2E%2C+Joshua+A%2E+Tucker%2C+Jonathan+Nagler+and+Solomon+Messing%2E+2023%2E+%E2%80%9CLarge+Language+Models+Can+Be+Used+to+Estimate+the+Ideologies+of+Politicians+in+a+Zero%2DShot+Learning+Setting%2E%E2%80%9D%2E)