---
title: "ChatGPT-4 Outperforms Experts and Crowd — Tornberg 2023"
description: "The paper asks whether a general-purpose LLM can handle interpretive annotation that depends on world knowledge and on inferring what an author intends."
authors: ["Petter Tornberg"]
year: 2023
venue: "arXiv:2304.06588"
source: "https://arxiv.org/abs/2304.06588"
doi: "10.48550/arxiv.2304.06588"
arxiv: "2304.06588"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tornberg-2023-chatgpt-4-outperforms-experts-crowd/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tornberg-2023-chatgpt-4-outperforms-experts-crowd/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/2/) / ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning

# ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning

**Authors:** Petter Tornberg · **Year:** 2023 · **Venue:** arXiv:2304.06588 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2304.06588) · [PDF](https://arxiv.org/pdf/2304.06588) · [DOI 10.48550/arxiv.2304.06588](https://doi.org/10.48550/arxiv.2304.06588) · [arXiv 2304.06588](https://arxiv.org/abs/2304.06588) · [OpenAlex](https://openalex.org/W4365601444)

**Citations:** 150 citations (openalex-snapshot · 2026-10-14) · [80](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper asks whether a general-purpose LLM can handle interpretive annotation that depends on world knowledge and on inferring what an author intends. Traditionally, such work is left to human coders. The test task is guessing whether a tweet was written by a Democrat or a Republican, where the true party is known. The author sampled 500 tweets from US senators posted in the two months before the 2020 election, drawn from the Twitter parliamentarian database. ChatGPT-4 was given a single zero-shot instruction through the API. Its labels were compared with those from expert coders and from MTurk workers who received the same instructions. ChatGPT-4 was more accurate and more reliable than both human groups, and its bias was equal or lower. It also correctly labeled tweets that required contextual reasoning, such as religious messages aimed at a party's base. The author argues this could make large-scale interpretive text analysis feasible in the social sciences. He also warns that prompt sensitivity, inherited biases and opaque emergent abilities mean every task must be validated.

## Key points
- **Data:** tweets posted by US senators between 3 September and 3 November 2020, taken from the van Vliet et al. Twitter parliamentarian database.
  - Retweets, replies, tweets with URLs and messages under 100 characters were filtered out.
  - The final sample is balanced: 250 Republican and 250 Democratic tweets.
- **Setup:** ChatGPT-4 was queried through the API with one zero-shot prompt asking for an educated guess of "Democrat" or "Republican".
  - It ran 5 times at temperature 0.2 and 5 times at 1.0, for 5000 classifications in total.
- **Results:** every individual human coder scored below the model.
  - The model's mean accuracy was almost the same at both temperatures.
  - MTurk answers were evaluated as a per-tweet majority vote.
- **MTurk quality:** without strict quality controls, MTurk accuracy in pre-testing was barely distinguishable from chance, so best-practice safeguards were added.
- **Caveats:** the author lists prompt sensitivity, reproduced social biases, poorly understood emergent abilities, and the model's failure to signal uncertainty. His recommendation is careful task-specific validation rather than out-of-the-box use.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) · [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

## Cited by

Cited by

80 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 74 not yet.

- [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://md.research.elamur.ai/free/en/papers/bavaresco-2024-llms-instead-human-judges-large/)
- [Exploring the Potential of ChatGPT on Sentence Level Relations: A Focus on Temporal, Causal, and Discourse Relations](https://md.research.elamur.ai/free/en/papers/chan-2024-exploring-potential-chatgpt-sentence-level/)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://md.research.elamur.ai/free/en/papers/li-2024-llms-judges-comprehensive-survey-llm/)
- [Annotating Indian Regional Biases using Large Language Models: Evaluation and Analysis](https://md.research.elamur.ai/free/en/papers/panda-2026-annotating-indian-regional-biases-large/)
- [Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models](https://md.research.elamur.ai/free/en/papers/sengupta-2023-jais-jais-chat-arabic-centric/)
- [SEFL: A Framework for Generating Synthetic Educational Assignment Feedback with LLM Agents](https://md.research.elamur.ai/free/en/papers/zhang-2026-sefl-framework-generating-synthetic-educational/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{tornberg2023chatgpt,
  title = {{ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning}},
  author = {Tornberg, Petter},
  year = {2023},
  doi = {10.48550/arxiv.2304.06588},
  eprint = {2304.06588},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.06588}
}
```

- [← previous ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfer](https://md.research.elamur.ai/free/en/papers/liu-2023-chatgpt-vs-human-authored-text/)
- [next → ChatGPT: Jack of all trades, master of none](https://md.research.elamur.ai/free/en/papers/kocon-2023-chatgpt-jack-all-trades-master/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/2/)

## References

25 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. M Weber, Sociological writings. (Continuum International Publishing Group) Vol. 60, (1994). [find](https://scholar.google.com/scholar?q=M+Weber%2C+Sociological+writings%2E+%28Continuum+International+Publishing+Group%29+Vol%2E+60%2C+%281994%29%2E)
2. A Sayer, Realism and social science. Realism Soc. Sci. pp. 1–224 (1999). [find](https://scholar.google.com/scholar?q=A+Sayer%2C+Realism+and+social+science%2E+Realism+Soc%2E+Sci%2E+pp%2E+1%E2%80%93224+%281999%29%2E)
3. JS Murthy, GM Siddesh, KG Srinivasa, TwitSenti: a real-time Twitter sentiment analysis and visualization framework. J. Inf. & Knowl. Manag. 18, 1950013 (2019) Publisher: World Scientific. [find](https://scholar.google.com/scholar?q=JS+Murthy%2C+GM+Siddesh%2C+KG+Srinivasa%2C+TwitSenti%3A+a+real%2Dtime+Twitter+sentiment+analysis+and+visualization+framework%2E+J%2E+Inf%2E+%26+Knowl%2E+Manag%2E+18%2C+1950013+%282019%29+Publisher%3A+World+Scientific%2E)
4. W Van Atteveldt, MA Van der Velden, M Boukes, The validity of sentiment analysis: Comparing manual annotation, crowd-coding, dictionary approaches, and machine learning algorithms. Commun. Methods Meas. 15, 121–140 (2021) Publisher: Taylor & Francis. [find](https://scholar.google.com/scholar?q=W+Van+Atteveldt%2C+MA+Van+der+Velden%2C+M+Boukes%2C+The+validity+of+sentiment+analysis%3A+Comparing+manual+annotation%2C+crowd%2Dcoding%2C+dictionary+approaches%2C+and+machine+learning+algorithms%2E+Commun%2E+Methods+Mea)
5. M Stubbs, Grammar, text, and ideology: computer-assisted methods in the linguistics of representation. Appl. linguistics 15, 201–223 (1994) Publisher: Oxford University Press. [find](https://scholar.google.com/scholar?q=M+Stubbs%2C+Grammar%2C+text%2C+and+ideology%3A+computer%2Dassisted+methods+in+the+linguistics+of+representation%2E+Appl%2E+linguistics+15%2C+201%E2%80%93223+%281994%29+Publisher%3A+Oxford+University+Press%2E)
6. J Eisenstein, Introduction to natural language processing. (MIT press), (2019). [find](https://scholar.google.com/scholar?q=J+Eisenstein%2C+Introduction+to+natural+language+processing%2E+%28MIT+press%29%2C+%282019%29%2E)
7. T Brown, et al., Language models are few-shot learners. Adv. neural information processing systems 33, 1877–1901 (2020). [find](https://scholar.google.com/scholar?q=T+Brown%2C+et+al%2E%2C+Language+models+are+few%2Dshot+learners%2E+Adv%2E+neural+information+processing+systems+33%2C+1877%E2%80%931901+%282020%29%2E)
8. J Wei, et al., Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022). [find](https://scholar.google.com/scholar?q=J+Wei%2C+et+al%2E%2C+Emergent+abilities+of+large+language+models%2E+arXiv+preprint+arXiv%3A2206%2E07682+%282022%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/)
9. S Bubeck, et al., Sparks of Artificial General Intelligence: Early experiments with GPT-4. arXiv preprint arXiv:2303.12712 (2023). [find](https://scholar.google.com/scholar?q=S+Bubeck%2C+et+al%2E%2C+Sparks+of+Artificial+General+Intelligence%3A+Early+experiments+with+GPT%2D4%2E+arXiv+preprint+arXiv%3A2303%2E12712+%282023%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/)
10. T Kojima, SS Gu, M Reid, Y Matsuo, Y Iwasawa, Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916 (2022). [find](https://scholar.google.com/scholar?q=T+Kojima%2C+SS+Gu%2C+M+Reid%2C+Y+Matsuo%2C+Y+Iwasawa%2C+Large+language+models+are+zero%2Dshot+reasoners%2E+arXiv+preprint+arXiv%3A2205%2E11916+%282022%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/)
11. PY Wu, JA Tucker, J Nagler, S Messing, Large Language Models Can Be Used to Estimate the Ideologies of Politicians in a Zero-Shot Learning Setting. arXiv preprint arXiv:2303.12057 (2023). — [arXiv:2303.12057](https://arxiv.org/abs/2303.12057) [find](https://scholar.google.com/scholar?q=PY+Wu%2C+JA+Tucker%2C+J+Nagler%2C+S+Messing%2C+Large+Language+Models+Can+Be+Used+to+Estimate+the+Ideologies+of+Politicians+in+a+Zero%2DShot+Learning+Setting%2E+arXiv+preprint+arXiv%3A2303%2E12057+%282023%29%2E+%E2%80%94+%5BarXiv%3A230)
12. F Gilardi, M Alizadeh, M Kubli, Chatgpt outperforms crowd-workers for text-annotation tasks. arXiv preprint arXiv:2303.15056 (2023). [find](https://scholar.google.com/scholar?q=F+Gilardi%2C+M+Alizadeh%2C+M+Kubli%2C+Chatgpt+outperforms+crowd%2Dworkers+for+text%2Dannotation+tasks%2E+arXiv+preprint+arXiv%3A2303%2E15056+%282023%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/gilardi-2023-chatgpt-outperforms-crowd-workers-text/)
13. LP Argyle, et al., Out of One, Many: Using Language Models to Simulate Human Samples. arXiv preprint arXiv:2209.06899 (2022). [find](https://scholar.google.com/scholar?q=LP+Argyle%2C+et+al%2E%2C+Out+of+One%2C+Many%3A+Using+Language+Models+to+Simulate+Human+Samples%2E+arXiv+preprint+arXiv%3A2209%2E06899+%282022%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/argyle-2022-out-one-many-using-language/)
14. M Chmielewski, SC Kucker, An mturk crisis? shifts in data quality and the impact on study results. Soc. Psychol. Pers. Sci. 11, 464–473 (2020). [find](https://scholar.google.com/scholar?q=M+Chmielewski%2C+SC+Kucker%2C+An+mturk+crisis%3F+shifts+in+data+quality+and+the+impact+on+study+results%2E+Soc%2E+Psychol%2E+Pers%2E+Sci%2E+11%2C+464%E2%80%93473+%282020%29%2E)
15. DM Ziegler, et al., Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 (2019). [find](https://scholar.google.com/scholar?q=DM+Ziegler%2C+et+al%2E%2C+Fine%2Dtuning+language+models+from+human+preferences%2E+arXiv+preprint+arXiv%3A1909%2E08593+%282019%29%2E)
16. J Schulman, et al., ChatGPT: Optimizing language models for dialogue (2022). [find](https://scholar.google.com/scholar?q=J+Schulman%2C+et+al%2E%2C+ChatGPT%3A+Optimizing+language+models+for+dialogue+%282022%29%2E)
17. PW Anderson, More is different. Science 177, 393–396 (1972) Publisher: JSTOR. [find](https://scholar.google.com/scholar?q=PW+Anderson%2C+More+is+different%2E+Science+177%2C+393%E2%80%93396+%281972%29+Publisher%3A+JSTOR%2E)
18. Y Bang, et al., A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 (2023). [find](https://scholar.google.com/scholar?q=Y+Bang%2C+et+al%2E%2C+A+multitask%2C+multilingual%2C+multimodal+evaluation+of+chatgpt+on+reasoning%2C+hallucination%2C+and+interactivity%2E+arXiv+preprint+arXiv%3A2302%2E04023+%282023%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/)
19. HG Frankfurt, On bullshit. (Princeton University Press), (2005). [find](https://scholar.google.com/scholar?q=HG+Frankfurt%2C+On+bullshit%2E+%28Princeton+University+Press%29%2C+%282005%29%2E)
20. L Lucy, D Bamman, Gender and representation bias in GPT-3 generated stories in Proceedings of the Third Workshop on Narrative Understanding. pp. 48–55 (2021). [find](https://scholar.google.com/scholar?q=L+Lucy%2C+D+Bamman%2C+Gender+and+representation+bias+in+GPT%2D3+generated+stories+in+Proceedings+of+the+Third+Workshop+on+Narrative+Understanding%2E+pp%2E+48%E2%80%9355+%282021%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/lucy-2021-gender-representation-bias-gpt-3/)
21. JT Ornstein, EN Blasingame, JS Truscott, How to Train Your Stochastic Parrot: Large Language Models for Political Texts. Work. paper. [https://joeornstein.github.io/publications/ornstein-blasingame-truscott.pdf](https://joeornstein.github.io/publications/ornstein-blasingame-truscott.pdf) (2022). [↗](https://joeornstein.github.io/publications/ornstein-blasingame-truscott.pdf)
22. A Palmer, A Spirling, Large Language Models Can Argue in Convincing and Novel Ways About Politics: Evidence from Experiments and Human Judgement, (Working paper), Technical report (2023). [find](https://scholar.google.com/scholar?q=A+Palmer%2C+A+Spirling%2C+Large+Language+Models+Can+Argue+in+Convincing+and+Novel+Ways+About+Politics%3A+Evidence+from+Experiments+and+Human+Judgement%2C+%28Working+paper%29%2C+Technical+report+%282023%29%2E)
23. L van Vliet, P Törnberg, J Uitermark, The Twitter parliamentarian database: Analyzing Twitter politics across 26 countries. PloS one 15, e0237073 (2020). [find](https://scholar.google.com/scholar?q=L+van+Vliet%2C+P+T%C3%B6rnberg%2C+J+Uitermark%2C+The+Twitter+parliamentarian+database%3A+Analyzing+Twitter+politics+across+26+countries%2E+PloS+one+15%2C+e0237073+%282020%29%2E)
24. M Reiss, Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark. OSF Prepr. (2023). [find](https://scholar.google.com/scholar?q=M+Reiss%2C+Testing+the+Reliability+of+ChatGPT+for+Text+Annotation+and+Classification%3A+A+Cautionary+Remark%2E+OSF+Prepr%2E+%282023%29%2E)
25. J Grimmer, BM Stewart, Text as data: The promise and pitfalls of automatic content analysis methods for political texts. Polit. analysis 21, 267–297 (2013). [find](https://scholar.google.com/scholar?q=J+Grimmer%2C+BM+Stewart%2C+Text+as+data%3A+The+promise+and+pitfalls+of+automatic+content+analysis+methods+for+political+texts%2E+Polit%2E+analysis+21%2C+267%E2%80%93297+%282013%29%2E)