---
title: "Artificial Artificial Artificial Intelligence: Crowd Workers Widely"
description: "Crowdsourcing platforms are a cheap, standard source of human annotations. 34 references, cited by 57 papers here."
authors: ["Veniamin Veselovsky", "Manoel Horta Ribeiro", "Robert West"]
year: 2023
venue: "arXiv:2306.07899"
source: "https://arxiv.org/abs/2306.07899"
doi: "10.48550/arxiv.2306.07899"
arxiv: "2306.07899"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/veselovsky-2023-artificial-artificial-artificial-intelligence-crowd/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/veselovsky-2023-artificial-artificial-artificial-intelligence-crowd/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks

# Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks

**Authors:** [Veniamin Veselovsky](https://md.research.elamur.ai/free/en/authors/veniamin-veselovsky/), Manoel Horta Ribeiro and [Robert West](https://md.research.elamur.ai/free/en/authors/robert-west/) · **Year:** 2023 · **Venue:** arXiv:2306.07899 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2306.07899) · [PDF](https://arxiv.org/pdf/2306.07899) · [DOI 10.48550/arxiv.2306.07899](https://doi.org/10.48550/arxiv.2306.07899) · [arXiv 2306.07899](https://arxiv.org/abs/2306.07899) · [OpenAlex](https://openalex.org/W4380715494)

**Citations:** 70 citations (OpenAlex, 2026-10-14) · [57](https://md.research.elamur.ai/free/en/papers/veselovsky-2023-artificial-artificial-artificial-intelligence-crowd/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary

Crowdsourcing platforms are a cheap, standard source of human annotations. Researchers increasingly use them to collect gold-standard data for checking LLM outputs. This paper asks whether that data is still produced by people, since workers paid per task have a financial reason to hand the job to ChatGPT. On Amazon Mechanical Turk, the authors re-ran a reduced version of a task introduced by Horta Ribeiro et al. (2019), in which workers summarize medical abstracts. Workers condensed NEJM abstracts into summaries of about 100 words, and the task logged their keystrokes and copy–paste events. Off-the-shelf detectors such as GPTZero did poorly on this material. The team therefore fine-tuned a task-specific classifier that separates pre-LLM human summaries from ChatGPT-generated ones. They applied it to the new submissions and checked the predictions against the keystroke logs. They estimate that roughly a third to nearly half of workers relied on LLMs. This is a warning for anyone who treats crowd work as authentic human text, especially when task instructions can be pasted straight into a prompt. Whether the finding generalizes to tasks that are hard for LLMs remains open.

## Key points

- Estimated LLM usage among MTurk workers in the summarization task is 33–46%, based on a synthetic-vs.-real classifier combined with keystroke analysis. Code and data are released as GPTurk (github.com/epfl-dlab/GPTurk).
- Setup: 16 NEJM abstracts covering four topics (vaccination, breast cancer, cardiovascular disease, nutrition). Workers wrote 48 summaries, 44 distinct workers took part, and 46 summaries remained after de-duplication. Pay was 1 USD per summary (about 15 USD/hour), and data was collected around 1 June 2023.
- Generic detectors proved unreliable. GPTZero flagged only 6 of 10 ChatGPT-written summaries, and OpenAI's own classifier is reported to catch only 26% of LLM text. This motivated a bespoke, low-cost fine-tuned detector.
- Paste behavior supports the classifier's predictions. Of 13 pasted summaries with under 10% overlap with the source abstract, 10 (76%) were classified as synthetic. This suggests workers were pasting LLM output rather than copying from the abstract.
- The authors expect crowd work to shift toward humans acting as filters or validators of model output. They also note privacy concerns around keystroke tracking, which they restricted to the edit box and copy–paste actions.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [École Polytechnique Fédérale de Lausanne](https://md.research.elamur.ai/free/en/universities/ecole-polytechnique-federale-de-lausanne/) — University, Lausanne, Switzerland · 23 works · 6,687 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 744 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 221 works · [🤝 General Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/general-social-sciences/) · 87 works · [🖥️ Computer Science Applications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-science-applications/) · 42 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works

## Cite

Cite as: Veniamin Veselovsky, Manoel Horta Ribeiro and Robert West. 2023. Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks. arXiv:2306.07899.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{veselovsky2023artificial,
  title = {{Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks}},
  author = {Veniamin Veselovsky, Manoel Horta Ribeiro and Robert West},
  year = {2023},
  doi = {10.48550/arxiv.2306.07899},
  eprint = {2306.07899},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.07899}
}
```

- [← previous A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capability](https://md.research.elamur.ai/free/en/papers/liu-2023-comprehensive-evaluation-chatgpt-s-zero/)
- [next → SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents](https://md.research.elamur.ai/free/en/papers/zhou-2023-sotopia-interactive-evaluation-social-intelligence/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

34 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Panagiotis G Ipeirotis. 2010. Demographics of mechanical turk. [find](https://scholar.google.com/scholar?q=Panagiotis+G+Ipeirotis%2E+2010%2E+Demographics+of+mechanical+turk%2E)
2. Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua Gubler, Christopher Rytting, and David Wingate. 2022. [Out of one, many: Using language models to simulate human samples](https://md.research.elamur.ai/free/en/papers/argyle-2022-out-one-many-using-language/). arXiv preprint arXiv:2209.06899.
3. Manoel Horta Ribeiro, Kristina Gligoric, and Robert West. 2019. Message distortion in information cascades. In The World Wide Web Conference, pages 681–692. [find](https://scholar.google.com/scholar?q=Manoel+Horta+Ribeiro%2C+Kristina+Gligoric%2C+and+Robert+West%2E+2019%2E+Message+distortion+in+information+cascades%2E+In+The+World+Wide+Web+Conference%2C+pages+681%E2%80%93692%2E)
4. John J Horton. 2023. Large language models as simulated economic agents: What can we learn from homo silicus? arXiv preprint arXiv:2301.07543. — [arXiv:2301.07543](https://arxiv.org/abs/2301.07543) [find](https://scholar.google.com/scholar?q=John+J+Horton%2E+2023%2E+Large+language+models+as+simulated+economic+agents%3A+What+can+we+learn+from+homo+silicus%3F+arXiv+preprint+arXiv%3A2301%2E07543%2E+%E2%80%94+%5BarXiv%3A2301%2E07543%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2301%2E07543%29)
5. Maurice Jakesch, Jeffrey T Hancock, and Mor Naaman. 2023. Human heuristics for ai-generated language are flawed. Proceedings of the National Academy of Sciences, 120(11):e2208839120. [find](https://scholar.google.com/scholar?q=Maurice+Jakesch%2C+Jeffrey+T+Hancock%2C+and+Mor+Naaman%2E+2023%2E+Human+heuristics+for+ai%2Dgenerated+language+are+flawed%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+120%2811%29%3Ae2208839120%2E)
6. Ryan Kennedy, Scott Clifford, Tyler Burleigh, Philip D Waggoner, Ryan Jewell, and Nicholas JG Winter. 2020. The shape of and solutions to the mturk quality crisis. Political Science Research and Methods, 8(4):614–629. [find](https://scholar.google.com/scholar?q=Ryan+Kennedy%2C+Scott+Clifford%2C+Tyler+Burleigh%2C+Philip+D+Waggoner%2C+Ryan+Jewell%2C+and+Nicholas+JG+Winter%2E+2020%2E+The+shape+of+and+solutions+to+the+mturk+quality+crisis%2E+Political+Science+Research+and+Metho)
7. John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023. [A watermark for large language models](https://md.research.elamur.ai/free/en/papers/kirchenbauer-2023-watermark-large-language-models/). arXiv preprint arXiv:2301.10226.
8. Jan Kocoń, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, et al. 2023. [Chatgpt: Jack of all trades, master of none](https://md.research.elamur.ai/free/en/papers/kocon-2023-chatgpt-jack-all-trades-master/). arXiv preprint arXiv:2302.10724.
9. Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. 2023. [Can ai-generated text be reliably detected?](https://md.research.elamur.ai/free/en/papers/sadasivan-2023-ai-generated-text-reliably-detected/) arXiv preprint arXiv:2303.11156.
10. Matthew J Salganik. 2019. Bit by bit: Social research in the digital age. Princeton University Press. [find](https://scholar.google.com/scholar?q=Matthew+J+Salganik%2E+2019%2E+Bit+by+bit%3A+Social+research+in+the+digital+age%2E+Princeton+University+Press%2E)
11. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. [Whose opinions do language models reflect?](https://md.research.elamur.ai/free/en/papers/santurkar-2023-whose-opinions-language-models-reflect/) arXiv preprint arXiv:2303.17548.
12. Oscar Schwartz. 2019. Untold history of ai: How amazon’s mechanical turkers got squeezed inside the machine. IEEE Spectrum. [find](https://scholar.google.com/scholar?q=Oscar+Schwartz%2E+2019%2E+Untold+history+of+ai%3A+How+amazon%E2%80%99s+mechanical+turkers+got+squeezed+inside+the+machine%2E+IEEE+Spectrum%2E)
13. Alisa Liu, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. 2022. [Wanli: Worker and ai collaboration for natural language inference dataset creation](https://md.research.elamur.ai/free/en/papers/liu-2022-wanli-worker-ai-collaboration-natural/). arXiv preprint arXiv:2201.05955.
14. Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. 2023. [The curse of recursion: Training on generated data makes models forget](https://md.research.elamur.ai/free/en/papers/shumailov-2023-curse-recursion-training-generated-data/).
15. Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. [Chatgpt as a factual inconsistency evaluator for text summarization](https://md.research.elamur.ai/free/en/papers/luo-2023-chatgpt-factual-inconsistency-evaluator-text/).
16. Scott M Smith, Catherine A Roster, Linda L Golden, and Gerald S Albaum. 2016. A multi-group analysis of online survey respondent data quality: Comparing a regular usa consumer panel to mturk samples. Journal of Business Research, 69(8):3139–3148. [find](https://scholar.google.com/scholar?q=Scott+M+Smith%2C+Catherine+A+Roster%2C+Linda+L+Golden%2C+and+Gerald+S+Albaum%2E+2016%2E+A+multi%2Dgroup+analysis+of+online+survey+respondent+data+quality%3A+Comparing+a+regular+usa+consumer+panel+to+mturk+samples%2E+)
17. Matthew Marge, Satanjeev Banerjee, and Alexander I Rudnicky. 2010. Using the amazon mechanical turk for transcription of spoken language. In 2010 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 5270–5273. IEEE. [find](https://scholar.google.com/scholar?q=Matthew+Marge%2C+Satanjeev+Banerjee%2C+and+Alexander+I+Rudnicky%2E+2010%2E+Using+the+amazon+mechanical+turk+for+transcription+of+spoken+language%2E+In+2010+IEEE+International+Conference+on+Acoustics%2C+Speech+and)
18. Morgan N McCredie and Leslie C Morey. 2019. Who are the turkers? a characterization of mturk workers using the personality assessment inventory. Assessment, 26(5):759–766. [find](https://scholar.google.com/scholar?q=Morgan+N+McCredie+and+Leslie+C+Morey%2E+2019%2E+Who+are+the+turkers%3F+a+characterization+of+mturk+workers+using+the+personality+assessment+inventory%2E+Assessment%2C+26%285%29%3A759%E2%80%93766%2E)
19. Aaron M Ogletree and Benjamin Katz. 2021. How do older adults recruited using mturk differ from those in a national probability sample? The International Journal of Aging and Human Development, 93(2):700–721. [find](https://scholar.google.com/scholar?q=Aaron+M+Ogletree+and+Benjamin+Katz%2E+2021%2E+How+do+older+adults+recruited+using+mturk+differ+from+those+in+a+national+probability+sample%3F+The+International+Journal+of+Aging+and+Human+Development%2C+93%282%29%3A)
20. Alexander Sorokin and David Forsyth. 2008. Utility data annotation with amazon mechanical turk. In 2008 IEEE computer society conference on computer vision and pattern recognition workshops, pages 1–8. IEEE. [find](https://scholar.google.com/scholar?q=Alexander+Sorokin+and+David+Forsyth%2E+2008%2E+Utility+data+annotation+with+amazon+mechanical+turk%2E+In+2008+IEEE+computer+society+conference+on+computer+vision+and+pattern+recognition+workshops%2C+pages+1%E2%80%938)
21. Mirac Suzgun, Stuart M Shieber, and Dan Jurafsky. 2023. string2string: A modern python library for string-to-string algorithms. arXiv preprint arXiv:2304.14395. [find](https://scholar.google.com/scholar?q=Mirac+Suzgun%2C+Stuart+M+Shieber%2C+and+Dan+Jurafsky%2E+2023%2E+string2string%3A+A+modern+python+library+for+string%2Dto%2Dstring+algorithms%2E+arXiv+preprint+arXiv%3A2304%2E14395%2E)
22. Petter Törnberg. 2023. [Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning](https://md.research.elamur.ai/free/en/papers/tornberg-2023-chatgpt-4-outperforms-experts-crowd/). arXiv preprint arXiv:2304.06588.
23. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
24. Yikang Pan, Liangming Pan, Wenhu Chen, Preslav Nakov, Min-Yen Kan, and William Yang Wang. 2023. [On the risk of misinformation pollution with large language models](https://md.research.elamur.ai/free/en/papers/pan-2023-risk-misinformation-pollution-large-language/). arXiv preprint arXiv:2305.13661.
25. Vivek Verma, Eve Fleisig, Nicholas Tomlin, and Dan Klein. 2023. [Ghostbuster: Detecting text ghostwritten by large language models](https://md.research.elamur.ai/free/en/papers/verma-2024-ghostbuster-detecting-text-ghostwritten-large/). arXiv preprint arXiv:2305.15047.
26. Amandalynne Paullada, Inioluwa Deborah Raji, Emily M Bender, Emily Denton, and Alex Hanna. 2021. Data and its (dis) contents: A survey of dataset development and use in machine learning research. Patterns, 2(11):100336. [find](https://scholar.google.com/scholar?q=Amandalynne+Paullada%2C+Inioluwa+Deborah+Raji%2C+Emily+M+Bender%2C+Emily+Denton%2C+and+Alex+Hanna%2E+2021%2E+Data+and+its+%28dis%29+contents%3A+A+survey+of+dataset+development+and+use+in+machine+learning+research%2E+Patt)
27. Veniamin Veselovsky, Manoel Horta Ribeiro, Akhil Arora, Martin Josifoski, Ashton Anderson, and Robert West. 2023. Generating faithful synthetic data with large language models: A case study in computational social science. arXiv preprint arXiv:2305.15041. — [arXiv:2305.15041](https://arxiv.org/abs/2305.15041) [find](https://scholar.google.com/scholar?q=Veniamin+Veselovsky%2C+Manoel+Horta+Ribeiro%2C+Akhil+Arora%2C+Martin+Josifoski%2C+Ashton+Anderson%2C+and+Robert+West%2E+2023%2E+Generating+faithful+synthetic+data+with+large+language+models%3A+A+case+study+in+computa)
28. Gordon Pennycook, Ziv Epstein, Mohsen Mosleh, Antonio A Arechar, Dean Eckles, and David G Rand. 2021. Shifting attention to accuracy can reduce misinformation online. Nature, 592(7855):590–595. [find](https://scholar.google.com/scholar?q=Gordon+Pennycook%2C+Ziv+Epstein%2C+Mohsen+Mosleh%2C+Antonio+A+Arechar%2C+Dean+Eckles%2C+and+David+G+Rand%2E+2021%2E+Shifting+attention+to+accuracy+can+reduce+misinformation+online%2E+Nature%2C+592%287855%29%3A590%E2%80%93595%2E)
29. Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. [Is chatgpt a good nlg evaluator? a preliminary study](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-nlg-evaluator-preliminary/). arXiv preprint arXiv:2303.04048.
30. Jürgen Rudolph, Shannon Tan, and Samson Tan. 2023. War of the chatbots: Bard, bing chat, chatgpt, ernie and beyond. the new ai gold rush and its impact on higher education. Journal of Applied Learning and Teaching, 6(1). [find](https://scholar.google.com/scholar?q=J%C3%BCrgen+Rudolph%2C+Shannon+Tan%2C+and+Samson+Tan%2E+2023%2E+War+of+the+chatbots%3A+Bard%2C+bing+chat%2C+chatgpt%2C+ernie+and+beyond%2E+the+new+ai+gold+rush+and+its+impact+on+higher+education%2E+Journal+of+Applied+Learning)
31. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. [Text embeddings by weakly-supervised contrastive pre-training](https://md.research.elamur.ai/free/en/papers/wang-2022-text-embeddings-weakly-supervised-contrastive/). arXiv preprint arXiv:2212.03533.
32. Peipeng Yu, Jiahan Chen, Xuan Feng, and Zhihua Xia. 2023. Cheat: A large-scale dataset for detecting chatgpt-written abstracts. arXiv preprint arXiv:2304.12008. — [arXiv:2304.12008](https://arxiv.org/abs/2304.12008) [find](https://scholar.google.com/scholar?q=Peipeng+Yu%2C+Jiahan+Chen%2C+Xuan+Feng%2C+and+Zhihua+Xia%2E+2023%2E+Cheat%3A+A+large%2Dscale+dataset+for+detecting+chatgpt%2Dwritten+abstracts%2E+arXiv+preprint+arXiv%3A2304%2E12008%2E+%E2%80%94+%5BarXiv%3A2304%2E12008%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
33. Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song. 2022. A survey of controllable text generation using transformer-based pre-trained language models. arXiv preprint arXiv:2201.05337. [find](https://scholar.google.com/scholar?q=Hanqing+Zhang%2C+Haolin+Song%2C+Shaoyu+Li%2C+Ming+Zhou%2C+and+Dawei+Song%2E+2022%2E+A+survey+of+controllable+text+generation+using+transformer%2Dbased+pre%2Dtrained+language+models%2E+arXiv+preprint+arXiv%3A2201%2E05337%2E)
34. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023. [Can large language models transform computational social science?](https://md.research.elamur.ai/free/en/papers/ziems-2023-large-language-models-transform-computational/) arXiv preprint arXiv:2305.03514.

*Updated 24 October 2026*