---
title: "Knowledge Distillation in Automated Annotation: Supervised Text"
description: "The authors test a knowledge-distillation workflow with four steps: Tune and validate GPT-4 few-shot… 51 references, cited by 9 papers here."
authors: ["Nicholas Pangakis", "Samuel Wolken"]
year: 2024
venue: "arXiv:2406.17633"
source: "https://arxiv.org/abs/2406.17633"
doi: "10.48550/arxiv.2406.17633"
arxiv: "2406.17633"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pangakis-2024-knowledge-distillation-automated-annotation-supervised/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pangakis-2024-knowledge-distillation-automated-annotation-supervised/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels

# Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels

**Authors:** Nicholas Pangakis and Samuel Wolken · **Year:** 2024 · **Venue:** arXiv:2406.17633 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2406.17633) · [PDF](https://arxiv.org/pdf/2406.17633) · [DOI 10.48550/arxiv.2406.17633](https://doi.org/10.48550/arxiv.2406.17633) · [arXiv 2406.17633](https://arxiv.org/abs/2406.17633) · [OpenAlex](https://openalex.org/W4401042631)

**Citations:** 33 citations (OpenAlex, 2026-10-14) · [9](https://md.research.elamur.ai/free/en/papers/pangakis-2024-knowledge-distillation-automated-annotation-supervised/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Supervised text classifiers in computational social science (CSS) usually need human-annotated training data, which is costly and slow to produce. Prompting a large generative model to label an entire corpus is an alternative, but it becomes expensive at scale. The authors test a knowledge-distillation workflow with four steps:

1. Tune and validate GPT-4 few-shot prompts on 250 human-labeled texts.
2. Have GPT-4 label 1,000 more texts.
3. Fine-tune compact "student" classifiers (BERT, RoBERTa, DistilBERT) on those labels.
4. Evaluate the students against 1,000 held-out human-labeled items.

They replicate 14 English classification tasks from recent CSS articles in high-impact journals. These datasets are kept in password-protected archives, so they are less likely to appear in LLM training data. Students trained on GPT-4 labels perform only slightly below those trained on human labels and nearly match GPT-4 few-shot itself. Researchers can therefore use an LLM to build training data instead of labeling everything with it. Human validation remains essential, because students inherit the LLM's mistakes and biases.

## Key points
- Median F1 across tasks: 0.641 for DistilBERT and 0.624 for BERT trained on 1,000 human labels, 0.592 for GPT-4 few-shot, and 0.586 for BERT trained on 1,000 GPT-4 labels. The median gap between GPT-label and human-label models is 0.039.
- Students trained on GPT-4 labels differ from GPT-4 few-shot by a median F1 of just 0.006.
- LLM-derived models favor recall over precision. GPT-4 few-shot reaches a median recall of 0.80, and BERT/DistilBERT on GPT labels reach 0.746. By contrast, BERT trained on human labels has precision 0.214 higher than its GPT-label counterpart.
- Ablation studies examine:
  - noisy GPT labels;
  - re-running GPT-4 six months later to check for drift;
  - XLNet and Mistral-7B, including Mistral-7B as an open-source few-shot baseline.
- Cost motivation: classifying 6.2M tweets on four dimensions with GPT-4 alone would cost roughly $9,000. The approach distills a very large teacher (estimated at about 1.7T parameters) into BERT Base, which has about 110M parameters.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Pennsylvania](https://md.research.elamur.ai/free/en/universities/university-of-pennsylvania/) — University, Philadelphia, United States · 221 works · 63,234 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 General Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/general-social-sciences/) · 227 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 242 works

## Cite

Cite as: Nicholas Pangakis and Samuel Wolken. 2024. Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels. arXiv:2406.17633.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{pangakis2024knowledge,
  title = {{Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels}},
  author = {Nicholas Pangakis and Samuel Wolken},
  year = {2024},
  doi = {10.48550/arxiv.2406.17633},
  eprint = {2406.17633},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.17633}
}
```

- [← previous Improving Dictionary Learning with Gated Sparse Autoencoders](https://md.research.elamur.ai/free/en/papers/rajamanoharan-2024-improving-dictionary-learning-gated-sparse/)
- [next → LUQ: Long-text Uncertainty Quantification for LLMs](https://md.research.elamur.ai/free/en/papers/zhang-2024-luq-long-text-uncertainty-quantification/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/)

## References

51 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Nishant Balepur, Abhilasha Ravichander, and Rachel Rudinger. 2024. [Artifacts or abduction: How do llms answer multiple-choice questions without the question?](https://md.research.elamur.ai/free/en/papers/balepur-2024-artifacts-or-abduction-how-do/)
2. Berfu Büyüköz, Ali Hürriyetoğlu, and Arzucan Özgür. 2020. Analyzing elmo and distilbert on socio-political news classification. In Proceedings of the Workshop on Automated Extraction of Socio-political Events from News 2020, pages 9–18. [find](https://scholar.google.com/scholar?q=Berfu+B%C3%BCy%C3%BCk%C3%B6z%2C+Ali+H%C3%BCrriyetog%CC%86lu%2C+and+Arzucan+%C3%96zg%C3%BCr%2E+2020%2E+Analyzing+elmo+and+distilbert+on+socio%2Dpolitical+news+classification%2E+In+Proceedings+of+the+Workshop+on+Automated+Extraction+of+Socio%2Dpolitic)
3. Dallas Card, Serina Chang, Chris Becker, Julia Mendelsohn, Rob Voigt, Leah Boustan, Ran Abramitzky, and Dan Jurafsky. 2022. Computational analysis of 140 years of us political speeches reveals more positive but increasingly polarized framing of immigration. Proceedings of the National Academy of Sci… [find](https://scholar.google.com/scholar?q=Dallas+Card%2C+Serina+Chang%2C+Chris+Becker%2C+Julia+Mendelsohn%2C+Rob+Voigt%2C+Leah+Boustan%2C+Ran+Abramitzky%2C+and+Dan+Jurafsky%2E+2022%2E+Computational+analysis+of+140+years+of+us+political+speeches+reveals+more+po)
4. Lingjiao Chen, Matei Zaharia, and James Zou. 2023a. How is chatgpt’s behavior changing over time? [find](https://scholar.google.com/scholar?q=Lingjiao+Chen%2C+Matei+Zaharia%2C+and+James+Zou%2E+2023a%2E+How+is+chatgpt%E2%80%99s+behavior+changing+over+time%3F)
5. Zhikai Chen, Haitao Mao, Hongzhi Wen, Haoyu Han, Wei Jin, Haiyang Zhang, and Hui Liuand Jiliang Tang. 2023b. Label-free node classification on graphs with large language models (llms). [find](https://scholar.google.com/scholar?q=Zhikai+Chen%2C+Haitao+Mao%2C+Hongzhi+Wen%2C+Haoyu+Han%2C+Wei+Jin%2C+Haiyang+Zhang%2C+and+Hui+Liuand+Jiliang+Tang%2E+2023b%2E+Label%2Dfree+node+classification+on+graphs+with+large+language+models+%28llms%29%2E)
6. Cheng-Han Chiang and Hung-yi Lee. 2023. Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937. [find](https://scholar.google.com/scholar?q=Cheng%2DHan+Chiang+and+Hung%2Dyi+Lee%2E+2023%2E+Can+large+language+models+be+an+alternative+to+human+evaluations%3F+arXiv+preprint+arXiv%3A2305%2E01937%2E)
7. Michael Chmielewski and Sarah C. Kucker. 2020. An mturk crisis? shifts in data quality and the impact on study results. Social Psychological and Personality Science, 11(4):464–473. [find](https://scholar.google.com/scholar?q=Michael+Chmielewski+and+Sarah+C%2E+Kucker%2E+2020%2E+An+mturk+crisis%3F+shifts+in+data+quality+and+the+impact+on+study+results%2E+Social+Psychological+and+Personality+Science%2C+11%284%29%3A464%E2%80%93473%2E)
8. Sayantan Dasgupta, Trevor Cohn, and Timothy Baldwin. 2023. Cost-effective distillation of large language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7346–7354, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sayantan+Dasgupta%2C+Trevor+Cohn%2C+and+Timothy+Baldwin%2E+2023%2E+Cost%2Deffective+distillation+of+large+language+models%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2023%2C+pages+7346%E2%80%937354)
9. Jacob Devlin, Ming-Wei Chang, Kenton Le, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
10. Bosheng Ding, Chengwei Qin, Linlin Liu, Lidong Bing, Shafiq Joty, and Boyang Li. 2022. [Is gpt-3 a good data annotator?](https://md.research.elamur.ai/free/en/papers/ding-2022-gpt-3-good-data-annotator/)
11. Justin Grimmer, Margaret E. Roberts, and Brandon Stewart. 2022. Text as Data: A New Framework for Machine Learning and the Social Sciences. Princeton University Press. [find](https://scholar.google.com/scholar?q=Justin+Grimmer%2C+Margaret+E%2E+Roberts%2C+and+Brandon+Stewart%2E+2022%2E+Text+as+Data%3A+A+New+Framework+for+Machine+Learning+and+the+Social+Sciences%2E+Princeton+University+Press%2E)
12. Justin Grimmer and Brandon M. Stewart. 2013. Text as data: The promise and pitfalls of automatic content analysis methods for political texts. Political Analysis, 21(3):267–297. [find](https://scholar.google.com/scholar?q=Justin+Grimmer+and+Brandon+M%2E+Stewart%2E+2013%2E+Text+as+data%3A+The+promise+and+pitfalls+of+automatic+content+analysis+methods+for+political+texts%2E+Political+Analysis%2C+21%283%29%3A267%E2%80%93297%2E)
13. Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2023. [Annollm: Making large language models to be better crowdsourced annotators](https://md.research.elamur.ai/free/en/papers/he-2023-annollm-making-large-language-models/).
14. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/).
15. Daniel J. Hopkins, Yphtach Lelkes, and Samuel Wolken. 2024. The rise of and demand for identity-oriented media coverage. American Journal of Political Science. [find](https://scholar.google.com/scholar?q=Daniel+J%2E+Hopkins%2C+Yphtach+Lelkes%2C+and+Samuel+Wolken%2E+2024%2E+The+rise+of+and+demand+for+identity%2Doriented+media+coverage%2E+American+Journal+of+Political+Science%2E)
16. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
17. Ross Deans Kristensen-McLachlan, Miceal Canavan, Márton Kardos, Mia Jacobsen, and Lene Aarøe. 2023. Chatbots are not reliable text annotators. [find](https://scholar.google.com/scholar?q=Ross+Deans+Kristensen%2DMcLachlan%2C+Miceal+Canavan%2C+M%C3%A1rton+Kardos%2C+Mia+Jacobsen%2C+and+Lene+Aar%C3%B8e%2E+2023%2E+Chatbots+are+not+reliable+text+annotators%2E)
18. David MJ Lazer, Alex Pentland, Duncan J Watts, Sinan Aral, Susan Athey, Noshir Contractor, Deen Freelon, Sandra Gonzalez-Bailon, Gary King, and Helen Margetts. 2020. Computational social science: Obstacles and opportunities. Science, 369(6507):1060–1062. [find](https://scholar.google.com/scholar?q=David+MJ+Lazer%2C+Alex+Pentland%2C+Duncan+J+Watts%2C+Sinan+Aral%2C+Susan+Athey%2C+Noshir+Contractor%2C+Deen+Freelon%2C+Sandra+Gonzalez%2DBailon%2C+Gary+King%2C+and+Helen+Margetts%2E+2020%2E+Computational+social+science%3A+Obst)
19. Benjamin D. Douglas, Patrick J. Ewell, and Markus Braue. 2023. Data quality in online human-subjects research: Comparisons between mturk, prolific, cloudresearch, qualtrics, and sona. PLoS One, 18. [find](https://scholar.google.com/scholar?q=Benjamin+D%2E+Douglas%2C+Patrick+J%2E+Ewell%2C+and+Markus+Braue%2E+2023%2E+Data+quality+in+online+human%2Dsubjects+research%3A+Comparisons+between+mturk%2C+prolific%2C+cloudresearch%2C+qualtrics%2C+and+sona%2E+PLoS+One%2C+18%2E)
20. Changmao Li and Jeffrey Flanigan. 2023. Task contamination: Language models may not be few-shot anymore. [find](https://scholar.google.com/scholar?q=Changmao+Li+and+Jeffrey+Flanigan%2E+2023%2E+Task+contamination%3A+Language+models+may+not+be+few%2Dshot+anymore%2E)
21. Naoki Egami, Christian J. Fong, Justin Grimmer, Margaret E. Roberts, and Brandon M. Stewart. 2022. How to make causal inferences using texts. Science Advances, 8(42):eabg2652. [find](https://scholar.google.com/scholar?q=Naoki+Egami%2C+Christian+J%2E+Fong%2C+Justin+Grimmer%2C+Margaret+E%2E+Roberts%2C+and+Brandon+M%2E+Stewart%2E+2022%2E+How+to+make+causal+inferences+using+texts%2E+Science+Advances%2C+8%2842%29%3Aeabg2652%2E)
22. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/).
23. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. [Chatgpt outperforms crowd-workers for text-annotation tasks](https://md.research.elamur.ai/free/en/papers/gilardi-2023-chatgpt-outperforms-crowd-workers-text/).
24. Inbal Magar and Roy Schwartz. 2022. Data contamination: From memorization to exploitation. [find](https://scholar.google.com/scholar?q=Inbal+Magar+and+Roy+Schwartz%2E+2022%2E+Data+contamination%3A+From+memorization+to+exploitation%2E)
25. Jonas Golde, Patrick Haller, Felix Hamborg, Julian Risch, and Alan Akbik. 2023. Fabricator: An open source toolkit for generating labeled training data with teacher llms. [find](https://scholar.google.com/scholar?q=Jonas+Golde%2C+Patrick+Haller%2C+Felix+Hamborg%2C+Julian+Risch%2C+and+Alan+Akbik%2E+2023%2E+Fabricator%3A+An+open+source+toolkit+for+generating+labeled+training+data+with+teacher+llms%2E)
26. Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. 2021. Knowledge distillation: A survey. International Journal of Computer Vision, page 1789–1819. [find](https://scholar.google.com/scholar?q=Jianping+Gou%2C+Baosheng+Yu%2C+Stephen+J%2E+Maybank%2C+and+Dacheng+Tao%2E+2021%2E+Knowledge+distillation%3A+A+survey%2E+International+Journal+of+Computer+Vision%2C+page+1789%E2%80%931819%2E)
27. Wes McKinney. 2011. pandas: a foundational python library for data analysis and statistics. Python for high performance and scientific computing, 14(9):1–9. [find](https://scholar.google.com/scholar?q=Wes+McKinney%2E+2011%2E+pandas%3A+a+foundational+python+library+for+data+analysis+and+statistics%2E+Python+for+high+performance+and+scientific+computing%2C+14%289%29%3A1%E2%80%939%2E)
28. Jonathan Mellon, Jack Bailey, Ralph Scott, James Breckwoldt, and Marta Miori. 2022. Does gpt-3 know what the most important issue is? using large language models to code open-text social survey responses at scale. Working paper. [find](https://scholar.google.com/scholar?q=Jonathan+Mellon%2C+Jack+Bailey%2C+Ralph+Scott%2C+James+Breckwoldt%2C+and+Marta+Miori%2E+2022%2E+Does+gpt%2D3+know+what+the+most+important+issue+is%3F+using+large+language+models+to+code+open%2Dtext+social+survey+respon)
29. Stefan Müller. 2022. The temporal focus of campaign communication. The Journal of Politics, 84(1):585– 590. [find](https://scholar.google.com/scholar?q=Stefan+M%C3%BCller%2E+2022%2E+The+temporal+focus+of+campaign+communication%2E+The+Journal+of+Politics%2C+84%281%29%3A585%E2%80%93+590%2E)
30. Kimberly A. Neuendorf. 2016. The Content Analysis Guidebook. Sage Publications. [find](https://scholar.google.com/scholar?q=Kimberly+A%2E+Neuendorf%2E+2016%2E+The+Content+Analysis+Guidebook%2E+Sage+Publications%2E)
31. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
32. Alexander Pan, Jun Shern Chan, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside, Hanlin Zhang, Scott Emmons, and Dan Hendrycks. 2023. Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the machiavelli benchmark. Proceedings of the 40th International… [find](https://scholar.google.com/scholar?q=Alexander+Pan%2C+Jun+Shern+Chan%2C+Andy+Zou%2C+Nathaniel+Li%2C+Steven+Basart%2C+Thomas+Woodside%2C+Hanlin+Zhang%2C+Scott+Emmons%2C+and+Dan+Hendrycks%2E+2023%2E+Do+the+rewards+justify+the+means%3F+measuring+trade%2Doffs+betwe)
33. Nicholas Pangakis, Samuel Wolken, and Neil Fasching. 2023. [Automated annotation with generative ai requires validation](https://md.research.elamur.ai/free/en/papers/pangakis-2023-automated-annotation-generative-ai-requires/).
34. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, and Trevor Killeen et al. 2019. [Pytorch: An imperative style, high-performance deep learning library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/). Advances in neural information processing systems.
35. Hao Peng, Daniel M. Romero, and Emoke-Agnes Horvat. 2022. Dynamics of cross-platform attention to retracted papers. Proceedings of the National Academy of Sciences, 119(25):585–590. [find](https://scholar.google.com/scholar?q=Hao+Peng%2C+Daniel+M%2E+Romero%2C+and+Emoke%2DAgnes+Horvat%2E+2022%2E+Dynamics+of+cross%2Dplatform+attention+to+retracted+papers%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+119%2825%29%3A585%E2%80%93590%2E)
36. Michael Reiss. 2023. Testing the reliability of chatgpt for text annotation and classification: A cautionary remark. Working paper. [find](https://scholar.google.com/scholar?q=Michael+Reiss%2E+2023%2E+Testing+the+reliability+of+chatgpt+for+text+annotation+and+classification%3A+A+cautionary+remark%2E+Working+paper%2E)
37. Christopher Michael Rytting, Taylor Sorensen, Lisa Argyle, Ethan Busby, Nancy Fulda, Joshua Gubler, and David Wingate. 2023. Towards coding social science datasets with language models. [find](https://scholar.google.com/scholar?q=Christopher+Michael+Rytting%2C+Taylor+Sorensen%2C+Lisa+Argyle%2C+Ethan+Busby%2C+Nancy+Fulda%2C+Joshua+Gubler%2C+and+David+Wingate%2E+2023%2E+Towards+coding+social+science+datasets+with+language+models%2E)
38. Punyajoy Saha, Narla, Komal Kalyan, and Animesh Mukherjee. 2023. On the rise of fear speech in online social media. Proceedings of the National Academy of Sciences of the United States of America. [find](https://scholar.google.com/scholar?q=Punyajoy+Saha%2C+Narla%2C+Komal+Kalyan%2C+and+Animesh+Mukherjee%2E+2023%2E+On+the+rise+of+fear+speech+in+online+social+media%2E+Proceedings+of+the+National+Academy+of+Sciences+of+the+United+States+of+America%2E)
39. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Lysandre+Debut%2C+Julien+Chaumond%2C+and+Thomas+Wolf%2E+2019%2E+Distilbert%2C+a+distilled+version+of+bert%3A+smaller%2C+faster%2C+cheaper+and+lighter%2E)
40. Maximilian Schreiner. 2023. Gpt-4 architecture, datasets, costs and more leaked. Blog post. [find](https://scholar.google.com/scholar?q=Maximilian+Schreiner%2E+2023%2E+Gpt%2D4+architecture%2C+datasets%2C+costs+and+more+leaked%2E+Blog+post%2E)
41. Saurabh Srivastava, Annarose M B, Anto P V au2, Shashank Menon, Ajay Sukumar, Adwaith Samod T, Alan Philipose, Stevin Prince, and Sooraj Thomas. 2024. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap. [find](https://scholar.google.com/scholar?q=Saurabh+Srivastava%2C+Annarose+M+B%2C+Anto+P+V+au2%2C+Shashank+Menon%2C+Ajay+Sukumar%2C+Adwaith+Samod+T%2C+Alan+Philipose%2C+Stevin+Prince%2C+and+Sooraj+Thomas%2E+2024%2E+Functional+benchmarks+for+robust+evaluation+of+re)
42. Zhanna Terechshenko, Fridolin Linder, Vishakh Padmakumar, Michael Liu, Jonathan Nagler, Joshua A Tucker, and Richard Bonneau. 2020. A comparison of methods in political science text classification: Transfer learning language models for politics. Working Paper. [find](https://scholar.google.com/scholar?q=Zhanna+Terechshenko%2C+Fridolin+Linder%2C+Vishakh+Padmakumar%2C+Michael+Liu%2C+Jonathan+Nagler%2C+Joshua+A+Tucker%2C+and+Richard+Bonneau%2E+2020%2E+A+comparison+of+methods+in+political+science+text+classification%3A+Tr)
43. Surendrabikram Thapa, Usman Naseem, and Mehwish Nasim. 2023. From humans to machines: can chatgpt-like llms effectively replace human annotators in nlp tasks. Workshop Proceedings of the 17th International AAAI Conference on Web and Social Media. [find](https://scholar.google.com/scholar?q=Surendrabikram+Thapa%2C+Usman+Naseem%2C+and+Mehwish+Nasim%2E+2023%2E+From+humans+to+machines%3A+can+chatgpt%2Dlike+llms+effectively+replace+human+annotators+in+nlp+tasks%2E+Workshop+Proceedings+of+the+17th+Internat)
44. Petter Törnberg. 2023. [Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning](https://md.research.elamur.ai/free/en/papers/tornberg-2023-chatgpt-4-outperforms-experts-crowd/).
45. Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. 2023. [Artificial artificial artificial intelligence: Crowd workers widely use large language models for text production tasks](https://md.research.elamur.ai/free/en/papers/veselovsky-2023-artificial-artificial-artificial-intelligence-crowd/).
46. Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. [Want to reduce labeling cost? gpt-3 can help](https://md.research.elamur.ai/free/en/papers/wang-2021-want-reduce-labeling-cost-gpt/).
47. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison and´ Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
48. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2020. [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/).
49. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. [Pegasus: Pre-training with extracted gap-sentences for abstractive summarization](https://md.research.elamur.ai/free/en/papers/zhang-2019-pegasus-pre-training-extracted-gap/). International Conference on Machine Learning, page 11328–11339.
50. Yiming Zhu, Peixian Zhang, Ehsan-Ul Haq, Pan Hui, and Gareth Tyson. 2023. Can chatgpt reproduce human-generated labels? a study of social computing tasks. [find](https://scholar.google.com/scholar?q=Yiming+Zhu%2C+Peixian+Zhang%2C+Ehsan%2DUl+Haq%2C+Pan+Hui%2C+and+Gareth+Tyson%2E+2023%2E+Can+chatgpt+reproduce+human%2Dgenerated+labels%3F+a+study+of+social+computing+tasks%2E)
51. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023. [Can large language models transform computational social science?](https://md.research.elamur.ai/free/en/papers/ziems-2023-large-language-models-transform-computational/) Working paper.

*Updated 30 October 2026*