---
title: "CoAnnotating — paper by Li et al. (2023) — summary and ideas"
description: "The proposed framework, CoAnnotating, estimates the model's reliability for each individual instance… 56 references, cited by 52 papers here."
authors: ["Minzhi Li", "Taiwei Shi", "Caleb Ziems", "Min-Yen Kan", "Nancy Chen", "Zhengyuan Liu", "Diyi Yang"]
year: 2023
venue: "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2023.emnlp-main.92/"
doi: "10.18653/v1/2023.emnlp-main.92"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2023-coannotating-uncertainty-guided-work-allocation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2023-coannotating-uncertainty-guided-work-allocation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation

# CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation

**Authors:** Minzhi Li, Taiwei Shi, [Caleb Ziems](https://md.research.elamur.ai/free/en/authors/caleb-ziems/), [Min-Yen Kan](https://md.research.elamur.ai/free/en/authors/min-yen-kan/), [Nancy Chen](https://md.research.elamur.ai/free/en/authors/nancy-chen/), [Zhengyuan Liu](https://md.research.elamur.ai/free/en/authors/zhengyuan-liu/) and [Diyi Yang](https://md.research.elamur.ai/free/en/authors/diyi-yang/) · **Year:** 2023 · **Venue:** [Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2023](https://md.research.elamur.ai/free/en/venues/emnlp/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.emnlp-main.92/) · [PDF](https://aclanthology.org/2023.emnlp-main.92.pdf) · [DOI 10.18653/v1/2023.emnlp-main.92](https://doi.org/10.18653/v1/2023.emnlp-main.92) · [OpenAlex](https://openalex.org/W4389523857)

**Citations:** 46 citations (OpenAlex, 2026-10-14) · [52](https://md.research.elamur.ai/free/en/papers/li-2023-coannotating-uncertainty-guided-work-allocation/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Large language models such as ChatGPT can label text at zero-shot, cheaply and at scale. Earlier studies usually treated them as replacements for human annotators. This paper asks a different question: within one dataset, which items should go to the model and which to people? The proposed framework, CoAnnotating, estimates the model's reliability for each individual instance rather than for a task as a whole. It queries ChatGPT several times per item using deliberately varied prompts. It then measures uncertainty in two ways: the model's self-reported confidence, and the entropy of its answers. Items with low uncertainty are labeled by the LLM, and the uncertain ones are sent to humans. Pareto analysis on a pilot subset shows the trade-off between cost and quality, so practitioners can pick an allocation that fits their budget. On several English classification datasets, uncertainty-guided allocation clearly beat random assignment. For simpler tasks, part of the labeling could be handed to the LLM at no loss in quality. This matters because it offers a principled recipe for cutting annotation costs while keeping data quality.

## Key points
- Each instance gets 7 prompt variants: the basic instruction, sentence-order swap, paraphrased instruction, True/False, short textual answer, multiple choice, and a confirmation-bias question. These are used to compute entropy and self-evaluation uncertainty.
- Uncertainty-guided allocation improved performance by as much as 21% over a random allocation baseline.
- On AG News (topic classification), about 33% of the annotation could go to ChatGPT while still matching human-level quality. For Stance Detection and Conversation Gone Awry, any amount of outsourcing reduced quality.
- In an ablation, entropy computed from repeating one identical prompt was a much weaker signal, because most items showed zero entropy. Varied prompts consistently produced better allocation.
- Items with high entropy (>0.8) tended to involve sarcasm, incomplete sentences, numerical reasoning, or misleading keywords. The experiments cover only English classification tasks, and the code is published at github.com/SALT-NLP/CoAnnotating.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 310 works · 284,480 citations
- [University of Southern California](https://md.research.elamur.ai/free/en/universities/university-of-southern-california/) — University, Los Angeles, United States · 104 works · 33,876 citations
- [National University of Singapore](https://md.research.elamur.ai/free/en/universities/national-university-of-singapore/) — University, Singapore, Singapore · 84 works · 32,204 citations
- [Institute for Infocomm Research](https://md.research.elamur.ai/free/en/labs/institute-for-infocomm-research/) — Research institute, Singapore, Singapore · 33 works · 14,373 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 744 works · [🤝 General Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/general-social-sciences/) · 87 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works

## Cite

Cite as: Minzhi Li, Taiwei Shi, Caleb Ziems, Min-Yen Kan, Nancy Chen, Zhengyuan Liu and Diyi Yang. 2023. CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{li2023coannotating,
  title = {{CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation}},
  author = {Minzhi Li, Taiwei Shi, Caleb Ziems, Min-Yen Kan, Nancy Chen, Zhengyuan Liu and Diyi Yang},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.92},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.92}
}
```

- [← previous Inverse Scaling: When Bigger Isn't Better](https://md.research.elamur.ai/free/en/papers/mckenzie-2023-inverse-scaling-bigger-isn-t/)
- [next → MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning](https://md.research.elamur.ai/free/en/papers/wang-2023-mathcoder-seamless-code-integration-llms/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

56 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. AmirAli Abdolrashidi, Lisa Wang, Shivani Agrawal, Jonathan Malmaud, Oleg Rybakov, Chas Leichner, and Lukasz Lew. 2021. Pareto-optimal quantized resnet is mostly 4-bit. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 3091– 3099. [find](https://scholar.google.com/scholar?q=AmirAli+Abdolrashidi%2C+Lisa+Wang%2C+Shivani+Agrawal%2C+Jonathan+Malmaud%2C+Oleg+Rybakov%2C+Chas+Leichner%2C+and+Lukasz+Lew%2E+2021%2E+Pareto%2Doptimal+quantized+resnet+is+mostly+4%2Dbit%2E+In+Proceedings+of+the+IEEE%2FCVF+C)
2. Mostafa M Amin, Erik Cambria, and Björn W Schuller. 2023. Will affective computing emerge from foundation models and general ai? a first evaluation on chatgpt. arXiv preprint arXiv:2303.03186. — [arXiv:2303.03186](https://arxiv.org/abs/2303.03186) [find](https://scholar.google.com/scholar?q=Mostafa+M+Amin%2C+Erik+Cambria%2C+and+Bj%C3%B6rn+W+Schuller%2E+2023%2E+Will+affective+computing+emerge+from+foundation+models+and+general+ai%3F+a+first+evaluation+on+chatgpt%2E+arXiv+preprint+arXiv%3A2303%2E03186%2E+%E2%80%94+%5BarXi)
3. Rie Kubota Ando and Tong Zhang. 2005. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6:1817–1853. [find](https://scholar.google.com/scholar?q=Rie+Kubota+Ando+and+Tong+Zhang%2E+2005%2E+A+framework+for+learning+predictive+structures+from+multiple+tasks+and+unlabeled+data%2E+Journal+of+Machine+Learning+Research%2C+6%3A1817%E2%80%931853%2E)
4. Galen Andrew and Jianfeng Gao. 2007. Scalable training of L 1 -regularized log-linear models. In Proceedings of the 24th International Conference on Machine Learning, pages 33–40. [find](https://scholar.google.com/scholar?q=Galen+Andrew+and+Jianfeng+Gao%2E+2007%2E+Scalable+training+of+L+1+%2Dregularized+log%2Dlinear+models%2E+In+Proceedings+of+the+24th+International+Conference+on+Machine+Learning%2C+pages+33%E2%80%9340%2E)
5. Abhijeet Awasthi, Sabyasachi Ghosh, Rasna Goyal, and Sunita Sarawagi. 2020. Learning from rules generalizing labeled exemplars. arXiv preprint arXiv:2004.06025. — [arXiv:2004.06025](https://arxiv.org/abs/2004.06025) [find](https://scholar.google.com/scholar?q=Abhijeet+Awasthi%2C+Sabyasachi+Ghosh%2C+Rasna+Goyal%2C+and+Sunita+Sarawagi%2E+2020%2E+Learning+from+rules+generalizing+labeled+exemplars%2E+arXiv+preprint+arXiv%3A2004%2E06025%2E+%E2%80%94+%5BarXiv%3A2004%2E06025%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
6. Stephen H Bach, Daniel Rodriguez, Yintao Liu, Chong Luo, Haidong Shao, Cassandra Xia, Souvik Sen, Alex Ratner, Braden Hancock, Houman Alborzi, et al. 2019. Snorkel drybell: A case study in deploying weak supervision at industrial scale. In Proceedings of the 2019 International Conference on Manageme… [find](https://scholar.google.com/scholar?q=Stephen+H+Bach%2C+Daniel+Rodriguez%2C+Yintao+Liu%2C+Chong+Luo%2C+Haidong+Shao%2C+Cassandra+Xia%2C+Souvik+Sen%2C+Alex+Ratner%2C+Braden+Hancock%2C+Houman+Alborzi%2C+et+al%2E+2019%2E+Snorkel+drybell%3A+A+case+study+in+deploying+w)
7. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. [A multi-task, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity](https://md.research.elamur.ai/free/en/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/). arXiv preprint arXiv:2302.04023.
8. Bill Dolan and Chris Brockett. 2005. [Automatically constructing a corpus of sentential paraphrases](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/). In Third International Workshop on Paraphrasing (IWP2005).
9. Claudio Gentile, Zhilei Wang, and Tong Zhang. 2022. Fast rates in pool-based batch active learning. arXiv preprint arXiv:2202.05448. — [arXiv:2202.05448](https://arxiv.org/abs/2202.05448) [find](https://scholar.google.com/scholar?q=Claudio+Gentile%2C+Zhilei+Wang%2C+and+Tong+Zhang%2E+2022%2E+Fast+rates+in+pool%2Dbased+batch+active+learning%2E+arXiv+preprint+arXiv%3A2202%2E05448%2E+%E2%80%94+%5BarXiv%3A2202%2E05448%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2202%2E05448%29)
10. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. [Chatgpt outperforms crowd-workers for text-annotation tasks](https://md.research.elamur.ai/free/en/papers/gilardi-2023-chatgpt-outperforms-crowd-workers-text/). arXiv preprint arXiv:2303.15056.
11. Julius Gonsior, Maik Thiele, and Wolfgang Lehner. 2020. Weakal: Combining active learning and weak supervision. In Discovery Science: 23rd International Conference, DS 2020, Thessaloniki, Greece, October 19–21, 2020, Proceedings 23, pages 34–49. Springer. [find](https://scholar.google.com/scholar?q=Julius+Gonsior%2C+Maik+Thiele%2C+and+Wolfgang+Lehner%2E+2020%2E+Weakal%3A+Combining+active+learning+and+weak+supervision%2E+In+Discovery+Science%3A+23rd+International+Conference%2C+DS+2020%2C+Thessaloniki%2C+Greece%2C+Octo)
12. Jonatas S Grosman, Pedro HT Furtado, Ariane MB Rodrigues, Guilherme G Schardong, Simone DJ Barbosa, and Hélio CV Lopes. 2020. Eras: Improving the quality control in the annotation process for natural language processing tasks. Information Systems, 93:101553. [find](https://scholar.google.com/scholar?q=Jonatas+S+Grosman%2C+Pedro+HT+Furtado%2C+Ariane+MB+Rodrigues%2C+Guilherme+G+Schardong%2C+Simone+DJ+Barbosa%2C+and+H%C3%A9lio+CV+Lopes%2E+2020%2E+Eras%3A+Improving+the+quality+control+in+the+annotation+process+for+natural+)
13. Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S Weld. 2011. Knowledge-based weak supervision for information extraction of overlapping relations. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 5… [find](https://scholar.google.com/scholar?q=Raphael+Hoffmann%2C+Congle+Zhang%2C+Xiao+Ling%2C+Luke+Zettlemoyer%2C+and+Daniel+S+Weld%2E+2011%2E+Knowledge%2Dbased+weak+supervision+for+information+extraction+of+overlapping+relations%2E+In+Proceedings+of+the+49th+a)
14. Fan Huang, Haewoon Kwak, and Jisun An. 2023. Is chatgpt better than human annotators? potential and limitations of chatgpt in explaining implicit hate speech. arXiv preprint arXiv:2302.07736. [find](https://scholar.google.com/scholar?q=Fan+Huang%2C+Haewoon+Kwak%2C+and+Jisun+An%2E+2023%2E+Is+chatgpt+better+than+human+annotators%3F+potential+and+limitations+of+chatgpt+in+explaining+implicit+hate+speech%2E+arXiv+preprint+arXiv%3A2302%2E07736%2E)
15. Myeongjun Jang and Thomas Lukasiewicz. 2023. Consistency analysis of chatgpt. arXiv preprint arXiv:2303.06273. [find](https://scholar.google.com/scholar?q=Myeongjun+Jang+and+Thomas+Lukasiewicz%2E+2023%2E+Consistency+analysis+of+chatgpt%2E+arXiv+preprint+arXiv%3A2303%2E06273%2E)
16. Max Bartolo, Tristan Thrush, Sebastian Riedel, Pontus Stenetorp, Robin Jia, and Douwe Kiela. 2021. Models in the loop: Aiding crowdworkers with generative annotation assistants. arXiv preprint arXiv:2112.09062. [find](https://scholar.google.com/scholar?q=Max+Bartolo%2C+Tristan+Thrush%2C+Sebastian+Riedel%2C+Pontus+Stenetorp%2C+Robin+Jia%2C+and+Douwe+Kiela%2E+2021%2E+Models+in+the+loop%3A+Aiding+crowdworkers+with+generative+annotation+assistants%2E+arXiv+preprint+arXiv%3A2)
17. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. [find](https://scholar.google.com/scholar?q=Saurav+Kadavath%2C+Tom+Conerly%2C+Amanda+Askell%2C+Tom+Henighan%2C+Dawn+Drain%2C+Ethan+Perez%2C+Nicholas+Schiefer%2C+Zac+Hatfield+Dodds%2C+Nova+DasSarma%2C+Eli+Tran%2DJohnson%2C+et+al%2E+2022%2E+Language+models+%28mostly%29+know+w)
18. Arun Tejasvi Chaganty, Stephen Mussman, and Percy Liang. 2018. The price of debiasing automatic metrics in natural language evaluation. arXiv preprint arXiv:1807.02202. — [arXiv:1807.02202](https://arxiv.org/abs/1807.02202) [find](https://scholar.google.com/scholar?q=Arun+Tejasvi+Chaganty%2C+Stephen+Mussman%2C+and+Percy+Liang%2E+2018%2E+The+price+of+debiasing+automatic+metrics+in+natural+language+evaluation%2E+arXiv+preprint+arXiv%3A1807%2E02202%2E+%E2%80%94+%5BarXiv%3A1807%2E02202%5D%28https%3A%2F%2Far)
19. Junmo Kang, Wei Xu, and Alan Ritter. 2023. Distill or annotate? cost-efficient fine-tuning of compact models. arXiv preprint arXiv:2305.01645. [find](https://scholar.google.com/scholar?q=Junmo+Kang%2C+Wei+Xu%2C+and+Alan+Ritter%2E+2023%2E+Distill+or+annotate%3F+cost%2Defficient+fine%2Dtuning+of+compact+models%2E+arXiv+preprint+arXiv%3A2305%2E01645%2E)
20. Shizhe Diao, Pengcheng Wang, Yong Lin, and Tong Zhang. 2023. Active prompting with chain-of-thought for large language models. arXiv preprint arXiv:2302.12246. [find](https://scholar.google.com/scholar?q=Shizhe+Diao%2C+Pengcheng+Wang%2C+Yong+Lin%2C+and+Tong+Zhang%2E+2023%2E+Active+prompting+with+chain%2Dof%2Dthought+for+large+language+models%2E+arXiv+preprint+arXiv%3A2302%2E12246%2E)
21. Aniket Kittur, Ed H Chi, and Bongwon Suh. 2008. Crowdsourcing user studies with mechanical turk. In Proceedings of the SIGCHI conference on human factors in computing systems, pages 453–456. [find](https://scholar.google.com/scholar?q=Aniket+Kittur%2C+Ed+H+Chi%2C+and+Bongwon+Suh%2E+2008%2E+Crowdsourcing+user+studies+with+mechanical+turk%2E+In+Proceedings+of+the+SIGCHI+conference+on+human+factors+in+computing+systems%2C+pages+453%E2%80%93456%2E)
22. Bosheng Ding, Chengwei Qin, Linlin Liu, Lidong Bing, Shafiq Joty, and Boyang Li. 2022. [Is gpt-3 a good data annotator?](https://md.research.elamur.ai/free/en/papers/ding-2022-gpt-3-good-data-annotator/) arXiv preprint arXiv:2212.10450.
23. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. arXiv preprint arXiv:2302.14520. [find](https://scholar.google.com/scholar?q=Tom+Kocmi+and+Christian+Federmann%2E+2023%2E+Large+language+models+are+state%2Dof%2Dthe%2Dart+evaluators+of+translation+quality%2E+arXiv+preprint+arXiv%3A2302%2E14520%2E)
24. Jan Kocoń, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, et al. 2023. [Chatgpt: Jack of all trades, master of none](https://md.research.elamur.ai/free/en/papers/kocon-2023-chatgpt-jack-all-trades-master/). arXiv preprint arXiv:2302.10724.
25. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016a. [Semeval-2016 task 6: Detecting stance in tweets](https://md.research.elamur.ai/free/en/papers/mohammad-2016-semeval-2016-task-6-detecting/). In Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016), pages 31–41.
26. Taja Kuzman, Nikola Ljubešić, and Igor Mozetič. 2023. Chatgpt: Beginning of an end of manual annotation? use case of automatic genre identification. arXiv preprint arXiv:2303.03953. [find](https://scholar.google.com/scholar?q=Taja+Kuzman%2C+Nikola+Ljube%C5%A1ic%CC%81%2C+and+Igor+Mozetic%CC%8C%2E+2023%2E+Chatgpt%3A+Beginning+of+an+end+of+manual+annotation%3F+use+case+of+automatic+genre+identification%2E+arXiv+preprint+arXiv%3A2303%2E03953%2E)
27. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016b. [SemEval-2016 task 6: Detecting stance in tweets](https://md.research.elamur.ai/free/en/papers/mohammad-2016-semeval-2016-task-6-detecting/). In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 31– 41, San Diego, California. Association for Computational Lin…
28. Xin Li and Dan Roth. 2002. [Learning question classifiers](https://md.research.elamur.ai/free/en/papers/li-2002-learning-question-classifiers/). In COLING 2002: The 19th International Conference on Computational Linguistics.
29. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334. [find](https://scholar.google.com/scholar?q=Stephanie+Lin%2C+Jacob+Hilton%2C+and+Owain+Evans%2E+2022%2E+Teaching+models+to+express+their+uncertainty+in+words%2E+arXiv+preprint+arXiv%3A2205%2E14334%2E)
30. Todd Lingren, Louise Deleger, Katalin Molnar, Haijun Zhai, Jareen Meinzen-Derr, Megan Kaiser, Laura Stoutenborough, Qi Li, and Imre Solti. 2014. Evaluating the impact of pre-annotation on annotation speed and potential bias: natural language processing gold standard development for clinical named en… [find](https://scholar.google.com/scholar?q=Todd+Lingren%2C+Louise+Deleger%2C+Katalin+Molnar%2C+Haijun+Zhai%2C+Jareen+Meinzen%2DDerr%2C+Megan+Kaiser%2C+Laura+Stoutenborough%2C+Qi+Li%2C+and+Imre+Solti%2E+2014%2E+Evaluating+the+impact+of+pre%2Dannotation+on+annotation+s)
31. Pierre Lison, Aliaksandr Hubin, Jeremy Barnes, and Samia Touileb. 2020. Named entity recognition without labelled data: A weak supervision approach. arXiv preprint arXiv:2004.14723. [find](https://scholar.google.com/scholar?q=Pierre+Lison%2C+Aliaksandr+Hubin%2C+Jeremy+Barnes%2C+and+Samia+Touileb%2E+2020%2E+Named+entity+recognition+without+labelled+data%3A+A+weak+supervision+approach%2E+arXiv+preprint+arXiv%3A2004%2E14723%2E)
32. Alisa Liu, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. 2022. [Wanli: Worker and ai collaboration for natural language inference dataset creation](https://md.research.elamur.ai/free/en/papers/liu-2022-wanli-worker-ai-collaboration-natural/). arXiv preprint arXiv:2201.05955.
33. Daniel Loureiro, Aminette D’Souza, Areej Nasser Muhajab, Isabella A White, Gabriel Wong, Luis Espinosa Anke, Leonardo Neves, Francesco Barbieri, and Jose Camacho-Collados. 2022. Tempowic: An evaluation benchmark for detecting meaning shift in social media. arXiv preprint arXiv:2209.07216. [find](https://scholar.google.com/scholar?q=Daniel+Loureiro%2C+Aminette+D%E2%80%99Souza%2C+Areej+Nasser+Muhajab%2C+Isabella+A+White%2C+Gabriel+Wong%2C+Luis+Espinosa+Anke%2C+Leonardo+Neves%2C+Francesco+Barbieri%2C+and+Jose+Camacho%2DCollados%2E+2022%2E+Tempowic%3A+An+evaluatio)
34. Gideon S Mann and Andrew McCallum. 2010. Generalized expectation criteria for semi-supervised learning with weakly labeled data. Journal of machine learning research, 11(2). [find](https://scholar.google.com/scholar?q=Gideon+S+Mann+and+Andrew+McCallum%2E+2010%2E+Generalized+expectation+criteria+for+semi%2Dsupervised+learning+with+weakly+labeled+data%2E+Journal+of+machine+learning+research%2C+11%282%29%2E)
35. Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. 2018. Weakly-supervised neural text classification. In proceedings of the 27th ACM International Conference on information and knowledge management, pages 983–992. [find](https://scholar.google.com/scholar?q=Yu+Meng%2C+Jiaming+Shen%2C+Chao+Zhang%2C+and+Jiawei+Han%2E+2018%2E+Weakly%2Dsupervised+neural+text+classification%2E+In+proceedings+of+the+27th+ACM+International+Conference+on+information+and+knowledge+management%2C+)
36. Bonan Min, Ralph Grishman, Li Wan, Chang Wang, and David Gondek. 2013. [Distant supervision for relation extraction with an incomplete knowledge base](https://md.research.elamur.ai/free/en/papers/min-2013-distant-supervision-relation-extraction-incomplete/). In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pag…
37. Matúš Pikuliak. 2023. Chatgpt survey: Performance on nlp datasets. [https://www.opensamizdat.com/posts/chatgpt_survey](https://www.opensamizdat.com/posts/chatgpt_survey). [↗](https://www.opensamizdat.com/posts/chatgpt_survey)
38. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. [Is chatgpt a general-purpose natural language processing task solver?](https://md.research.elamur.ai/free/en/papers/qin-2023-chatgpt-general-purpose-natural-language/) arXiv preprint arXiv:2302.06476.
39. Mohammad Sadegh Rasooli and Joel R. Tetreault. 2015. Yara parser: A fast and accurate dependency parser. Computing Research Repository, arXiv:1503.06733. Version 2. — [arXiv:1503.06733](https://arxiv.org/abs/1503.06733) [find](https://scholar.google.com/scholar?q=Mohammad+Sadegh+Rasooli+and+Joel+R%2E+Tetreault%2E+2015%2E+Yara+parser%3A+A+fast+and+accurate+dependency+parser%2E+Computing+Research+Repository%2C+arXiv%3A1503%2E06733%2E+Version+2%2E+%E2%80%94+%5BarXiv%3A1503%2E06733%5D%28https%3A%2F%2Farxiv%2E)
40. Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. 2017. Snorkel: Rapid training data creation with weak supervision. In Proceedings of the VLDB Endowment. International Conference on Very Large Data Bases, volume 11, page 269. NIH Public Access. [find](https://scholar.google.com/scholar?q=Alexander+Ratner%2C+Stephen+H+Bach%2C+Henry+Ehrenberg%2C+Jason+Fries%2C+Sen+Wu%2C+and+Christopher+R%C3%A9%2E+2017%2E+Snorkel%3A+Rapid+training+data+creation+with+weak+supervision%2E+In+Proceedings+of+the+VLDB+Endowment%2E+Int)
41. Michael Reiss. 2023. Testing the reliability of chatgpt for text annotation and classification: A cautionary remark. [find](https://scholar.google.com/scholar?q=Michael+Reiss%2E+2023%2E+Testing+the+reliability+of+chatgpt+for+text+annotation+and+classification%3A+A+cautionary+remark%2E)
42. Rion Snow, Brendan O’connor, Dan Jurafsky, and Andrew Y Ng. 2008. [Cheap and fast–but is it good?](https://md.research.elamur.ai/free/en/papers/snow-2008-cheap-fast-but-is-it/) evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 conference on empirical methods in natural language processing, pages 254–263.
43. Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen, and Guilin Qi. 2023. Evaluation of chatgpt as a question answering system for answering complex questions. arXiv preprint arXiv:2303.07992. — [arXiv:2303.07992](https://arxiv.org/abs/2303.07992) [find](https://scholar.google.com/scholar?q=Yiming+Tan%2C+Dehai+Min%2C+Yu+Li%2C+Wenbo+Li%2C+Nan+Hu%2C+Yongrui+Chen%2C+and+Guilin+Qi%2E+2023%2E+Evaluation+of+chatgpt+as+a+question+answering+system+for+answering+complex+questions%2E+arXiv+preprint+arXiv%3A2303%2E07992)
44. Serra Sinem Tekiroglu, Yi-Ling Chung, and Marco Guerini. 2020. [Generating counter narratives against online hate speech: Data and strategies](https://md.research.elamur.ai/free/en/papers/tekiroglu-2020-generating-counter-narratives-against-online/). arXiv preprint arXiv:2004.04216.
45. Petter Törnberg. 2023. [Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning](https://md.research.elamur.ai/free/en/papers/tornberg-2023-chatgpt-4-outperforms-experts-crowd/). arXiv preprint arXiv:2304.06588.
46. Marcos Treviso, António Góis, Patrick Fernandes, Erick Fonseca, and Andre Martins. 2022. Predicting attention sparsity in transformers. In Proceedings of the Sixth Workshop on Structured Prediction for NLP, pages 67–81, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Marcos+Treviso%2C+Ant%C3%B3nio+G%C3%B3is%2C+Patrick+Fernandes%2C+Erick+Fonseca%2C+and+Andre+Martins%2E+2022%2E+Predicting+attention+sparsity+in+transformers%2E+In+Proceedings+of+the+Sixth+Workshop+on+Structured+Prediction+fo)
47. Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. [Is chatgpt a good nlg evaluator? a preliminary study](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-nlg-evaluator-preliminary/). arXiv preprint arXiv:2303.04048.
48. Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. [Want to reduce labeling cost? gpt-3 can help](https://md.research.elamur.ai/free/en/papers/wang-2021-want-reduce-labeling-cost-gpt/). arXiv preprint arXiv:2108.13487.
49. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2022. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). arXiv preprint arXiv:2203.11171.
50. Ann Yuan, Daphne Ippolito, Vitaly Nikolaev, Chris Callison-Burch, Andy Coenen, and Sebastian Gehrmann. 2021. Synthbio: A case study in human-ai collaborative curation of text datasets. arXiv preprint arXiv:2111.06467. — [arXiv:2111.06467](https://arxiv.org/abs/2111.06467) [find](https://scholar.google.com/scholar?q=Ann+Yuan%2C+Daphne+Ippolito%2C+Vitaly+Nikolaev%2C+Chris+Callison%2DBurch%2C+Andy+Coenen%2C+and+Sebastian+Gehrmann%2E+2021%2E+Synthbio%3A+A+case+study+in+human%2Dai+collaborative+curation+of+text+datasets%2E+arXiv+preprint+)
51. Jieyu Zhang, Cheng-Yu Hsieh, Yue Yu, Chao Zhang, and Alexander Ratner. 2022. A survey on programmatic weak supervision. arXiv preprint arXiv:2202.05433. — [arXiv:2202.05433](https://arxiv.org/abs/2202.05433) [find](https://scholar.google.com/scholar?q=Jieyu+Zhang%2C+Cheng%2DYu+Hsieh%2C+Yue+Yu%2C+Chao+Zhang%2C+and+Alexander+Ratner%2E+2022%2E+A+survey+on+programmatic+weak+supervision%2E+arXiv+preprint+arXiv%3A2202%2E05433%2E+%E2%80%94+%5BarXiv%3A2202%2E05433%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2202)
52. Jieyu Zhang, Bohan Wang, Xiangchen Song, Yujing Wang, Yaming Yang, Jing Bai, and Alexander Ratner. 2021. Creating training sets via weak indirect supervision. arXiv preprint arXiv:2110.03484. — [arXiv:2110.03484](https://arxiv.org/abs/2110.03484) [find](https://scholar.google.com/scholar?q=Jieyu+Zhang%2C+Bohan+Wang%2C+Xiangchen+Song%2C+Yujing+Wang%2C+Yaming+Yang%2C+Jing+Bai%2C+and+Alexander+Ratner%2E+2021%2E+Creating+training+sets+via+weak+indirect+supervision%2E+arXiv+preprint+arXiv%3A2110%2E03484%2E+%E2%80%94+%5BarXiv)
53. Justine Zhang, Jonathan P Chang, Cristian Danescu-Niculescu-Mizil, Lucas Dixon, Yiqing Hua, Nithum Thain, and Dario Taraborelli. 2018. [Conversations gone awry: Detecting early signs of conversational failure](https://md.research.elamur.ai/free/en/papers/zhang-2018-conversations-gone-awry-detecting-early/). arXiv preprint arXiv:1805.05345.
54. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. [Character-level convolutional networks for text classification](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/). Advances in neural information processing systems, 28.
55. Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023. Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert. arXiv preprint arXiv:2302.10198. [find](https://scholar.google.com/scholar?q=Qihuang+Zhong%2C+Liang+Ding%2C+Juhua+Liu%2C+Bo+Du%2C+and+Dacheng+Tao%2E+2023%2E+Can+chatgpt+understand+too%3F+a+comparative+study+on+chatgpt+and+fine%2Dtuned+bert%2E+arXiv+preprint+arXiv%3A2302%2E10198%2E)
56. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023. [Can large language models transform computational social science?](https://md.research.elamur.ai/free/en/papers/ziems-2023-large-language-models-transform-computational/) arXiv preprint arXiv:2305.03514.

*Updated 24 October 2026*