---
title: "LLMs as Annotators — Zhao 2026"
description: "Cultural heritage institutions that digitize old collections need to flag wording that is offensive by today's standards."
authors: ["Yahui Zhao", "Clemencia Siro", "Laura Hollink"]
year: 2026
venue: "Proceedings of the Fifteenth Language Resources and Evaluation Conference"
source: "https://aclanthology.org/2026.lrec-1.852/"
doi: "10.63317/3dhy55mxo9zb"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/zhao-2026-llms-as-annotators-evaluating-model/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/zhao-2026-llms-as-annotators-evaluating-model/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2026](https://md.research.elamur.ai/free/papers/2026/page/3/) / LLMs as Annotators: Evaluating Model–Human Alignment in Detecting Contentious Language in Historical Corpora

# LLMs as Annotators: Evaluating Model–Human Alignment in Detecting Contentious Language in Historical Corpora

**Authors:** Yahui Zhao, Clemencia Siro, Laura Hollink · **Year:** 2026 · **Venue:** Proceedings of the Fifteenth Language Resources and Evaluation Conference · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.lrec-1.852/) · [PDF](https://aclanthology.org/2026.lrec-1.852.pdf) · [DOI 10.63317/3dhy55mxo9zb](https://doi.org/10.63317/3dhy55mxo9zb) · [OpenAlex](https://openalex.org/W7160065811)

**Citations:** 0 citations (openalex-snapshot · 2026-10-14)

## Summary
Cultural heritage institutions that digitize old collections need to flag wording that is offensive by today's standards. Doing this by hand takes expert knowledge and does not scale. The authors test whether LLMs could act as additional annotators for this work, using historical Dutch newspaper text. They treat the problem as zero-shot binary classification: a model sees a target term inside its surrounding excerpt and decides whether that use is contentious. GPT-4o and LLaMA-3-70B-Instruct were run on a filtered subset of ConConCor, with prompts that differ in language, wording and whether the model must explain its answer. Model outputs were compared with expert and crowd labels using Krippendorff's α, accuracy against majority labels and label-distribution similarity, followed by a qualitative look at where models and humans disagree. On clear-cut cases, the models agree with humans about as well as humans agree with each other. They falter when a judgment depends on historical or contextual nuance. The authors conclude that LLMs can help with this task only if people keep oversight of the results.

## Key points
- **Data:** ConConCor (Brate et al., 2021) draws on Europeana/Delpher Dutch newspapers from 1890–1941. Filtering reduced it from 2,715 to 2,090 excerpts, labelled by 19 expert and 329 crowdworker IDs. The excerpts cover 88 unigram target terms taken from Words Matter, and 1,254 of them have unanimous human labels.
- **Human baseline:** inter-annotator agreement is α=0.578 among experts and α=0.510 among crowdworkers.
- **Prompts:** there are eight templates, each a combination of three choices:
  - original guideline wording or a simplified question;
  - Dutch or English;
  - a label with a reason or the label alone.
- **Results:** the models reach α≈0.45–0.53 on explicit cases. They flag identity-related and colonial terms more often than humans do. They miss figurative uses and terms whose meaning has shifted over time.
- **Resource:** the code is released publicly on GitHub.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhao2026llms,
  title = {{LLMs as Annotators: Evaluating Model–Human Alignment in Detecting Contentious Language in Historical Corpora}},
  author = {Zhao, Yahui and Siro, Clemencia and Hollink, Laura},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3dhy55mxo9zb},
  url = {https://doi.org/10.63317/3dhy55mxo9zb}
}
```

- [← previous LLM-as-a-Jury for Machine Translation Publishability Assessment](https://md.research.elamur.ai/free/papers/yanishevsky-2026-llm-as-jury-machine-translation/)
- [next → Lost in Translation: Repurposing semantic similarity benchmarks for evaluating lexical-semantic consistency in LLM-based machine translation](https://md.research.elamur.ai/free/papers/ye-2026-lost-translation-repurposing-semantic-similarity/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/papers/2026/page/3/)

## References

28 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Persistent anti-muslim bias in large language models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’21, page 298–306, New York, NY, USA. Association for Computing Machinery. OpenAI Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haim ing Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Made laine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Benjamin Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Sim’on Posada Fishman, Juston Forte, Is abella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Raphael Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Ryan Kiros, Matthew Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Li, Rachel Lim, Molly Lin, Stephanie Lin, Ma teusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, An drey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel P. Mossing, Tong Mu, Mira Murati, Oleg Murk, David M’ely, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Ouyang Long, Cullen O’Keefe, Jakub W. Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alexandre Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Pondé de Oliveira Pinto, Michael Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack W. Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario D. Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas A. Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cer’on Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll L. Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qim ing Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2023. Gpt-4 technical report. [find](https://scholar.google.com/scholar?q=Abubakar+Abid%2C+Maheen+Farooqi%2C+and+James+Zou%2E+2021%2E+Persistent+anti%2Dmuslim+bias+in+large+language+models%2E+In+Proceedings+of+the+2021+AAAI%2FACM+Conference+on+AI%2C+Ethics%2C+and+Society%2C+AIES+%E2%80%9921%2C+page+298%E2%80%93) [our card](https://md.research.elamur.ai/free/papers/abid-2021-persistent-anti-muslim-bias-large/)
2. Csilla E. Ariese. 2020. Amplifying voices: Engaging and disengaging with colonial pasts in amsterdam. Heritage & Society, 13(1-2):117–142. [↗](https://doi.org/10.1080/2159032x.2021.1901335)
3. Lora Aroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1):15–24. [↗](https://doi.org/10.1609/aimag.v36i1.2564) [our card](https://md.research.elamur.ai/free/papers/aroyo-2015-truth-lie-crowd-truth-seven/)
4. David Bamman, Kent K. Chang, Li Lucy, and Naitian Zhou. 2024. On classification with large language models in cultural analytics. arXiv preprint arXiv:2410.12029. [↗](https://arxiv.org/abs/2410.12029)
5. Ryan Brate, Andrei Nesterov, Valentin Vogelmann, Jacco Van Ossenbruggen, Laura Hollink, and Marieke Van Erp. 2021. Capturing contentiousness: constructing the contentious terms in context corpus. In Proceedings of the 11th Knowledge Capture Conference, K-CAP ’21, page 17– 24, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Ryan+Brate%2C+Andrei+Nesterov%2C+Valentin+Vogelmann%2C+Jacco+Van+Ossenbruggen%2C+Laura+Hollink%2C+and+Marieke+Van+Erp%2E+2021%2E+Capturing+contentiousness%3A+constructing+the+contentious+terms+in+context+corpus%2E+In+P)
6. Naoki Egami, Musashi Hinck, Brandon M. Stewart, and Hanying Wei. 2023. Using large language model annotations for valid downstream statistical inference in social science: Design-based semi-supervised learning. arXiv preprint arXiv:2306.04746. [↗](https://arxiv.org/abs/2306.04746)
7. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. [↗](https://arxiv.org/abs/2407.21783)
8. Endre Hamerlik, Marek Šuppa, Miroslav Blšták, Jozef Kubík, Martin Takáč, Marián Šimko, and Andrej Findor. 2024. ChatGPT as your n-th annotator: Experiments in leveraging large language models for social science text annotation in Slovak language. In Proceedings of the 4th Workshop on Computational Linguistics for the Political and Social Sciences: Long and short papers, pages 81–89, Vienna, Austria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Endre+Hamerlik%2C+Marek+%C5%A0uppa%2C+Miroslav+Bl%C5%A1t%C3%A1k%2C+Jozef+Kub%C3%ADk%2C+Martin+Tak%C3%A1%C4%8D%2C+Mari%C3%A1n+%C5%A0imko%2C+and+Andrej+Findor%2E+2024%2E+ChatGPT+as+your+n%2Dth+annotator%3A+Experiments+in+leveraging+large+language+models+for+soci)
9. Sanne Hoeken, Sophie Spliethoff, Silke Schwandt, Sina Zarrieß, and Özge Alacam. 2023. Towards detecting lexical change of hate speech in historical data. In Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change, pages 100–111, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sanne+Hoeken%2C+Sophie+Spliethoff%2C+Silke+Schwandt%2C+Sina+Zarrie%C3%9F%2C+and+%C3%96zge+Alacam%2E+2023%2E+Towards+detecting+lexical+change+of+hate+speech+in+historical+data%2E+In+Proceedings+of+the+4th+Workshop+on+Computat)
10. Haoyang Huang, Tianyi Tang, Dongdong Zhang, Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. Not all languages are created equal in LLMs: Improving multilingual capability by cross-lingual-thought prompting. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12365–12394, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Haoyang+Huang%2C+Tianyi+Tang%2C+Dongdong+Zhang%2C+Xin+Zhao%2C+Ting+Song%2C+Yan+Xia%2C+and+Furu+Wei%2E+2023%2E+Not+all+languages+are+created+equal+in+LLMs%3A+Improving+multilingual+capability+by+cross%2Dlingual%2Dthought+pr) [our card](https://md.research.elamur.ai/free/papers/huang-2023-all-languages-created-equal-llms/)
11. Eirini Kaldeli, Orfeas Menis-Mastromichalakis, Spyros Bekiaris, Maria Ralli, Vassilis Tzouvaras, and Giorgos Stamou. 2021. Crowdheritage: crowdsourcing for improving the quality of cultural heritage metadata. Information, 12(2). [↗](https://doi.org/10.3390/info12020064)
12. Brendan Kennedy, Xisen Jin, Aida Mostafazadeh Davani, Morteza Dehghani, and Xiang Ren. 2020. Contextualizing hate speech classifiers with post-hoc explanation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5435–5442, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Brendan+Kennedy%2C+Xisen+Jin%2C+Aida+Mostafazadeh+Davani%2C+Morteza+Dehghani%2C+and+Xiang+Ren%2E+2020%2E+Contextualizing+hate+speech+classifiers+with+post%2Dhoc+explanation%2E+In+Proceedings+of+the+58th+Annual+Meetin)
13. Urja Khurana, Eric Nalisnick, Antske Fokkens, and Swabha Swayamdipta. 2024. Crowd-calibrator: Can annotator disagreement inform calibration in subjective tasks? arXiv preprint arXiv:2408.14141. [↗](https://arxiv.org/abs/2408.14141)
14. Lucija Krusic. 2024. Constructing a sentiment-annotated corpus of Austrian historical newspapers: Challenges, tools, and annotator experience. In Proceedings of the 4th International Conference on Natural Language Processing for Digital Humanities, pages 51–62, Miami, USA. Association for Computational Linguistics. [↗](https://doi.org/10.5281/zenodo.14244177)
15. Sean MacAvaney, Hao-Ren Yao, Eugene Yang, Katina Russell, Nazli Goharian, and Ophir Frieder. 2019. Hate speech detection: Challenges and solutions. PloS one, 14(8):e0221152. [↗](https://doi.org/10.1371/journal.pone.0221152)
16. Andreas Madsen, Sarath Chandar, and Siva Reddy. 2024. Are self-explanations from large language models faithful? In Findings of the Association for Computational Linguistics: ACL 2024, pages 295–337, Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Andreas+Madsen%2C+Sarath+Chandar%2C+and+Siva+Reddy%2E+2024%2E+Are+self%2Dexplanations+from+large+language+models+faithful%3F+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2024%2C+pages+295%E2%80%93337%2C+)
17. Orfeas Menis Mastromichalakis, Jason Liartis, Kristina Rose, Antoine Isaac, and Giorgos Stamou. 2025. Don’t erase, inform! detecting and contextualizing harmful language in cultural heritage collections. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 21836–21850, Vienna, Austria. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2505.24538)
18. Wayne Modest. 2018. Words Matter: an unfinished guide to word choices in the cultural sector. Tropenmuseum, Afrikamuseum, Museum Volkenkunde, Wereldmuseum. [find](https://scholar.google.com/scholar?q=Wayne+Modest%2E+2018%2E+Words+Matter%3A+an+unfinished+guide+to+word+choices+in+the+cultural+sector%2E+Tropenmuseum%2C+Afrikamuseum%2C+Museum+Volkenkunde%2C+Wereldmuseum%2E)
19. Andrei Nesterov, Laura Hollink, Marieke van Erp, and Jacco van Ossenbruggen. 2023. A knowledge graph of contentious terminology for inclusive representation of cultural heritage. In The Semantic Web, pages 502–519, Cham. Springer Nature Switzerland. [find](https://scholar.google.com/scholar?q=Andrei+Nesterov%2C+Laura+Hollink%2C+Marieke+van+Erp%2C+and+Jacco+van+Ossenbruggen%2E+2023%2E+A+knowledge+graph+of+contentious+terminology+for+inclusive+representation+of+cultural+heritage%2E+In+The+Semantic+Web%2C+)
20. Andrei Nesterov, Laura Hollink, and Jacco van Ossenbruggen. 2024. How contentious terms about people and cultures are used in linked open data. In Proceedings of the ACM Web Conference 2024, WWW ’24, page 4523–4533, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Andrei+Nesterov%2C+Laura+Hollink%2C+and+Jacco+van+Ossenbruggen%2E+2024%2E+How+contentious+terms+about+people+and+cultures+are+used+in+linked+open+data%2E+In+Proceedings+of+the+ACM+Web+Conference+2024%2C+WWW+%E2%80%9924%2C+)
21. Temi Odumosu. 2020. The crying child: On colonial archives, digitization, and ethics of care in the cultural commons. Current Anthropology, 61(S22):S289–S302. [find](https://scholar.google.com/scholar?q=Temi+Odumosu%2E+2020%2E+The+crying+child%3A+On+colonial+archives%2C+digitization%2C+and+ethics+of+care+in+the+cultural+commons%2E+Current+Anthropology%2C+61%28S22%29%3AS289%E2%80%93S302%2E)
22. Giulia Osti and Elizabeth Russey Roke. 2024. Collaborating for change? assessing metadata inclusivity in digital collections with large language models (llms). In 2024 IEEE International Conference on Big Data (BigData), pages 2479– 2488. IEEE. [find](https://scholar.google.com/scholar?q=Giulia+Osti+and+Elizabeth+Russey+Roke%2E+2024%2E+Collaborating+for+change%3F+assessing+metadata+inclusivity+in+digital+collections+with+large+language+models+%28llms%29%2E+In+2024+IEEE+International+Conference+on)
23. Ji Ho Park, Jamin Shin, and Pascale Fung. 2018. Reducing gender bias in abusive language detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2799–2804, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ji+Ho+Park%2C+Jamin+Shin%2C+and+Pascale+Fung%2E+2018%2E+Reducing+gender+bias+in+abusive+language+detection%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+279) [our card](https://md.research.elamur.ai/free/papers/park-2018-reducing-gender-bias-abusive-language/)
24. Sarthak Roy, Ashish Harshvardhan, Animesh Mukherjee, and Punyajoy Saha. 2023. Probing LLMs for hate speech detection: strengths and vulnerabilities. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6116–6128, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sarthak+Roy%2C+Ashish+Harshvardhan%2C+Animesh+Mukherjee%2C+and+Punyajoy+Saha%2E+2023%2E+Probing+LLMs+for+hate+speech+detection%3A+strengths+and+vulnerabilities%2E+In+Findings+of+the+Association+for+Computational+Li)
25. Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5884– 5906, Seattle, United States. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2111.07997) [our card](https://md.research.elamur.ai/free/papers/sap-2022-annotators-attitudes-how-annotator-beliefs/)
26. Vithya Yogarajan, Gillian Dobbie, Tim Pistotti, Joshua Bensemann, and Kobe Knowles. 2023. Challenges in annotating datasets to quantify bias in under-represented society. ArXiv, abs/2309.08624. [↗](https://arxiv.org/abs/2309.08624)
27. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12697–12706. PMLR. [↗](https://doi.org/10.48550/arxiv.2102.09690) [our card](https://md.research.elamur.ai/free/papers/zhao-2021-calibrate-before-use-improving-few/)
28. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2024. Can large language models transform computational social science? Computational Linguistics, 50(1):237–291. [↗](https://doi.org/10.1162/coli_a_00502) [our card](https://md.research.elamur.ai/free/papers/ziems-2023-large-language-models-transform-computational/)