---
title: "Sensitivity, Performance, Robustness: Deconstructing the Effect"
description: "Annotators of subjective NLP tasks such as toxicity labeling often disagree, partly because of their… 73 references, cited by 54 papers here."
authors: ["Tilman Beck", "Hendrik Schuff", "Anne Lauscher", "Iryna Gurevych"]
year: 2024
venue: "Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.eacl-long.159/"
doi: "10.18653/v1/2024.eacl-long.159"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/beck-2024-sensitivity-performance-robustness-deconstructing-effect/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/beck-2024-sensitivity-performance-robustness-deconstructing-effect/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting

# Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting

**Authors:** [Tilman Beck](https://md.research.elamur.ai/free/en/authors/tilman-beck/), [Hendrik Schuff](https://md.research.elamur.ai/free/en/authors/hendrik-schuff/), [Anne Lauscher](https://md.research.elamur.ai/free/en/authors/anne-lauscher/) and [Iryna Gurevych](https://md.research.elamur.ai/free/en/authors/iryna-gurevych/) · **Year:** 2024 · **Venue:** [Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2024](https://md.research.elamur.ai/free/en/venues/eacl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.eacl-long.159/) · [PDF](https://aclanthology.org/2024.eacl-long.159.pdf) · [DOI 10.18653/v1/2024.eacl-long.159](https://doi.org/10.18653/v1/2024.eacl-long.159) · [OpenAlex](https://openalex.org/W4411630036)

**Citations:** 14 citations (OpenAlex, 2026-10-14) · [54](https://md.research.elamur.ai/free/en/papers/beck-2024-sensitivity-performance-robustness-deconstructing-effect/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Annotators of subjective NLP tasks such as toxicity labeling often disagree, partly because of their gender, age, education and politics. Prior work disagrees on whether sociodemographic prompting, which adds a described persona to an LLM prompt so it answers like that group, actually works. Beck et al. run the largest study of this technique to date. They test 17 instruction-tuned LLMs from six model families, including InstructGPT and Flan-T5, on seven English datasets covering four tasks: sentiment, hate speech, toxicity and stance.

They measure three things: how often predictions change (sensitivity), zero-shot accuracy (performance), and stability across prompt wordings and models (robustness). Persona information changes outputs substantially and can raise zero-shot accuracy. However, the effects depend heavily on model type, model size, dataset and phrasing.

The authors conclude that the method needs careful evaluation before it is used for annotation or for judging LLM alignment.

## Key points
- Each profile is built from five attributes: a person's gender and race, which age bracket they fall into, how much schooling they have completed, and which political party they support. These follow US-specific categories.
- Two datasets (DP and Diaz) come with real annotator profiles. The other five reuse DP annotator profiles, following Wan et al. (2023), with five profiles per example.
- Adding personas altered up to 80% of predictions. Zero-shot accuracy improved by up to +8pp.
- Open-source models were scored by the likelihood of each candidate label. Closed-source model outputs were mapped to the label space afterwards.
- Code and data are released at the UKPLab GitHub repository (arxiv2023-sociodemographic-prompting).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Technische Universität Darmstadt](https://md.research.elamur.ai/free/en/universities/technische-universitat-darmstadt/) — University, Darmstadt, Germany · 76 works · 35,008 citations
- [Universität Hamburg](https://md.research.elamur.ai/free/en/universities/universitat-hamburg/) — University, Hamburg, Germany · 39 works · 4,669 citations
- [Hessian Center for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/hessian-center-for-artificial-intelligence/) — Research institute, Darmstadt, Germany · 4 works · 297 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 744 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 221 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works

## Cite

Cite as: Tilman Beck, Hendrik Schuff, Anne Lauscher and Iryna Gurevych. 2024. Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{beck2024sensitivity,
  title = {{Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting}},
  author = {Tilman Beck, Hendrik Schuff, Anne Lauscher and Iryna Gurevych},
  booktitle = {Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.eacl-long.159},
  url = {https://doi.org/10.18653/v1/2024.eacl-long.159}
}
```

- [← previous VeriScore: Evaluating the factuality of verifiable claims in long-form text generation](https://md.research.elamur.ai/free/en/papers/song-2024-veriscore-evaluating-factuality-verifiable-claims/)
- [next → Language Models as Inductive Reasoners](https://md.research.elamur.ai/free/en/papers/yang-2024-language-models-as-inductive-reasoners/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

73 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Hala Al Kuwatly, Maximilian Wich, and Georg Groh. 2020. [Identifying and measuring annotator bias based on annotators’ demographic characteristics](https://md.research.elamur.ai/free/en/papers/kuwatly-2020-identifying-measuring-annotator-bias-based/). In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 184–190, Online. Association for Computational Linguistics.
2. Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023. [Out of one, many: Using language models to simulate human samples](https://md.research.elamur.ai/free/en/papers/argyle-2022-out-one-many-using-language/). Political Analysis, 31(3):337–351.
3. Giuseppe Attanasio, Flor Plaza del Arco, Debora Nozza, and Anne Lauscher. 2023. A tale of pronouns: Interpretability informs gender bias mitigation for fairer instruction-tuned machine translation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3996–… [find](https://scholar.google.com/scholar?q=Giuseppe+Attanasio%2C+Flor+Plaza+del+Arco%2C+Debora+Nozza%2C+and+Anne+Lauscher%2E+2023%2E+A+tale+of+pronouns%3A+Interpretability+informs+gender+bias+mitigation+for+fairer+instruction%2Dtuned+machine+translation%2E+In)
4. Alexandra Balahur, Ralf Steinberger, Mijail Kabadjov, Vanni Zavarella, Erik van der Goot, Matina Halkia, Bruno Pouliquen, and Jenya Belyaeva. 2010. Sentiment analysis in the news. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta.… [find](https://scholar.google.com/scholar?q=Alexandra+Balahur%2C+Ralf+Steinberger%2C+Mijail+Kabadjov%2C+Vanni+Zavarella%2C+Erik+van+der+Goot%2C+Matina+Halkia%2C+Bruno+Pouliquen%2C+and+Jenya+Belyaeva%2E+2010%2E+Sentiment+analysis+in+the+news%2E+In+Proceedings+of+th)
5. Soumya Barikeri, Anne Lauscher, Ivan Vulić, and Goran Glavaš. 2021. RedditBias: A real-world resource for bias evaluation and debiasing of conversational language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conf… [find](https://scholar.google.com/scholar?q=Soumya+Barikeri%2C+Anne+Lauscher%2C+Ivan+Vulic%CC%81%2C+and+Goran+Glava%C5%A1%2E+2021%2E+RedditBias%3A+A+real%2Dworld+resource+for+bias+evaluation+and+debiasing+of+conversational+language+models%2E+In+Proceedings+of+the+59th+A)
6. Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. 2021. [We need to consider disagreement in evaluation](https://md.research.elamur.ai/free/en/papers/basile-2021-we-need-consider-disagreement-evaluation/). In Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future, pages 15–21, Online. Association for Computa…
7. Tilman Beck, Ji-Ung Lee, Christina Viehmann, Marcus Maurer, Oliver Quiring, and Iryna Gurevych. 2021. Investigating label suggestions for opinion mining in German covid-19 social media. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internatio… [find](https://scholar.google.com/scholar?q=Tilman+Beck%2C+Ji%2DUng+Lee%2C+Christina+Viehmann%2C+Marcus+Maurer%2C+Oliver+Quiring%2C+and+Iryna+Gurevych%2E+2021%2E+Investigating+label+suggestions+for+opinion+mining+in+German+covid%2D19+social+media%2E+In+Proceedings)
8. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International Conferenc… [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Quentin+Gregory+Anthony%2C+Herbie+Bradley%2C+Kyle+O%E2%80%99Brien%2C+Eric+Hallahan%2C+Mohammad+Aflah+Khan%2C+Shivanshu+Purohit%2C+USVSN+Sai+Prashanth%2C+Edward+Raff%2C+et+al%2E+2023%2E+Pythia%3A)
9. Laura Biester, Vanita Sharma, Ashkan Kazemi, Naihao Deng, Steven Wilson, and Rada Mihalcea. 2022. Analyzing the effects of annotator gender across NLP tasks. In Proceedings of the 1st Workshop on Perspectivist Approaches to NLP @LREC2022, pages 10–19, Marseille, France. European Language Resources A… [find](https://scholar.google.com/scholar?q=Laura+Biester%2C+Vanita+Sharma%2C+Ashkan+Kazemi%2C+Naihao+Deng%2C+Steven+Wilson%2C+and+Rada+Mihalcea%2E+2022%2E+Analyzing+the+effects+of+annotator+gender+across+NLP+tasks%2E+In+Proceedings+of+the+1st+Workshop+on+Pers)
10. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. [Language (technology) is power: A critical survey of “bias” in NLP](https://md.research.elamur.ai/free/en/papers/blodgett-2020-language-technology-is-power-critical/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454– 5476, Online. Association for Computational Linguistics.
11. [Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,…](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
12. Myra Cheng, Esin Durmus, and Dan Jurafsky. 2023. [Marked personas: Using natural language prompts to measure stereotypes in language models](https://md.research.elamur.ai/free/en/papers/cheng-2023-marked-personas-natural-language-prompts/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1504–1532, Toronto, Canada. Associatio…
13. Processing, pages 55–65, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Processing%2C+pages+55%E2%80%9365%2C+Online%2E+Association+for+Computational+Linguistics%2E)
14. [Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang…](https://md.research.elamur.ai/free/en/papers/chung-2022-scaling-instruction-finetuned-language-models/)
15. Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselut. 2023. [Mitigating label biases for in-context learning](https://md.research.elamur.ai/free/en/papers/fei-2023-mitigating-label-biases-context-learning/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14014–14031, Toronto, Canada. Association for Computational Linguisti…
16. Aida Mostafazadeh Davani, Mohammad Atari, Brendan Kennedy, and Morteza Dehghani. 2023. Hate speech classifiers learn normative social stereotypes. Transactions of the Association for Computational Linguistics, 11:300–319. [find](https://scholar.google.com/scholar?q=Aida+Mostafazadeh+Davani%2C+Mohammad+Atari%2C+Brendan+Kennedy%2C+and+Morteza+Dehghani%2E+2023%2E+Hate+speech+classifiers+learn+normative+social+stereotypes%2E+Transactions+of+the+Association+for+Computational+Lin)
17. Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023. [Toxicity in chatgpt: Analyzing persona-assigned language models](https://md.research.elamur.ai/free/en/papers/deshpande-2023-toxicity-chatgpt-analyzing-persona-assigned/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1236–1270, Singapore. Association for Computational Ling…
18. Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2022. [8-bit optimizers via block-wise quantization](https://md.research.elamur.ai/free/en/papers/dettmers-2021-8-bit-optimizers-block-wise/). In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
19. Mark Diaz, Isaac Johnson, Amanda Lazar, Anne Marie Piper, and Darren Gergle. 2018. Addressing age-related bias in sentiment analysis. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI 2018, Montreal, QC, Canada, April 21-26, 2018, page 412. ACM. [find](https://scholar.google.com/scholar?q=Mark+Diaz%2C+Isaac+Johnson%2C+Amanda+Lazar%2C+Anne+Marie+Piper%2C+and+Darren+Gergle%2E+2018%2E+Addressing+age%2Drelated+bias+in+sentiment+analysis%2E+In+Proceedings+of+the+2018+CHI+Conference+on+Human+Factors+in+Comp)
20. Mark Díaz, Ian D. Kivlichan, Rachel Rosen, Dylan K. Baker, Razvan Amironesei, Vinodkumar Prabhakaran, and Emily Denton. 2022. Crowdworksheets: Accounting for individual and collective identities underlying crowdsourced dataset annotation. In FAccT ’22: 2022 ACM Conference on Fairness, Accountability… [find](https://scholar.google.com/scholar?q=Mark+D%C3%ADaz%2C+Ian+D%2E+Kivlichan%2C+Rachel+Rosen%2C+Dylan+K%2E+Baker%2C+Razvan+Amironesei%2C+Vinodkumar+Prabhakaran%2C+and+Emily+Denton%2E+2022%2E+Crowdworksheets%3A+Accounting+for+individual+and+collective+identities+under)
21. Esin Durmus, Karina Nyugen, Thomas I. Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. 2023. [Towards measuring t…](https://md.research.elamur.ai/free/en/papers/durmus-2023-towards-measuring-representation-subjective-global/)
22. Elizabeth Excell and Noura Al Moubayed. 2021. Towards equal gender representation in the annotations of toxic language detection. In Proceedings of the 3rd Workshop on Gender Bias in Natural Language Eve Fleisig, Rediet Abebe, and Dan Klein. 2023. When the majority is wrong: Modeling annotator disag… [find](https://scholar.google.com/scholar?q=Elizabeth+Excell+and+Noura+Al+Moubayed%2E+2021%2E+Towards+equal+gender+representation+in+the+annotations+of+toxic+language+detection%2E+In+Proceedings+of+the+3rd+Workshop+on+Gender+Bias+in+Natural+Language+)
23. Mitchell L. Gordon, Michelle S. Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S. Bernstein. 2022. Jury learning: Integrating dissenting voices into machine learning models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ’22, Ne… [find](https://scholar.google.com/scholar?q=Mitchell+L%2E+Gordon%2C+Michelle+S%2E+Lam%2C+Joon+Sung+Park%2C+Kayur+Patel%2C+Jeff+Hancock%2C+Tatsunori+Hashimoto%2C+and+Michael+S%2E+Bernstein%2E+2022%2E+Jury+learning%3A+Integrating+dissenting+voices+into+machine+learning+)
24. Nitesh Goyal, Ian D. Kivlichan, Rachel Rosen, and Lucy Vasserman. 2022. Is your toxicity my toxicity? exploring the impact of rater identity on toxicity annotation. Proc. ACM Hum. Comput. Interact., 6(CSCW2):1–28. [find](https://scholar.google.com/scholar?q=Nitesh+Goyal%2C+Ian+D%2E+Kivlichan%2C+Rachel+Rosen%2C+and+Lucy+Vasserman%2E+2022%2E+Is+your+toxicity+my+toxicity%3F+exploring+the+impact+of+rater+identity+on+toxicity+annotation%2E+Proc%2E+ACM+Hum%2E+Comput%2E+Interact%2E%2C+6)
25. Soumyajit Gupta, Sooyong Lee, Maria De-Arteaga, and Matthew Lease. 2023. Same same, but different: Conditional multi-task learning for demographic-specific toxicity detection. In Proceedings of the ACM Web Conference 2023, WWW ’23, page 3689–3700, New York, NY, USA. Association for Computing Machine… [find](https://scholar.google.com/scholar?q=Soumyajit+Gupta%2C+Sooyong+Lee%2C+Maria+De%2DArteaga%2C+and+Matthew+Lease%2E+2023%2E+Same+same%2C+but+different%3A+Conditional+multi%2Dtask+learning+for+demographic%2Dspecific+toxicity+detection%2E+In+Proceedings+of+the+AC)
26. Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. [ToxiGen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection](https://md.research.elamur.ai/free/en/papers/hartvigsen-2022-toxigen-large-scale-machine-generated/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1:…
27. EunJeong Hwang, Bodhisattwa Majumder, and Niket Tandon. 2023. [Aligning language models to user opinions](https://md.research.elamur.ai/free/en/papers/hwang-2023-aligning-language-models-user-opinions/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5906– 5919, Singapore. Association for Computational Linguistics.
28. Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, Xian Li, Brian O’Horo, Gabriel Pereyra, Jeff Wang, Christopher Dewan, Asli Celikyilmaz, Luke Zettlemoyer, and Ves Stoyanov. 2022. [OPT-IML: scaling lang…](https://md.research.elamur.ai/free/en/papers/iyer-2022-opt-iml-scaling-language-model/)
29. Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81–93. [find](https://scholar.google.com/scholar?q=Maurice+G+Kendall%2E+1938%2E+A+new+measure+of+rank+correlation%2E+Biometrika%2C+30%281%2F2%29%3A81%E2%80%9393%2E)
30. Brendan Kennedy, Mohammad Atari, Aida Mostafazadeh Davani, Leigh Yeh, Ali Omrani, Yehsong Kim, Kris Coombs, Shreya Havaldar, Gwenyth Portillo-Wightman, Elaine Gonzalez, et al. 2022. Introducing the gab hate corpus: defining and applying hate-based rhetoric to social media posts at scale. Language Re… [find](https://scholar.google.com/scholar?q=Brendan+Kennedy%2C+Mohammad+Atari%2C+Aida+Mostafazadeh+Davani%2C+Leigh+Yeh%2C+Ali+Omrani%2C+Yehsong+Kim%2C+Kris+Coombs%2C+Shreya+Havaldar%2C+Gwenyth+Portillo%2DWightman%2C+Elaine+Gonzalez%2C+et+al%2E+2022%2E+Introducing+the+ga)
31. Dilek Küçük and Fazli Can. 2020. Stance detection: A survey. ACM Comput. Surv., 53(1). [find](https://scholar.google.com/scholar?q=Dilek+K%C3%BC%C3%A7%C3%BCk+and+Fazli+Can%2E+2020%2E+Stance+detection%3A+A+survey%2E+ACM+Comput%2E+Surv%2E%2C+53%281%29%2E)
32. Deepak Kumar, Patrick Gage Kelley, Sunny Consolvo, Joshua Mason, Elie Bursztein, Zakir Durumeric, Kurt Thomas, and Michael Bailey. 2021. Designing toxic content classification for a diversity of perspectives. In Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021), pages 299–318. [find](https://scholar.google.com/scholar?q=Deepak+Kumar%2C+Patrick+Gage+Kelley%2C+Sunny+Consolvo%2C+Joshua+Mason%2C+Elie+Bursztein%2C+Zakir+Durumeric%2C+Kurt+Thomas%2C+and+Michael+Bailey%2E+2021%2E+Designing+toxic+content+classification+for+a+diversity+of+persp)
33. Sawan Kumar and Partha Talukdar. 2021. Reordering examples helps during priming-based few-shot learning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4507–4518, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sawan+Kumar+and+Partha+Talukdar%2E+2021%2E+Reordering+examples+helps+during+priming%2Dbased+few%2Dshot+learning%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL%2DIJCNLP+2021%2C+pages+4507%E2%80%934518%2C)
34. Savannah Larimore, Ian Kennedy, Breon Haskett, and Alina Arseniev-Koehler. 2021. Reconsidering annotator disagreement about racist language: Noise or signal? In Proceedings of the Ninth International Workshop on Natural Language Processing for Social Media, pages 81–90, Online. Association for Compu… [find](https://scholar.google.com/scholar?q=Savannah+Larimore%2C+Ian+Kennedy%2C+Breon+Haskett%2C+and+Alina+Arseniev%2DKoehler%2E+2021%2E+Reconsidering+annotator+disagreement+about+racist+language%3A+Noise+or+signal%3F+In+Proceedings+of+the+Ninth+International+)
35. Anne Lauscher, Federico Bianchi, Samuel R. Bowman, and Dirk Hovy. 2022a. SocioProbe: What, when, and where language models learn about sociodemographics. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7901–7918, Abu Dhabi, United Arab Emirates. Assoc… [find](https://scholar.google.com/scholar?q=Anne+Lauscher%2C+Federico+Bianchi%2C+Samuel+R%2E+Bowman%2C+and+Dirk+Hovy%2E+2022a%2E+SocioProbe%3A+What%2C+when%2C+and+where+language+models+learn+about+sociodemographics%2E+In+Proceedings+of+the+2022+Conference+on+Empir)
36. Anne Lauscher, Henning Wachsmuth, Iryna Gurevych, and Goran Glavaš. 2022b. Scientia potentia Est—On the role of knowledge in computational argumentation. Transactions of the Association for Computational Linguistics, 10:1392–1422. [find](https://scholar.google.com/scholar?q=Anne+Lauscher%2C+Henning+Wachsmuth%2C+Iryna+Gurevych%2C+and+Goran+Glava%C5%A1%2E+2022b%2E+Scientia+potentia+Est%E2%80%94On+the+role+of+knowledge+in+computational+argumentation%2E+Transactions+of+the+Association+for+Computatio)
37. Noah Lee, Na Min An, and James Thorne. 2023. Can large language models capture dissenting human voices? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4569–4585, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Noah+Lee%2C+Na+Min+An%2C+and+James+Thorne%2E+2023%2E+Can+large+language+models+capture+dissenting+human+voices%3F+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages)
38. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. [Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity](https://md.research.elamur.ai/free/en/papers/lu-2022-fantastically-ordered-prompts-where-find/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p…
39. Yiwei Luo, Dallas Card, and Dan Jurafsky. 2020. Detecting stance in media on global warming. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3296–3315, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yiwei+Luo%2C+Dallas+Card%2C+and+Dan+Jurafsky%2E+2020%2E+Detecting+stance+in+media+on+global+warming%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2020%2C+pages+3296%E2%80%933315%2C+Online%2E+Associat)
40. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. [Rethinking the role of demonstrations: What makes in-context learning work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/) In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064, Ab…
41. Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016. [SemEval-2016 task 6: Detecting stance in tweets](https://md.research.elamur.ai/free/en/papers/mohammad-2016-semeval-2016-task-6-detecting/). In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 31– 41, San Diego, California. Association for Computational Ling…
42. John Nickolls, Ian Buck, Michael Garland, and Kevin Skadron. 2008. Scalable parallel programming with cuda: Is cuda the parallel programming model that application developers have been waiting for? Queue, 6(2):40–53. [find](https://scholar.google.com/scholar?q=John+Nickolls%2C+Ian+Buck%2C+Michael+Garland%2C+and+Kevin+Skadron%2E+2008%2E+Scalable+parallel+programming+with+cuda%3A+Is+cuda+the+parallel+programming+model+that+application+developers+have+been+waiting+for%3F+Qu)
43. Matthias Orlikowski, Paul Röttger, Philipp Cimiano, and Dirk Hovy. 2023. The ecological fallacy in annotation: Modeling human label variation goes beyond sociodemographics. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1017… [find](https://scholar.google.com/scholar?q=Matthias+Orlikowski%2C+Paul+R%C3%B6ttger%2C+Philipp+Cimiano%2C+and+Dirk+Hovy%2E+2023%2E+The+ecological+fallacy+in+annotation%3A+Modeling+human+label+variation+goes+beyond+sociodemographics%2E+In+Proceedings+of+the+61st+)
44. [Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022.…](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
45. Cecilia Ovesdotter Alm. 2011. Subjective natural language problems: Motivations, applications, characterizations, and implications. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 107–112, Portland, Oregon, USA. Associati… [find](https://scholar.google.com/scholar?q=Cecilia+Ovesdotter+Alm%2E+2011%2E+Subjective+natural+language+problems%3A+Motivations%2C+applications%2C+characterizations%2C+and+implications%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Com)
46. Joon Sung Park, Lindsay Popowski, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2022. Social simulacra: Creating populated prototypes for social computing systems. In The 35th Annual ACM Symposium on User Interface Software and Technology, UIST 2022, Bend, OR, USA, 29… [find](https://scholar.google.com/scholar?q=Joon+Sung+Park%2C+Lindsay+Popowski%2C+Carrie+J%2E+Cai%2C+Meredith+Ringel+Morris%2C+Percy+Liang%2C+and+Michael+S%2E+Bernstein%2E+2022%2E+Social+simulacra%3A+Creating+populated+prototypes+for+social+computing+systems%2E+In+T)
47. [Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, a…](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/)
48. Desmond Upton Patton, Philipp Blandfort, William R Frey, Michael B Gaskell, and Svebor Karaman. 2019. Annotating twitter data from vulnerable populations: Evaluating disagreement between domain experts and graduate student annotators.(2019). In Proceedings of the 52nd Hawaii International Conference… [find](https://scholar.google.com/scholar?q=Desmond+Upton+Patton%2C+Philipp+Blandfort%2C+William+R+Frey%2C+Michael+B+Gaskell%2C+and+Svebor+Karaman%2E+2019%2E+Annotating+twitter+data+from+vulnerable+populations%3A+Evaluating+disagreement+between+domain+expert)
49. Jiaxin Pei and David Jurgens. 2023. When do annotator demographics matter? measuring the influence of annotator demographics with the POPQUORN dataset. In Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII), pages 252–265, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jiaxin+Pei+and+David+Jurgens%2E+2023%2E+When+do+annotator+demographics+matter%3F+measuring+the+influence+of+annotator+demographics+with+the+POPQUORN+dataset%2E+In+Proceedings+of+the+17th+Linguistic+Annotation)
50. Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021. [On releasing annotator-level labels and information in datasets](https://md.research.elamur.ai/free/en/papers/prabhakaran-2021-releasing-annotator-level-labels-information/). In Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 133–138, Punta Cana, Dominican…
51. R Core Team. 2023. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. [find](https://scholar.google.com/scholar?q=R+Core+Team%2E+2023%2E+R%3A+A+Language+and+Environment+for+Statistical+Computing%2E+R+Foundation+for+Statistical+Computing%2C+Vienna%2C+Austria%2E)
52. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). The Journal of Machine Learning Research, 21(1):5485–5551.
53. Paul Rottger, Bertie Vidgen, Dirk Hovy, and Janet Pierrehumbert. 2022. [Two contrasting data annotation paradigms for subjective NLP tasks](https://md.research.elamur.ai/free/en/papers/rottger-2022-two-contrasting-data-annotation-paradigms/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 175–190,…
54. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. [Whose opinions do language models reflect?](https://md.research.elamur.ai/free/en/papers/santurkar-2023-whose-opinions-language-models-reflect/) In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research,…
55. Sebastin Santy, Jenny Liang, Ronan Le Bras, Katharina Reinecke, and Maarten Sap. 2023. [NLPositionality: Characterizing design biases of datasets and models](https://md.research.elamur.ai/free/en/papers/santy-2023-nlpositionality-characterizing-design-biases-datasets/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9080–9102, Toronto, C…
56. Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. [Annotators with attitudes: How annotator beliefs and identities bias toxic language detection](https://md.research.elamur.ai/free/en/papers/sap-2022-annotators-attitudes-how-annotator-beliefs/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Ling…
57. Qinlan Shen and Carolyn Rose. 2021. What sounds “right” to me? experiential factors in the perception of political ideology. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1762–1771, Online. Association for Computati… [find](https://scholar.google.com/scholar?q=Qinlan+Shen+and+Carolyn+Rose%2E+2021%2E+What+sounds+%E2%80%9Cright%E2%80%9D+to+me%3F+experiential+factors+in+the+perception+of+political+ideology%2E+In+Proceedings+of+the+16th+Conference+of+the+European+Chapter+of+the+Associ)
58. Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. 2023. [UL2: unifying language learning paradigms](https://md.research.elamur.ai/free/en/papers/tay-2022-ul2-unifying-language-learning-paradigms/). In The Eleventh International Conference on Learning Represen…
59. Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385–1470. [find](https://scholar.google.com/scholar?q=Alexandra+N+Uma%2C+Tommaso+Fornaciari%2C+Dirk+Hovy%2C+Silviu+Paun%2C+Barbara+Plank%2C+and+Massimo+Poesio%2E+2021%2E+Learning+from+disagreement%3A+A+survey%2E+Journal+of+Artificial+Intelligence+Research%2C+72%3A1385%E2%80%931470%2E)
60. Eddie Ungless, Bjorn Ross, and Anne Lauscher. 2023. Stereotypes and smut: The (mis)representation of non-cisgender identities by text-to-image models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7919–7942, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Eddie+Ungless%2C+Bjorn+Ross%2C+and+Anne+Lauscher%2E+2023%2E+Stereotypes+and+smut%3A+The+%28mis%29representation+of+non%2Dcisgender+identities+by+text%2Dto%2Dimage+models%2E+In+Findings+of+the+Association+for+Computational+)
61. Ruyuan Wan, Jaehyung Kim, and Dongyeop Kang. 2023. Everyone’s voice matters: Quantifying annotation disagreement using demographic information. In Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI… [find](https://scholar.google.com/scholar?q=Ruyuan+Wan%2C+Jaehyung+Kim%2C+and+Dongyeop+Kang%2E+2023%2E+Everyone%E2%80%99s+voice+matters%3A+Quantifying+annotation+disagreement+using+demographic+information%2E+In+Thirty%2DSeventh+AAAI+Conference+on+Artificial+Intellig)
62. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Swaroop+Mishra%2C+Pegah+Alipoormolabashi%2C+Yeganeh+Kordi%2C+Amirreza+Mirzaei%2C+Atharva+Naik%2C+Arjun+Ashok%2C+Arut+Selvan+Dhanasekaran%2C+Anjana+Arunkumar%2C+David+Stap%2C+Eshaan+Pathak%2C+Giannis+Karaman)
63. Zeerak Waseem. 2016. [Are you a racist or am I seeing things? annotator influence on hate speech detection on Twitter](https://md.research.elamur.ai/free/en/papers/waseem-2016-are-you-racist-or-am/). In Proceedings of the First Workshop on NLP and Computational Social Science, pages 138– 142, Austin, Texas. Association for Computational Linguistics.
64. [OPT: open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). CoRR, abs/2205.01068.
65. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. [Calibrate before use: Improving few-shot performance of language models](https://md.research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/). In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12697–12706. PMLR.
66. [Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho…](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
67. S. N. Wood. 2003. Thin-plate regression splines. Journal of the Royal Statistical Society (B), 65(1):95–114. [find](https://scholar.google.com/scholar?q=S%2E+N%2E+Wood%2E+2003%2E+Thin%2Dplate+regression+splines%2E+Journal+of+the+Royal+Statistical+Society+%28B%29%2C+65%281%29%3A95%E2%80%93114%2E)
68. S. N. Wood. 2004. Stable and efficient multiple smoothing parameter estimation for generalized additive models. Journal of the American Statistical Association, 99(467):673–686. [find](https://scholar.google.com/scholar?q=S%2E+N%2E+Wood%2E+2004%2E+Stable+and+efficient+multiple+smoothing+parameter+estimation+for+generalized+additive+models%2E+Journal+of+the+American+Statistical+Association%2C+99%28467%29%3A673%E2%80%93686%2E)
69. S. N. Wood. 2011. Fast stable restricted maximum likelihood and marginal likelihood estimation of semiparametric generalized linear models. Journal of the Royal Statistical Society (B), 73(1):3–36. [find](https://scholar.google.com/scholar?q=S%2E+N%2E+Wood%2E+2011%2E+Fast+stable+restricted+maximum+likelihood+and+marginal+likelihood+estimation+of+semiparametric+generalized+linear+models%2E+Journal+of+the+Royal+Statistical+Society+%28B%29%2C+73%281%29%3A3%E2%80%9336%2E)
70. Simon N Wood. 2017. Generalized additive models: an introduction with R. CRC press. [find](https://scholar.google.com/scholar?q=Simon+N+Wood%2E+2017%2E+Generalized+additive+models%3A+an+introduction+with+R%2E+CRC+press%2E)
71. S.N. Wood, N., Pya, and B. Säfken. 2016. Smoothing parameter and model selection for general smooth models (with discussion). Journal of the American Statistical Association, 111:1548–1575. [find](https://scholar.google.com/scholar?q=S%2EN%2E+Wood%2C+N%2E%2C+Pya%2C+and+B%2E+S%C3%A4fken%2E+2016%2E+Smoothing+parameter+and+model+selection+for+general+smooth+models+%28with+discussion%29%2E+Journal+of+the+American+Statistical+Association%2C+111%3A1548%E2%80%931575%2E)
72. Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. Compositional exemplars for in-context learning. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 39818–39833. PML… [find](https://scholar.google.com/scholar?q=Jiacheng+Ye%2C+Zhiyong+Wu%2C+Jiangtao+Feng%2C+Tao+Yu%2C+and+Lingpeng+Kong%2E+2023%2E+Compositional+exemplars+for+in%2Dcontext+learning%2E+In+International+Conference+on+Machine+Learning%2C+ICML+2023%2C+23%2D29+July+2023%2C+H)
73. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. [find](https://scholar.google.com/scholar?q=Susan+Zhang%2C+Stephen+Roller%2C+Naman+Goyal%2C+Mikel+Artetxe%2C+Moya+Chen%2C+Shuohui+Chen%2C+Christopher+Dewan%2C+Mona+T%2E+Diab%2C+Xian+Li%2C+Xi+Victoria+Lin%2C+Todor+Mihaylov%2C+Myle+Ott%2C+Sam+Shleifer%2C+Kurt+Shuster%2C+Danie)

*Updated 24 October 2026*