---
title: "Quantifying the Persona Effect in LLM Simulations — Hu 2024"
description: "The paper asks how much demographic, social, attitudinal and behavioral \"persona\" information… 61 references, cited by 75 papers here."
authors: ["Tiancheng Hu", "Nigel Collier"]
year: 2024
venue: "arXiv:2402.10811"
source: "https://arxiv.org/abs/2402.10811"
doi: "10.48550/arxiv.2402.10811"
arxiv: "2402.10811"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hu-2024-quantifying-persona-effect-llm-simulations/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hu-2024-quantifying-persona-effect-llm-simulations/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / Quantifying the Persona Effect in LLM Simulations

# Quantifying the Persona Effect in LLM Simulations

**Authors:** Tiancheng Hu and [Nigel Collier](https://md.research.elamur.ai/free/en/authors/nigel-collier/) · **Year:** 2024 · **Venue:** [arXiv:2402.10811](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2024](https://md.research.elamur.ai/free/en/venues/acl/2024/)) · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.10811) · [PDF](https://arxiv.org/pdf/2402.10811) · [DOI 10.48550/arxiv.2402.10811](https://doi.org/10.48550/arxiv.2402.10811) · [arXiv 2402.10811](https://arxiv.org/abs/2402.10811) · [OpenAlex](https://openalex.org/W4402671727)

**Citations:** 41 citations (OpenAlex, 2026-10-14) · [75](https://md.research.elamur.ai/free/en/papers/hu-2024-quantifying-persona-effect-llm-simulations/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary

The paper asks how much demographic, social, attitudinal and behavioral "persona" information actually helps LLMs reproduce how different people judge subjective content. Hu and Collier first measure how much annotation variance personas can explain at all. They fit mixed-effect linear regressions, with persona variables as fixed effects and a random intercept per text, on 10 subjective NLP datasets that keep unaggregated labels and annotator profiles. They also include a presidential-vote question from ANES 2012 for comparison. They then prepend annotator profiles to task prompts and check whether LLM predictions improve.

Personas turn out to explain only a small share of variance in these datasets. Persona prompting yields small but statistically significant gains. In a survey-based case study, the share of variance LLMs recover rises roughly linearly with the share personas can explain. Where personas matter little, prompting with them adds almost nothing.

This matters because it gives a cheap way to estimate, from limited human data, whether LLM-based simulation is likely to work before running large experiments. It also urges caution about substituting LLMs for human annotators.

## Key points

- **Datasets:** The 10 datasets include annWithAttitudes, POPQUORN (offensiveness and politeness), Kumar et al. (2021) toxicity, Diaz et al. (2018) sentiment, Social-Chem-101, NLPositionality, SBIC and EPIC. Most draw on social media and crowd-sourced labels.
- **Explained variance:** Persona variables account for only about 1.4%–10.6% of annotation variance (marginal R²), even after controlling for text. Text-level effects explain far more (conditional R² up to 0.758 for SBIC).
- **Where prompting helps:** Gains concentrate on items with high annotation entropy but low standard deviation, meaning many annotators disagree, but only slightly.
- **Survey case study:** In a zero-shot setting, a large preference-tuned 70b model with persona prompts recovers 81% of the variance a regression on ground-truth answers achieves. When the target R² falls below 0.1, predicted R² often collapses to roughly zero.
- **Limitations:** All available persona-annotated datasets are English and mostly U.S.-based, so effectiveness for other languages, such as Ukrainian, is untested and probably weaker. Code and data are announced at github.com/cambridgeltl/persona_effect.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Cambridge](https://md.research.elamur.ai/free/en/universities/university-of-cambridge/) — University, Cambridge, United Kingdom · 155 works · 115,562 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works

## Cite

Cite as: Tiancheng Hu and Nigel Collier. 2024. Quantifying the Persona Effect in LLM Simulations. arXiv:2402.10811.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{hu2024quantifying,
  title = {{Quantifying the Persona Effect in LLM Simulations}},
  author = {Tiancheng Hu and Nigel Collier},
  year = {2024},
  doi = {10.48550/arxiv.2402.10811},
  eprint = {2402.10811},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.10811}
}
```

- [← previous ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic](https://md.research.elamur.ai/free/en/papers/koto-2024-arabicmmlu-assessing-massive-multitask-language/)
- [next → QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks](https://md.research.elamur.ai/free/en/papers/tseng-2024-quip-even-better-llm-quantization/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

61 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Gati V. Aher, Rosa I. Arriaga, and Adam Tauman Kalai. 2023. Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning R… [find](https://scholar.google.com/scholar?q=Gati+V%2E+Aher%2C+Rosa+I%2E+Arriaga%2C+and+Adam+Tauman+Kalai%2E+2023%2E+Using+Large+Language+Models+to+Simulate+Multiple+Humans+and+Replicate+Human+Subject+Studies%2E+In+International+Conference+on+Machine+Learning)
2. Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2023. [MEGA: Multilingual evaluation of generative AI](https://md.research.elamur.ai/free/en/papers/ahuja-2023-mega-multilingual-evaluation-generative-ai/). In Proceedings of the 2023 Conference on Empirical Methods in N…
3. ANES. The american national election studies 2012 time series study. [find](https://scholar.google.com/scholar?q=ANES%2E+The+american+national+election+studies+2012+time+series+study%2E)
4. Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023. [Out of One, Many: Using Language Models to Simulate Human Samples](https://md.research.elamur.ai/free/en/papers/argyle-2022-out-one-many-using-language/). Political Analysis, 31(3):337–351. Publisher: Cambridge University Press.
5. April H. Bailey, Adina Williams, and Andrei Cimpian. 2022. Based on billions of words on the internet, PEOPLE = MEN . Science Advances, 8(13):eabm2463. [find](https://scholar.google.com/scholar?q=April+H%2E+Bailey%2C+Adina+Williams%2C+and+Andrei+Cimpian%2E+2022%2E+Based+on+billions+of+words+on+the+internet%2C+PEOPLE+%3D+MEN+%2E+Science+Advances%2C+8%2813%29%3Aeabm2463%2E)
6. Larry M. Bartels. 2002. Beyond the running tally: Partisan bias in political perceptions. Political Behavior, 24(2):117–150. [find](https://scholar.google.com/scholar?q=Larry+M%2E+Bartels%2E+2002%2E+Beyond+the+running+tally%3A+Partisan+bias+in+political+perceptions%2E+Political+Behavior%2C+24%282%29%3A117%E2%80%93150%2E)
7. Tilman Beck, Hendrik Schuff, Anne Lauscher, and Iryna Gurevych. 2024. [Sensitivity, performance, robustness: Deconstructing the effect of sociodemographic prompting](https://md.research.elamur.ai/free/en/papers/beck-2024-sensitivity-performance-robustness-deconstructing-effect/). In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers),…
8. Laura Biester, Vanita Sharma, Ashkan Kazemi, Naihao Deng, Steven R. Wilson, and Rada Mihalcea. 2022. Analyzing the effects of annotator gender across NLP tasks. In Proceedings of the 1st Workshop on Perspectivist Approaches to NLPerspectives@LREC 2022, Marseille, France, 20th June 2022, pages 10–19.… [find](https://scholar.google.com/scholar?q=Laura+Biester%2C+Vanita+Sharma%2C+Ashkan+Kazemi%2C+Naihao+Deng%2C+Steven+R%2E+Wilson%2C+and+Rada+Mihalcea%2E+2022%2E+Analyzing+the+effects+of+annotator+gender+across+NLP+tasks%2E+In+Proceedings+of+the+1st+Workshop+on+P)
9. James Bisbee, Joshua D. Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer M. Larson. 2024. Synthetic replacements for human survey data? the perils of large language models. Political Analysis, page 1–16. [find](https://scholar.google.com/scholar?q=James+Bisbee%2C+Joshua+D%2E+Clinton%2C+Cassy+Dorff%2C+Brenton+Kenkel%2C+and+Jennifer+M%2E+Larson%2E+2024%2E+Synthetic+replacements+for+human+survey+data%3F+the+perils+of+large+language+models%2E+Political+Analysis%2C+page+)
10. Lawrence Bobo and Frederick C Licari. 1989. Education and political tolerance: Testing the effects of cognitive sophistication and target group affect. Public Opinion Quarterly, 53(3):285–308. [find](https://scholar.google.com/scholar?q=Lawrence+Bobo+and+Frederick+C+Licari%2E+1989%2E+Education+and+political+tolerance%3A+Testing+the+effects+of+cognitive+sophistication+and+target+group+affect%2E+Public+Opinion+Quarterly%2C+53%283%29%3A285%E2%80%93308%2E)
11. Federico Cabitza, Andrea Campagner, and Valerio Basile. 2023. [Toward a Perspectivist Turn in Ground Truthing for Predictive Computing](https://md.research.elamur.ai/free/en/papers/basile-2021-toward-perspectivist-turn-ground-truthing/). Proceedings of the AAAI Conference on Artificial Intelligence, 37(6):6860–6868. Number: 6.
12. Myra Cheng, Tiziano Piccardi, and Diyi Yang. 2023. CoMPosT: Characterizing and evaluating caricature in LLM simulations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10853–10875, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Myra+Cheng%2C+Tiziano+Piccardi%2C+and+Diyi+Yang%2E+2023%2E+CoMPosT%3A+Characterizing+and+evaluating+caricature+in+LLM+simulations%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+Natural+Language+)
13. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. [find](https://scholar.google.com/scholar?q=Jacob+Cohen%2E+1960%2E+A+coefficient+of+agreement+for+nominal+scales%2E+Educational+and+Psychological+Measurement%2C+20%281%29%3A37%E2%80%9346%2E)
14. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314. [find](https://scholar.google.com/scholar?q=Tim+Dettmers%2C+Artidoro+Pagnoni%2C+Ari+Holtzman%2C+and+Luke+Zettlemoyer%2E+2023%2E+Qlora%3A+Efficient+finetuning+of+quantized+llms%2E+arXiv+preprint+arXiv%3A2305%2E14314%2E)
15. Mark Diaz, Isaac Johnson, Amanda Lazar, Anne Marie Piper, and Darren Gergle. 2018. Addressing Age-Related Bias in Sentiment Analysis. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, pages 1–14, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Mark+Diaz%2C+Isaac+Johnson%2C+Amanda+Lazar%2C+Anne+Marie+Piper%2C+and+Darren+Gergle%2E+2018%2E+Addressing+Age%2DRelated+Bias+in+Sentiment+Analysis%2E+In+Proceedings+of+the+2018+CHI+Conference+on+Human+Factors+in+Comp)
16. Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. 2023. Can ai language models replace human participants? Trends in Cognitive Sciences, 27(7):597–600. [find](https://scholar.google.com/scholar?q=Danica+Dillion%2C+Niket+Tandon%2C+Yuling+Gu%2C+and+Kurt+Gray%2E+2023%2E+Can+ai+language+models+replace+human+participants%3F+Trends+in+Cognitive+Sciences%2C+27%287%29%3A597%E2%80%93600%2E)
17. Yi Ding, Jacob You, Tonja-Katrin Machulla, Jennifer Jacobs, Pradeep Sen, and Tobias Höllerer. 2022. Impact of Annotator Demographics on Sentiment Dataset Labeling. Proceedings of the ACM on Human-Computer Interaction, 6(CSCW2):519:1– 519:22. [find](https://scholar.google.com/scholar?q=Yi+Ding%2C+Jacob+You%2C+Tonja%2DKatrin+Machulla%2C+Jennifer+Jacobs%2C+Pradeep+Sen%2C+and+Tobias+H%C3%B6llerer%2E+2022%2E+Impact+of+Annotator+Demographics+on+Sentiment+Dataset+Labeling%2E+Proceedings+of+the+ACM+on+Human%2DComp)
18. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language P… [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Maarten+Sap%2C+Ana+Marasovic%CC%81%2C+William+Agnew%2C+Gabriel+Ilharco%2C+Dirk+Groeneveld%2C+Margaret+Mitchell%2C+and+Matt+Gardner%2E+2021%2E+Documenting+large+webtext+corpora%3A+A+case+study+on+the+colossal+cl)
19. Esin Durmus, Karina Nyugen, Thomas I. Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. 2023. [Towards Measuring t…](https://md.research.elamur.ai/free/en/papers/durmus-2023-towards-measuring-representation-subjective-global/)
20. Bradley Efron. 1992. Bootstrap Methods: Another Look at the Jackknife, pages 569–593. Springer New York, New York, NY. [find](https://scholar.google.com/scholar?q=Bradley+Efron%2E+1992%2E+Bootstrap+Methods%3A+Another+Look+at+the+Jackknife%2C+pages+569%E2%80%93593%2E+Springer+New+York%2C+New+York%2C+NY%2E)
21. Eve Fleisig, Rediet Abebe, and Dan Klein. 2023. When the majority is wrong: Modeling annotator disagreement for subjective tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6715–6726, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Eve+Fleisig%2C+Rediet+Abebe%2C+and+Dan+Klein%2E+2023%2E+When+the+majority+is+wrong%3A+Modeling+annotator+disagreement+for+subjective+tasks%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+Natural+)
22. Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020. Social chemistry 101: Learning to reason about social and moral norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 653–670, Online. Association for Computatio… [find](https://scholar.google.com/scholar?q=Maxwell+Forbes%2C+Jena+D%2E+Hwang%2C+Vered+Shwartz%2C+Maarten+Sap%2C+and+Yejin+Choi%2E+2020%2E+Social+chemistry+101%3A+Learning+to+reason+about+social+and+moral+norms%2E+In+Proceedings+of+the+2020+Conference+on+Empiric)
23. Simona Frenda, Alessandro Pedrani, Valerio Basile, Soda Marem Lo, Alessandra Teresa Cignarella, Raffaella Panizzon, Cristina Marco, Bianca Scarlini, Viviana Patti, Cristina Bosco, and Davide Bernardi. 2023. [EPIC: Multi-perspective annotation of a corpus of irony](https://md.research.elamur.ai/free/en/papers/frenda-2023-epic-multi-perspective-annotation-corpus/). In Proceedings of the 61st Annual Me…
24. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. ArXiv:2101.00027 [cs]. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+Shawn+Presser%2C+and+Connor+Leahy%2E+2020%2E+The+Pile%3A+An+800GB+Datas)
25. Mitchell L. Gordon, Michelle S. Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S. Bernstein. 2022. Jury Learning: Integrating Dissenting Voices into Machine Learning Models. In Conference on Human Factors in Computing Systems - Proceedings. Association for Computing… — [arXiv:2202.02950](https://arxiv.org/abs/2202.02950) [find](https://scholar.google.com/scholar?q=Mitchell+L%2E+Gordon%2C+Michelle+S%2E+Lam%2C+Joon+Sung+Park%2C+Kayur+Patel%2C+Jeff+Hancock%2C+Tatsunori+Hashimoto%2C+and+Michael+S%2E+Bernstein%2E+2022%2E+Jury+Learning%3A+Integrating+Dissenting+Voices+into+Machine+Learning+)
26. Jesse Graham and Jonathan Haidt. 2010. Beyond beliefs: Religions bind individuals into moral communities. Personality and social psychology review, 14(1):140–150. [find](https://scholar.google.com/scholar?q=Jesse+Graham+and+Jonathan+Haidt%2E+2010%2E+Beyond+beliefs%3A+Religions+bind+individuals+into+moral+communities%2E+Personality+and+social+psychology+review%2C+14%281%29%3A140%E2%80%93150%2E)
27. Igor Grossmann, Matthew Feinberg, Dawn C. Parker, Nicholas A. Christakis, Philip E. Tetlock, and William A. Cunningham. 2023. Ai and the transformation of social science research. Science, 380(6650):1108–1109. [find](https://scholar.google.com/scholar?q=Igor+Grossmann%2C+Matthew+Feinberg%2C+Dawn+C%2E+Parker%2C+Nicholas+A%2E+Christakis%2C+Philip+E%2E+Tetlock%2C+and+William+A%2E+Cunningham%2E+2023%2E+Ai+and+the+transformation+of+social+science+research%2E+Science%2C+380%286650%29%3A1)
28. Perttu Hämäläinen, Mikke Tavast, and Anton Kunnari. 2023. Evaluating large language models in generating synthetic hci research data: a case study. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Perttu+H%C3%A4m%C3%A4l%C3%A4inen%2C+Mikke+Tavast%2C+and+Anton+Kunnari%2E+2023%2E+Evaluating+large+language+models+in+generating+synthetic+hci+research+data%3A+a+case+study%2E+In+Proceedings+of+the+2023+CHI+Conference+on+Human+F)
29. Danula Hettiachchi, Indigo Holcombe-James, Stephanie Livingstone, Anjalee de Silva, Matthew Lease, Flora D. Salim, and Mark Sanderson. 2023. How crowd worker factors influence subjective annotations: A study of tagging misogynistic hate speech in tweets. Proceedings of the AAAI Conference on Human C… [find](https://scholar.google.com/scholar?q=Danula+Hettiachchi%2C+Indigo+Holcombe%2DJames%2C+Stephanie+Livingstone%2C+Anjalee+de+Silva%2C+Matthew+Lease%2C+Flora+D%2E+Salim%2C+and+Mark+Sanderson%2E+2023%2E+How+crowd+worker+factors+influence+subjective+annotations%3A+)
30. John J Horton. 2023. Large language models as simulated economic agents: What can we learn from homo silicus? Technical report, National Bureau of Economic Research. [find](https://scholar.google.com/scholar?q=John+J+Horton%2E+2023%2E+Large+language+models+as+simulated+economic+agents%3A+What+can+we+learn+from+homo+silicus%3F+Technical+report%2C+National+Bureau+of+Economic+Research%2E)
31. Tiancheng Hu, Yara Kyrychenko, Steve Rathje, Nigel Collier, Sander van der Linden, and Jon Roozenbeek. 2023. Generative language models exhibit social identity biases. arXiv preprint arXiv:2310.15819. — [arXiv:2310.15819](https://arxiv.org/abs/2310.15819) [find](https://scholar.google.com/scholar?q=Tiancheng+Hu%2C+Yara+Kyrychenko%2C+Steve+Rathje%2C+Nigel+Collier%2C+Sander+van+der+Linden%2C+and+Jon+Roozenbeek%2E+2023%2E+Generative+language+models+exhibit+social+identity+biases%2E+arXiv+preprint+arXiv%3A2310%2E15819%2E)
32. Olivia Huang, Eve Fleisig, and Dan Klein. 2023. Incorporating worker perspectives into MTurk annotation practices for NLP. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1010–1028, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Olivia+Huang%2C+Eve+Fleisig%2C+and+Dan+Klein%2E+2023%2E+Incorporating+worker+perspectives+into+MTurk+annotation+practices+for+NLP%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+Natural+Languag)
33. Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. 2023. [Camels in a changing climate: Enhancing lm adaptation with tulu 2](https://md.research.elamur.ai/free/en/papers/ivison-2023-camels-changing-climate-enhancing-lm/). arXiv preprint arXiv:2311.10702.
34. Junsol Kim and Byungkyu Lee. 2023. AI-Augmented Surveys: Leveraging Large Language Models for Opinion Prediction in Nationally Representative Surveys. ArXiv:2305.09620 [cs]. — [arXiv:2305.09620](https://arxiv.org/abs/2305.09620) [find](https://scholar.google.com/scholar?q=Junsol+Kim+and+Byungkyu+Lee%2E+2023%2E+AI%2DAugmented+Surveys%3A+Leveraging+Large+Language+Models+for+Opinion+Prediction+in+Nationally+Representative+Surveys%2E+ArXiv%3A2305%2E09620+%5Bcs%5D%2E+%E2%80%94+%5BarXiv%3A2305%2E09620%5D%28https)
35. Grgur Kovač, Masataka Sawayama, Rémy Portelas, Cédric Colas, Peter Ford Dominey, and Pierre-Yves Oudeyer. 2023. Large Language Models as Superpositions of Cultural Perspectives. ArXiv:2307.07870 [cs]. — [arXiv:2307.07870](https://arxiv.org/abs/2307.07870) [find](https://scholar.google.com/scholar?q=Grgur+Kovac%CC%8C%2C+Masataka+Sawayama%2C+R%C3%A9my+Portelas%2C+C%C3%A9dric+Colas%2C+Peter+Ford+Dominey%2C+and+Pierre%2DYves+Oudeyer%2E+2023%2E+Large+Language+Models+as+Superpositions+of+Cultural+Perspectives%2E+ArXiv%3A2307%2E07870+%5Bcs%5D)
36. Deepak Kumar, Patrick Gage Kelley, Sunny Consolvo, Joshua Mason, Elie Bursztein, Zakir Durumeric, Kurt Thomas, and Michael Bailey. 2021. Designing toxic content classification for a diversity of perspectives. In Proceedings of the Seventeenth USENIX Conference on Usable Privacy and Security, SOUPS’2… [find](https://scholar.google.com/scholar?q=Deepak+Kumar%2C+Patrick+Gage+Kelley%2C+Sunny+Consolvo%2C+Joshua+Mason%2C+Elie+Bursztein%2C+Zakir+Durumeric%2C+Kurt+Thomas%2C+and+Michael+Bailey%2E+2021%2E+Designing+toxic+content+classification+for+a+diversity+of+persp)
37. Nayeon Lee, Chani Jung, Junho Myung, Jiho Jin, Juho Kim, and Alice Oh. 2023. Crehate: Cross-cultural re-annotation of english hate speech dataset. arXiv preprint arXiv:2308.16705. [find](https://scholar.google.com/scholar?q=Nayeon+Lee%2C+Chani+Jung%2C+Junho+Myung%2C+Jiho+Jin%2C+Juho+Kim%2C+and+Alice+Oh%2E+2023%2E+Crehate%3A+Cross%2Dcultural+re%2Dannotation+of+english+hate+speech+dataset%2E+arXiv+preprint+arXiv%3A2308%2E16705%2E)
38. Yinhong Liu, Yimai Fang, David Vandyke, and Nigel Collier. 2024. Toad: Task-oriented automatic dialogs with diverse response styles. arXiv preprint arXiv:2402.10137. [find](https://scholar.google.com/scholar?q=Yinhong+Liu%2C+Yimai+Fang%2C+David+Vandyke%2C+and+Nigel+Collier%2E+2024%2E+Toad%3A+Task%2Doriented+automatic+dialogs+with+diverse+response+styles%2E+arXiv+preprint+arXiv%3A2402%2E10137%2E)
39. Daniel Lüdecke, Mattan S. Ben-Shachar, Indrajeet Patil, Philip Waggoner, and Dominique Makowski. 2021. performance: An R package for assessment, comparison and testing of statistical models. Journal of Open Source Software, 6(60):3139. [find](https://scholar.google.com/scholar?q=Daniel+L%C3%BCdecke%2C+Mattan+S%2E+Ben%2DShachar%2C+Indrajeet+Patil%2C+Philip+Waggoner%2C+and+Dominique+Makowski%2E+2021%2E+performance%3A+An+R+package+for+assessment%2C+comparison+and+testing+of+statistical+models%2E+Journal+o)
40. Nicola Marsden and Monika Pröbster. 2019. Personas and identity: Looking at multiple identities to inform the construction of personas. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI ’19, page 1–14, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Nicola+Marsden+and+Monika+Pr%C3%B6bster%2E+2019%2E+Personas+and+identity%3A+Looking+at+multiple+identities+to+inform+the+construction+of+personas%2E+In+Proceedings+of+the+2019+CHI+Conference+on+Human+Factors+in+Co)
41. Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022. [Dealing with disagreements: Looking beyond the majority vote in subjective annotations](https://md.research.elamur.ai/free/en/papers/davani-2022-dealing-disagreements-looking-beyond-majority/). Transactions of the Association for Computational Linguistics, 10:92–110.
42. Shinichi Nakagawa and Holger Schielzeth. 2013. A general and simple method for obtaining r2 from generalized linear mixed-effects models. Methods in Ecology and Evolution, 4(2):133–142. [find](https://scholar.google.com/scholar?q=Shinichi+Nakagawa+and+Holger+Schielzeth%2E+2013%2E+A+general+and+simple+method+for+obtaining+r2+from+generalized+linear+mixed%2Deffects+models%2E+Methods+in+Ecology+and+Evolution%2C+4%282%29%3A133%E2%80%93142%2E)
43. OpenAI. 2023a. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
44. OpenAI. 2023b. Introducing ChatGPT. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2023b%2E+Introducing+ChatGPT%2E)
45. Matthias Orlikowski, Paul Röttger, Philipp Cimiano, and Dirk Hovy. 2023. The ecological fallacy in annotation: Modeling human label variation goes beyond sociodemographics. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1017… [find](https://scholar.google.com/scholar?q=Matthias+Orlikowski%2C+Paul+R%C3%B6ttger%2C+Philipp+Cimiano%2C+and+Dirk+Hovy%2E+2023%2E+The+ecological+fallacy+in+annotation%3A+Modeling+human+label+variation+goes+beyond+sociodemographics%2E+In+Proceedings+of+the+61st+)
46. Cecilia Ovesdotter Alm. 2011. Subjective natural language problems: Motivations, applications, characterizations, and implications. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 107–112, Portland, Oregon, USA. Associati… [find](https://scholar.google.com/scholar?q=Cecilia+Ovesdotter+Alm%2E+2011%2E+Subjective+natural+language+problems%3A+Motivations%2C+applications%2C+characterizations%2C+and+implications%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Com)
47. Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simulacra of Human Behavior. In In the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23), UIST ’23, New York, NY, USA. As… [find](https://scholar.google.com/scholar?q=Joon+Sung+Park%2C+Joseph+C%2E+O%E2%80%99Brien%2C+Carrie+J%2E+Cai%2C+Meredith+Ringel+Morris%2C+Percy+Liang%2C+and+Michael+S%2E+Bernstein%2E+2023%2E+Generative+Agents%3A+Interactive+Simulacra+of+Human+Behavior%2E+In+In+the+36th+Annual)
48. Peter S Park, Philipp Schoenegger, and Chongyang Zhu. 2024. Diminished diversity-of-thought in a standard large language model. Behavior Research Methods, pages 1–17. [find](https://scholar.google.com/scholar?q=Peter+S+Park%2C+Philipp+Schoenegger%2C+and+Chongyang+Zhu%2E+2024%2E+Diminished+diversity%2Dof%2Dthought+in+a+standard+large+language+model%2E+Behavior+Research+Methods%2C+pages+1%E2%80%9317%2E)
49. Jiaxin Pei and David Jurgens. 2023. When do annotator demographics matter? measuring the influence of annotator demographics with the POPQUORN dataset. In Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII), pages 252–265, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jiaxin+Pei+and+David+Jurgens%2E+2023%2E+When+do+annotator+demographics+matter%3F+measuring+the+influence+of+annotator+demographics+with+the+POPQUORN+dataset%2E+In+Proceedings+of+the+17th+Linguistic+Annotation)
50. Pew Research Center. 2014. Political polarization in the american public. Pew Research Center. Accessed: Dec 19, 2023. [find](https://scholar.google.com/scholar?q=Pew+Research+Center%2E+2014%2E+Political+polarization+in+the+american+public%2E+Pew+Research+Center%2E+Accessed%3A+Dec+19%2C+2023%2E)
51. Barbara Plank. 2022. [The “problem” of human label variation: On ground truth in data, modeling and evaluation](https://md.research.elamur.ai/free/en/papers/plank-2022-problem-human-label-variation-ground/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
52. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. [Whose opinions do language models reflect?](https://md.research.elamur.ai/free/en/papers/santurkar-2023-whose-opinions-language-models-reflect/) In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org.
53. Sebastin Santy, Jenny Liang, Ronan Le Bras, Katharina Reinecke, and Maarten Sap. 2023. [NLPositionality: Characterizing design biases of datasets and models](https://md.research.elamur.ai/free/en/papers/santy-2023-nlpositionality-characterizing-design-biases-datasets/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9080–9102, Toronto, C…
54. Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. [Social bias frames: Reasoning about social and power implications of language](https://md.research.elamur.ai/free/en/papers/sap-2020-social-bias-frames-reasoning-about/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5477–5490, Online. Association…
55. Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. [Annotators with attitudes: How annotator beliefs and identities bias toxic language detection](https://md.research.elamur.ai/free/en/papers/sap-2022-annotators-attitudes-how-annotator-beliefs/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Ling…
56. Amir Taubenfeld, Yaniv Dover, Roi Reichart, and Ariel Goldstein. 2024. Systematic biases in llm simulations of debates. arXiv preprint arXiv:2402.04049. [find](https://scholar.google.com/scholar?q=Amir+Taubenfeld%2C+Yaniv+Dover%2C+Roi+Reichart%2C+and+Ariel+Goldstein%2E+2024%2E+Systematic+biases+in+llm+simulations+of+debates%2E+arXiv+preprint+arXiv%3A2402%2E04049%2E)
57. Petter Törnberg, Diliara Valeeva, Justus Uitermark, and Christopher Bail. 2023. Simulating social media using large language models to evaluate alternative news feed algorithms. arXiv preprint arXiv:2310.05984. — [arXiv:2310.05984](https://arxiv.org/abs/2310.05984) [find](https://scholar.google.com/scholar?q=Petter+T%C3%B6rnberg%2C+Diliara+Valeeva%2C+Justus+Uitermark%2C+and+Christopher+Bail%2E+2023%2E+Simulating+social+media+using+large+language+models+to+evaluate+alternative+news+feed+algorithms%2E+arXiv+preprint+arXiv%3A2)
58. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+Dan+Bikel%2C+Lukas+Blecher%2C+Cristian+Canton+Fe)
59. Angelina Wang, Jamie Morgenstern, and John P Dickerson. 2024. Large language models cannot replace human participants because they cannot portray identity groups. arXiv preprint arXiv:2402.01908. — [arXiv:2402.01908](https://arxiv.org/abs/2402.01908) [find](https://scholar.google.com/scholar?q=Angelina+Wang%2C+Jamie+Morgenstern%2C+and+John+P+Dickerson%2E+2024%2E+Large+language+models+cannot+replace+human+participants+because+they+cannot+portray+identity+groups%2E+arXiv+preprint+arXiv%3A2402%2E01908%2E+%E2%80%94+%5Ba)
60. Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman, Linjie Li, Jena D Hwang, Liwei Jiang, Jillian Fisher, Abhilasha Ravichander, Khyathi Chandu, et al. 2023. The generative ai paradox:" what it can create, it may not understand". arXiv preprint arXiv:2311.00059. [find](https://scholar.google.com/scholar?q=Peter+West%2C+Ximing+Lu%2C+Nouha+Dziri%2C+Faeze+Brahman%2C+Linjie+Li%2C+Jena+D+Hwang%2C+Liwei+Jiang%2C+Jillian+Fisher%2C+Abhilasha+Ravichander%2C+Khyathi+Chandu%2C+et+al%2E+2023%2E+The+generative+ai+paradox%3A%22+what+it+can+cre)
61. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)

*Updated 24 October 2026*