---
title: "Can LLMs Simulate Human Behavioral Variability? A — Qiu 2026"
description: "Researchers increasingly consider LLMs as stand-ins for human participants, but it is unclear whether models can reproduce how people differ from one…"
authors: ["Mengyang Qiu", "Zoe Brisebois", "Siena Sun"]
year: 2026
venue: "Proceedings of the 15th Workshop on Cognitive Modeling and Computational Linguistics"
source: "https://aclanthology.org/2026.cmcl-1.19/"
doi: "10.63317/3ncnwh7mvnfp"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/qiu-2026-can-llms-simulate-human-behavioral/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/qiu-2026-can-llms-simulate-human-behavioral/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/) / Can LLMs Simulate Human Behavioral Variability? A Case Study in the Phonemic Fluency Task

# Can LLMs Simulate Human Behavioral Variability? A Case Study in the Phonemic Fluency Task

**Authors:** Mengyang Qiu, Zoe Brisebois, Siena Sun · **Year:** 2026 · **Venue:** Proceedings of the 15th Workshop on Cognitive Modeling and Computational Linguistics · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.cmcl-1.19/) · [PDF](https://aclanthology.org/2026.cmcl-1.19.pdf) · [DOI 10.63317/3ncnwh7mvnfp](https://doi.org/10.63317/3ncnwh7mvnfp) · [OpenAlex](https://openalex.org/W4416447808)

**Citations:** 1 citations (openalex-snapshot · 2026-10-14) · [1](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Researchers increasingly consider LLMs as stand-ins for human participants, but it is unclear whether models can reproduce how people differ from one another. The authors test this with phonemic fluency, a task in which people name as many words starting with a given letter as they can in one minute. Unlike semantic fluency, it depends less on meaning. The human data come from 106 native English speakers recruited on Amazon Mechanical Turk who produced words beginning with F (Qiu and Johns, 2021).

The authors tested 34 LLMs in 45 configurations, varying thinking mode and reasoning effort. Each configuration was prompted 106 times, once per real participant, with that person's age, education and number of correct answers. Outputs were compared on response counts, lexical diversity, item-level production frequencies, word co-occurrence networks and multi-model ensembles.

Some models, notably Claude 3.7 Sonnet, matched human averages and word preferences. However, every single model and every ensemble was less diverse than the human group, and newer or reasoning-enabled models often narrowed variability further. The results caution against treating LLMs as substitutes for human samples.

## Key points
- Humans produced 16.89 valid F-words on average (SD = 4.84). A configuration counted as adhering to the target count if its MAE was ≤ 1.69 (10% of the human mean). 33 of 45 configurations met this threshold, and the 21 of those with no large per-participant deviations (> 5 words) were kept for the main analyses.
- Extreme overproducers included GPT-4 Turbo (MAE = 57.17) and o3 (MAE = 35.39). Claude 4.6 Sonnet (Thinking) and Claude 4.6 Opus also missed the target count.
- Two metrics quantified lexical variability: TTR, which relates distinct words to all words produced, and ITTTR, which relates idiosyncratic words to all distinct words. LLM outputs were consistently less diverse than human ones.
- In stepwise regressions, human production frequency was explained by word frequency, orthographic neighborhood size and age of acquisition (40.0% of variance). For Claude 3.7 Sonnet (36.5%), word length replaced neighborhood size. o3-mini explained only 18.8% of variance.
- Ensembles that drew each simulated participant from a randomly chosen model still fell short of human diversity, probably because the models share much of their vocabulary. Network analysis also showed a different retrieval structure in the most human-like model.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

## Cited by

Cited by

1 paper of our library cites it (of the 140673 whose bibliographies we read); each has a page here.

- [Focus particles and scalar inferences across humans and language models](https://md.research.elamur.ai/free/en/papers/brousse-2026-focus-particles-scalar-inferences-across/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{qiu2026can,
  title = {{Can LLMs Simulate Human Behavioral Variability? A Case Study in the Phonemic Fluency Task}},
  author = {Qiu, Mengyang and Brisebois, Zoe and Sun, Siena},
  booktitle = {Proceedings of the 15th Workshop on Cognitive Modeling and Computational Linguistics},
  year = {2026},
  doi = {10.63317/3ncnwh7mvnfp},
  url = {https://doi.org/10.63317/3ncnwh7mvnfp}
}
```

- [← previous Can LLMs Faithfully Explain Themselves in Low-Resource Languages? A Case Study on Emotion Detection in Persian](https://md.research.elamur.ai/free/en/papers/mehrazar-2026-can-llms-faithfully-explain-themselves/)
- [next → CATENG Submission for the IWSLT 2026: Dialectal and Low-resource Speech Translation Task](https://md.research.elamur.ai/free/en/papers/zevallos-2026-cateng-submission-iwslt-2026-dialectal/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/)

## References

28 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. David A Balota, Melvin J Yap, Keith A Hutchison, Michael J Cortese, Brett Kessler, Bjorn Loftis, James H Neely, Douglas L Nelson, Greg B Simpson, and Rebecca Treiman. 2007. The English lexicon project. Behavior Research Methods, 39(3):445–459. [find](https://scholar.google.com/scholar?q=David+A+Balota%2C+Melvin+J+Yap%2C+Keith+A+Hutchison%2C+Michael+J+Cortese%2C+Brett+Kessler%2C+Bjorn+Loftis%2C+James+H+Neely%2C+Douglas+L+Nelson%2C+Greg+B+Simpson%2C+and+Rebecca+Treiman%2E+2007%2E+The+English+lexicon+project)
2. Katy Borodkin, Yoed N Kenett, Miriam Faust, and Nira Mashal. 2016. When pumpkin is closer to onion than to squash: The structure of the second language lexicon. Cognition, 156:60–70. [find](https://scholar.google.com/scholar?q=Katy+Borodkin%2C+Yoed+N+Kenett%2C+Miriam+Faust%2C+and+Nira+Mashal%2E+2016%2E+When+pumpkin+is+closer+to+onion+than+to+squash%3A+The+structure+of+the+second+language+lexicon%2E+Cognition%2C+156%3A60%E2%80%9370%2E)
3. Nichol Castro, Taylor Curley, and Christopher Hertzog. 2021. Category norms with a cross-sectional sample of adults in the united states: Consideration of cohort, age, and historical effects on semantic categories. Behavior Research Methods, 53:898–917. [find](https://scholar.google.com/scholar?q=Nichol+Castro%2C+Taylor+Curley%2C+and+Christopher+Hertzog%2E+2021%2E+Category+norms+with+a+cross%2Dsectional+sample+of+adults+in+the+united+states%3A+Consideration+of+cohort%2C+age%2C+and+historical+effects+on+semant)
4. Sunghye Cho, Naomi Nevler, Natalia Parjane, Christopher Cieri, Mark Liberman, Murray Grossman, and Katheryn AQ Cousins. 2021. Automated analysis of digitized letter fluency data. Frontiers in Psychology, 12:654214. [find](https://scholar.google.com/scholar?q=Sunghye+Cho%2C+Naomi+Nevler%2C+Natalia+Parjane%2C+Christopher+Cieri%2C+Mark+Liberman%2C+Murray+Grossman%2C+and+Katheryn+AQ+Cousins%2E+2021%2E+Automated+analysis+of+digitized+letter+fluency+data%2E+Frontiers+in+Psycholo)
5. Christine Cuskley, Rebecca Woods, and Molly Flaherty. 2024. The limitations of large language models for understanding human language and cognition. Open Mind, 8:1058–1083. [find](https://scholar.google.com/scholar?q=Christine+Cuskley%2C+Rebecca+Woods%2C+and+Molly+Flaherty%2E+2024%2E+The+limitations+of+large+language+models+for+understanding+human+language+and+cognition%2E+Open+Mind%2C+8%3A1058%E2%80%931083%2E)
6. Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. 2023. Can ai language models replace human participants? Trends in Cognitive Sciences, 27(7):597–600. [find](https://scholar.google.com/scholar?q=Danica+Dillion%2C+Niket+Tandon%2C+Yuling+Gu%2C+and+Kurt+Gray%2E+2023%2E+Can+ai+language+models+replace+human+participants%3F+Trends+in+Cognitive+Sciences%2C+27%287%29%3A597%E2%80%93600%2E)
7. Paul Erdős and Alfréd Rényi. 1960. On the evolution of random graphs. Publ. Math. Inst. Hung. Acad. Sci, 5(1):17–60. [find](https://scholar.google.com/scholar?q=Paul+Erdo%CC%8Bs+and+Alfr%C3%A9d+R%C3%A9nyi%2E+1960%2E+On+the+evolution+of+random+graphs%2E+Publ%2E+Math%2E+Inst%2E+Hung%2E+Acad%2E+Sci%2C+5%281%29%3A17%E2%80%9360%2E)
8. Julie Akiko Gladsjo, Catherine C Schuman, Jovier D Evans, Guerry M Peavy, S Walden Miller, and Robert K Heaton. 1999. Norms for letter and category fluency: demographic corrections for age, education, and ethnicity. Assessment, 6(2):147–178. [find](https://scholar.google.com/scholar?q=Julie+Akiko+Gladsjo%2C+Catherine+C+Schuman%2C+Jovier+D+Evans%2C+Guerry+M+Peavy%2C+S+Walden+Miller%2C+and+Robert+K+Heaton%2E+1999%2E+Norms+for+letter+and+category+fluency%3A+demographic+corrections+for+age%2C+education%2C)
9. Yufei Guo, Muzhe Guo, Juntao Su, Zhou Yang, Mengqiu Zhu, Hongfei Li, Mengyang Qiu, and Shuo Shuo Liu. 2024. Bias in large language models: Origin, evaluation, and mitigation. arXiv preprint arXiv:2411.10915. [find](https://scholar.google.com/scholar?q=Yufei+Guo%2C+Muzhe+Guo%2C+Juntao+Su%2C+Zhou+Yang%2C+Mengqiu+Zhu%2C+Hongfei+Li%2C+Mengyang+Qiu%2C+and+Shuo+Shuo+Liu%2E+2024%2E+Bias+in+large+language+models%3A+Origin%2C+evaluation%2C+and+mitigation%2E+arXiv+preprint+arXiv%3A2411)
10. Edith Haim, Lars van den Bergh, Cynthia SQ Siew, Yoed N Kenett, Daniele Marinazzo, and Massimo Stella. 2025. Cognitive networks highlight differences and similarities in the stem mindsets of human and llm-simulated trainees, experts and academics. arXiv. [find](https://scholar.google.com/scholar?q=Edith+Haim%2C+Lars+van+den+Bergh%2C+Cynthia+SQ+Siew%2C+Yoed+N+Kenett%2C+Daniele+Marinazzo%2C+and+Massimo+Stella%2E+2025%2E+Cognitive+networks+highlight+differences+and+similarities+in+the+stem+mindsets+of+human+and)
11. Hannes Hansen and Martin N Hebart. 2022. Semantic features of object concepts generated with gpt-3. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 44. [find](https://scholar.google.com/scholar?q=Hannes+Hansen+and+Martin+N+Hebart%2E+2022%2E+Semantic+features+of+object+concepts+generated+with+gpt%2D3%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Cognitive+Science+Society%2C+volume+44%2E)
12. Cassandra L Jacobs, Loïc Grobol, and Alvin Tsang. 2024. Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically aligned. arXiv preprint arXiv:2410.12057. — [arXiv:2410.12057](https://arxiv.org/abs/2410.12057) [find](https://scholar.google.com/scholar?q=Cassandra+L+Jacobs%2C+Lo%C3%AFc+Grobol%2C+and+Alvin+Tsang%2E+2024%2E+Large%2Dscale+cloze+evaluation+reveals+that+token+prediction+tasks+are+neither+lexically+nor+semantically+aligned%2E+arXiv+preprint+arXiv%3A2410%2E12057)
13. Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye. 2024. More agents is all you need. arXiv. [find](https://scholar.google.com/scholar?q=Junyou+Li%2C+Qin+Zhang%2C+Yangbin+Yu%2C+Qiang+Fu%2C+and+Deheng+Ye%2E+2024%2E+More+agents+is+all+you+need%2E+arXiv%2E)
14. Yu Li and Mengyang Qiu. 2025. A network analysis of the semantic evolution of ‘fruit’ and ‘stone’ in Tibeto-Burman languages. Poznan Studies in Contemporary Linguistics, 61(2):121–172. [find](https://scholar.google.com/scholar?q=Yu+Li+and+Mengyang+Qiu%2E+2025%2E+A+network+analysis+of+the+semantic+evolution+of+%E2%80%98fruit%E2%80%99+and+%E2%80%98stone%E2%80%99+in+Tibeto%2DBurman+languages%2E+Poznan+Studies+in+Contemporary+Linguistics%2C+61%282%29%3A121%E2%80%93172%2E)
15. Guido Previde Massara, Tiziana Di Matteo, and Tomaso Aste. 2017. Network filtering for big data: Triangulated maximally filtered graph. Journal of Complex Networks, 5(2):161–178. [find](https://scholar.google.com/scholar?q=Guido+Previde+Massara%2C+Tiziana+Di+Matteo%2C+and+Tomaso+Aste%2E+2017%2E+Network+filtering+for+big+data%3A+Triangulated+maximally+filtered+graph%2E+Journal+of+Complex+Networks%2C+5%282%29%3A161%E2%80%93178%2E)
16. Hamed Nili, Cai Wingfield, Alexander Walther, Li Su, William Marslen-Wilson, and Nikolaus Kriegeskorte. 2014. A toolbox for representational similarity analysis. PLoS Computational Biology, 10(4):e1003553. [find](https://scholar.google.com/scholar?q=Hamed+Nili%2C+Cai+Wingfield%2C+Alexander+Walther%2C+Li+Su%2C+William+Marslen%2DWilson%2C+and+Nikolaus+Kriegeskorte%2E+2014%2E+A+toolbox+for+representational+similarity+analysis%2E+PLoS+Computational+Biology%2C+10%284%29%3Ae100)
17. Mengyang Qiu, Nichol Castro, and Brendan T. Johns. 2021. Structural comparisons of noun and verb networks in the mental lexicon. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 43, pages 1649–1655. [find](https://scholar.google.com/scholar?q=Mengyang+Qiu%2C+Nichol+Castro%2C+and+Brendan+T%2E+Johns%2E+2021%2E+Structural+comparisons+of+noun+and+verb+networks+in+the+mental+lexicon%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Cognitive+Science+Society%2C+)
18. Mengyang Qiu, Nichol Castro, and Brendan T. Johns. 2024. Estimating type of print exposure across aging through author production. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 46, pages 1175–1181. [find](https://scholar.google.com/scholar?q=Mengyang+Qiu%2C+Nichol+Castro%2C+and+Brendan+T%2E+Johns%2E+2024%2E+Estimating+type+of+print+exposure+across+aging+through+author+production%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Cognitive+Science+Society)
19. Mengyang Qiu and Brendan T. Johns. 2021. A distributional and sensorimotor analysis of noun and verb fluency. PsyArXiv. [find](https://scholar.google.com/scholar?q=Mengyang+Qiu+and+Brendan+T%2E+Johns%2E+2021%2E+A+distributional+and+sensorimotor+analysis+of+noun+and+verb+fluency%2E+PsyArXiv%2E)
20. Cynthia SQ Siew and Anutra Guru. 2023. Investigating the network structure of domain-specific knowledge using the semantic fluency task. Memory & Cognition, 51(3):623–646. [find](https://scholar.google.com/scholar?q=Cynthia+SQ+Siew+and+Anutra+Guru%2E+2023%2E+Investigating+the+network+structure+of+domain%2Dspecific+knowledge+using+the+semantic+fluency+task%2E+Memory+%26+Cognition%2C+51%283%29%3A623%E2%80%93646%2E)
21. Cynthia SQ Siew, Dirk U Wulff, Nicole M Beckage, and Yoed N Kenett. 2019. Cognitive network science: A review of research on cognition through the lens of network representations, processes, and dynamics. [find](https://scholar.google.com/scholar?q=Cynthia+SQ+Siew%2C+Dirk+U+Wulff%2C+Nicole+M+Beckage%2C+and+Yoed+N+Kenett%2E+2019%2E+Cognitive+network+science%3A+A+review+of+research+on+cognition+through+the+lens+of+network+representations%2C+processes%2C+and+dynam)
22. Vanessa Taler, Brendan T Johns, and Michael N Jones. 2020. A large-scale semantic analysis of verbal fluency across the aging spectrum: Data from the Canadian Longitudinal Study on Aging. The Journals of Gerontology Series B: Psychological Sciences and Social Sciences, 75(9):e221–e230. [find](https://scholar.google.com/scholar?q=Vanessa+Taler%2C+Brendan+T+Johns%2C+and+Michael+N+Jones%2E+2020%2E+A+large%2Dscale+semantic+analysis+of+verbal+fluency+across+the+aging+spectrum%3A+Data+from+the+Canadian+Longitudinal+Study+on+Aging%2E+The+Journals)
23. Tom N Tombaugh, Jean Kozak, and Laura Rees. 1999. Normative data stratified by age and education for two measures of verbal fluency: FAS and animal naming. Archives of Clinical Neuropsychology, 14(2):167–177. [find](https://scholar.google.com/scholar?q=Tom+N+Tombaugh%2C+Jean+Kozak%2C+and+Laura+Rees%2E+1999%2E+Normative+data+stratified+by+age+and+education+for+two+measures+of+verbal+fluency%3A+FAS+and+animal+naming%2E+Archives+of+Clinical+Neuropsychology%2C+14%282%29%3A)
24. Sean Trott. 2024. Large language models and the wisdom of small crowds. Open Mind, 8:723–738. [find](https://scholar.google.com/scholar?q=Sean+Trott%2E+2024%2E+Large+language+models+and+the+wisdom+of+small+crowds%2E+Open+Mind%2C+8%3A723%E2%80%93738%2E)
25. Dolores Villalobos, Lucía Torres-Simón, Javier Pa-cios, Nuria Paul, and David Del Río. 2023. A systematic review of normative data for verbal fluency test in different languages. Neuropsychology Review, 33(4):733–764. [find](https://scholar.google.com/scholar?q=Dolores+Villalobos%2C+Luc%C3%ADa+Torres%2DSim%C3%B3n%2C+Javier+Pa%2Dcios%2C+Nuria+Paul%2C+and+David+Del+R%C3%ADo%2E+2023%2E+A+systematic+review+of+normative+data+for+verbal+fluency+test+in+different+languages%2E+Neuropsychology+Revie)
26. Qian Wang, Zhenheng Tang, and Bingsheng He. 2025a. Can llm simulations truly reflect humanity? a deep dive. In ICLR Blogposts 2025. [find](https://scholar.google.com/scholar?q=Qian+Wang%2C+Zhenheng+Tang%2C+and+Bingsheng+He%2E+2025a%2E+Can+llm+simulations+truly+reflect+humanity%3F+a+deep+dive%2E+In+ICLR+Blogposts+2025%2E)
27. Ye Wang, Yaling Deng, Ge Wang, Tong Li, Hongjiang Xiao, and Yuan Zhang. 2025b. The fluency-based semantic network of LLMs differs from humans. Computers in Human Behavior: Artificial Humans, 3:100103. [find](https://scholar.google.com/scholar?q=Ye+Wang%2C+Yaling+Deng%2C+Ge+Wang%2C+Tong+Li%2C+Hongjiang+Xiao%2C+and+Yuan+Zhang%2E+2025b%2E+The+fluency%2Dbased+semantic+network+of+LLMs+differs+from+humans%2E+Computers+in+Human+Behavior%3A+Artificial+Humans%2C+3%3A100103%2E)
28. Sergio E Zanotto and Segun Aroyehun. 2024. Human variability vs. machine consistency: A linguistic analysis of texts generated by humans and large language models. arXiv. [find](https://scholar.google.com/scholar?q=Sergio+E+Zanotto+and+Segun+Aroyehun%2E+2024%2E+Human+variability+vs%2E+machine+consistency%3A+A+linguistic+analysis+of+texts+generated+by+humans+and+large+language+models%2E+arXiv%2E)