---
title: "Susceptibility to Influence of Large Language Models (2023)"
description: "They compare this to how mice are used as physiological models in drug research. 14 references, cited by 10 papers here."
authors: ["Lewis D Griffin", "Bennett Kleinberg", "Maximilian Mozes", "Kimberly T Mai", "Maria Vau", "Matthew Caldwell", "Augustine Marvor-Parker"]
year: 2023
venue: "arXiv:2303.06074"
source: "https://arxiv.org/abs/2303.06074"
doi: "10.48550/arxiv.2303.06074"
arxiv: "2303.06074"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/griffin-2023-susceptibility-influence-large-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/griffin-2023-susceptibility-influence-large-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/) / Susceptibility to Influence of Large Language Models

# Susceptibility to Influence of Large Language Models

**Authors:** Lewis D Griffin, [Bennett Kleinberg](https://md.research.elamur.ai/free/en/authors/bennett-kleinberg/), [Maximilian Mozes](https://md.research.elamur.ai/free/en/authors/maximilian-mozes/), Kimberly T Mai, Maria Vau, Matthew Caldwell and Augustine Marvor-Parker · **Year:** 2023 · **Venue:** arXiv:2303.06074 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2303.06074) · [PDF](https://arxiv.org/pdf/2303.06074) · [DOI 10.48550/arxiv.2303.06074](https://doi.org/10.48550/arxiv.2303.06074) · [arXiv 2303.06074](https://arxiv.org/abs/2303.06074) · [OpenAlex](https://openalex.org/W4324108791)

**Citations:** 6 citations (OpenAlex, 2026-10-14) · [10](https://md.research.elamur.ai/free/en/papers/griffin-2023-susceptibility-influence-large-language-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The authors ask whether a large language model can serve as a stand-in for people when studying how influence changes beliefs. They compare this to how mice are used as physiological models in drug research. The motivation is that human experiments on persuasion are slow, costly and ethically constrained. Using GPT-3 (text-davinci-003) with carefully engineered prompts, they ran two studies.

The first examined the Illusory Truth Effect, where prior exposure to a statement later raises its perceived truthfulness. An online experiment with human participants was mirrored by an equal number of simulated respondents. GPT-3 showed the same pattern as people: a truth boost appeared only under specific conditions.

The second study re-ran a published multi-country experiment on populist news framing. GPT-3 reproduced several human findings, including one that had contradicted the original authors' expectations. It missed others, notably effects tied to participants' relative deprivation.

Overall, the results suggest LLMs could become cheap, partial models for research on influence and disinformation, though they do not fully match human behaviour.

## Key points
- **Model and setup:** GPT-3 text-davinci-003 was accessed via the playground for pilots and via the API for the main runs. Its 4000-token prompt limit made it possible to simulate full experimental procedures.
- **Illusory Truth Effect study:** 1000 human and 1000 LLM-simulated participants each gave 64 ratings. These covered every exposure–test pairing of four attributes: truth, interest, sentiment and importance.
- **Illusory Truth Effect results:** In both humans and GPT-3, earlier exposure raised truth ratings. No effect appeared for the other test attributes, or when the same attribute was used at both exposure and test.
- **Populist framing study:** Simulated data were compared with the 15-country human experiment by Bos et al. (2020), which involved 7286 participants.
- **Populist framing results:** GPT-3 replicated the counterintuitive human finding that anti-immigrant framing lowers persuasion and mobilization. It did not show the human pattern in which relative deprivation modulates how well populist framing works.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University College London](https://md.research.elamur.ai/free/en/universities/university-college-london/) — University, London, United Kingdom · 88 works · 19,432 citations
- [Tilburg University](https://md.research.elamur.ai/free/en/universities/tilburg-university/) — University, Tilburg, Netherlands · 50 works · 6,429 citations

📚 Shelves (4): [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 577 works · [⚛️ Physics and Astronomy](https://md.research.elamur.ai/free/en/shelves/physics-and-astronomy/) · 44 works · [⚛️ Statistical and Nonlinear Physics](https://md.research.elamur.ai/free/en/shelves/physics-and-astronomy/statistical-and-nonlinear-physics/) · 40 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Lewis D Griffin, Bennett Kleinberg, Maximilian Mozes, Kimberly T Mai, Maria Vau, Matthew Caldwell and Augustine Marvor-Parker. 2023. Susceptibility to Influence of Large Language Models. arXiv:2303.06074.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{griffin2023susceptibility,
  title = {{Susceptibility to Influence of Large Language Models}},
  author = {Lewis D Griffin, Bennett Kleinberg, Maximilian Mozes, Kimberly T Mai, Maria Vau, Matthew Caldwell and Augustine Marvor-Parker},
  year = {2023},
  doi = {10.48550/arxiv.2303.06074},
  eprint = {2303.06074},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.06074}
}
```

- [← previous Statistical Methods for Annotation Analysis](https://md.research.elamur.ai/free/en/papers/wilkens-2023-statistical-methods-annotation-analysis/)
- [next → VinaLLaMA: LLaMA-based Vietnamese Foundation Model](https://md.research.elamur.ai/free/en/papers/nguyen-2023-vinallama-llama-based-vietnamese-foundation/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/)

## References

14 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.11.12.13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. Miller, M.D. and T.R. Levine, Persuasion, in An integrated approach to communication theory and research. 2019, Routledge. p. 261-276. Miller, N., et al., Speed of speech and persuasion. Journal of personality an… [find](https://scholar.google.com/scholar?q=1%2E+2%2E+3%2E+4%2E+5%2E+6%2E+7%2E+8%2E+9%2E+10%2E11%2E12%2E13%2E+14%2E+15%2E+16%2E+17%2E+18%2E+19%2E+20%2E+21%2E+22%2E+23%2E+24%2E+25%2E+Miller%2C+M%2ED%2E+and+T%2ER%2E+Levine%2C+Persuasion%2C+in+An+integrated+approach+to+communication+theory+and+research%2E+2019%2C+)
2. Social and Personality Psychology Compass, 2011. 5(4): p. 206-219. Shao, J., et al., Credulity rather than general trust may increase vulnerability to fraud in older adults: A moderated mediation model. Journal of elder abuse & neglect, 2019. 31(2): p. 146-162. Moscovici, S., Attitudes and opinions.… [find](https://scholar.google.com/scholar?q=Social+and+Personality+Psychology+Compass%2C+2011%2E+5%284%29%3A+p%2E+206%2D219%2E+Shao%2C+J%2E%2C+et+al%2E%2C+Credulity+rather+than+general+trust+may+increase+vulnerability+to+fraud+in+older+adults%3A+A+moderated+mediation+mode)
3. American Economic Review, 2010. 100(5): p. 2361-2382. Argyle, L.P., et al., [Out of One, Many: Using Language Models to Simulate Human Samples](https://md.research.elamur.ai/free/en/papers/argyle-2022-out-one-many-using-language/). arXiv preprint arXiv:2209.06899, 2022. Brown, T., et al., Language models are few-shot learners. Advances in neural information processing systems, 2020. 33:…
4. Nature Machine Intelligence, 2022. 4(3): p. 258-268. Henderson, E.L., S.J. Westwood, and D.J. Simons, A reproducible systematic map of research on the illusory truth effect. [find](https://scholar.google.com/scholar?q=Nature+Machine+Intelligence%2C+2022%2E+4%283%29%3A+p%2E+258%2D268%2E+Henderson%2C+E%2EL%2E%2C+S%2EJ%2E+Westwood%2C+and+D%2EJ%2E+Simons%2C+A+reproducible+systematic+map+of+research+on+the+illusory+truth+effect%2E)
5. Psychonomic Bulletin & Review, 2022: p. 1-24. Razniewski, S., et al., Language models as or for knowledge bases. arXiv preprint arXiv:2110.04888, 2021. Harris, Z., Distributional structure. [find](https://scholar.google.com/scholar?q=Psychonomic+Bulletin+%26+Review%2C+2022%3A+p%2E+1%2D24%2E+Razniewski%2C+S%2E%2C+et+al%2E%2C+Language+models+as+or+for+knowledge+bases%2E+arXiv+preprint+arXiv%3A2110%2E04888%2C+2021%2E+Harris%2C+Z%2E%2C+Distributional+structure%2E)
6. Word, 1954. 10(23): p. 146-162. Peirce, C.S., The collected papers of Charles Sanders Peirce. 1923, Cambridge, MA: Harvard University Press. Griffin, L.D., M.H. Wahab, and A.J. Newell, Distributional Learning of Appearance. [find](https://scholar.google.com/scholar?q=Word%2C+1954%2E+10%2823%29%3A+p%2E+146%2D162%2E+Peirce%2C+C%2ES%2E%2C+The+collected+papers+of+Charles+Sanders+Peirce%2E+1923%2C+Cambridge%2C+MA%3A+Harvard+University+Press%2E+Griffin%2C+L%2ED%2E%2C+M%2EH%2E+Wahab%2C+and+A%2EJ%2E+Newell%2C+Distributional+)
7. PLoS ONE, 2013. 8(2): p. e58074. @goodside. twitter 2022 17/oct/2022. Vaswani, A., et al., [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 2017. 30. Miotto, M., N. Rossberg, and B. Kleinberg, Who is GPT-3? An exploration of personality, values and demographics. arXiv pr… — [arXiv:2209.14338](https://arxiv.org/abs/2209.14338)
8. Sap, M., et al., Neural theory-of-mind? on the limits of social intelligence in large lms. arXiv preprint arXiv:2210.13312, 2022. Langer, W.C., et al., A psychological analysis of Adolph Hitler: His life and legend. 1943: MO Branch, Office of Strategic Services. Hasher, L., D. Goldstein, and T. Topp… [find](https://scholar.google.com/scholar?q=Sap%2C+M%2E%2C+et+al%2E%2C+Neural+theory%2Dof%2Dmind%3F+on+the+limits+of+social+intelligence+in+large+lms%2E+arXiv+preprint+arXiv%3A2210%2E13312%2C+2022%2E+Langer%2C+W%2EC%2E%2C+et+al%2E%2C+A+psychological+analysis+of+Adolph+Hitler%3A+His+l)
9. Cognitive Research: Principles and Implications, 2021. 6(1): p. 1-12. Brashier, N.M., E.D. Eliseev, and E.J. Marsh, An initial accuracy focus prevents illusory truth. [find](https://scholar.google.com/scholar?q=Cognitive+Research%3A+Principles+and+Implications%2C+2021%2E+6%281%29%3A+p%2E+1%2D12%2E+Brashier%2C+N%2EM%2E%2C+E%2ED%2E+Eliseev%2C+and+E%2EJ%2E+Marsh%2C+An+initial+accuracy+focus+prevents+illusory+truth%2E)
10. Cognition, 2020. 194: p. 104054. Reber, R. and N. Schwarz, Effects of perceptual fluency on judgments of truth. Consciousness and cognition, 1999. 8(3): p. 338-342. Winer, B.J., D.R. Brown, and K.M. Michels, Statistical principles in experimental design. Vol. 2. 1971: Mcgraw-hill New York. Bos, L.,… [find](https://scholar.google.com/scholar?q=Cognition%2C+2020%2E+194%3A+p%2E+104054%2E+Reber%2C+R%2E+and+N%2E+Schwarz%2C+Effects+of+perceptual+fluency+on+judgments+of+truth%2E+Consciousness+and+cognition%2C+1999%2E+8%283%29%3A+p%2E+338%2D342%2E+Winer%2C+B%2EJ%2E%2C+D%2ER%2E+Brown%2C+and+K%2EM%2E+M)
11. Wittgenstein, L., Philosophical Investigations. 1953, Oxford: Blackwell. Bender, E.M., et al. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?? in Proceedings of the 2021 ACM conference on fairness, accountability, and transparency. 2021. Floridi, L., AI as Agency without Intell… [find](https://scholar.google.com/scholar?q=Wittgenstein%2C+L%2E%2C+Philosophical+Investigations%2E+1953%2C+Oxford%3A+Blackwell%2E+Bender%2C+E%2EM%2E%2C+et+al%2E+On+the+Dangers+of+Stochastic+Parrots%3A+Can+Language+Models+Be+Too+Big%3F%3F+in+Proceedings+of+the+2021+ACM+conf)
12. Philosophy and Technology, 2023. Olah, C., et al., The Building Blocks of Interpretability. [find](https://scholar.google.com/scholar?q=Philosophy+and+Technology%2C+2023%2E+Olah%2C+C%2E%2C+et+al%2E%2C+The+Building+Blocks+of+Interpretability%2E)
13. Distill, 2018. 3(3): p. e10. Noam Chomsky, I.R.a.J.W., The False Promise of ChatGPT, in The New York Times. 2023. Warren, H.S., et al., Mice are not men. Proceedings of the National Academy of Sciences, 2015. 112(4): p. E345-E345. Fox, J.G., et al., The mouse in biomedical research: history, wild mi… — [arXiv:2301.04246](https://arxiv.org/abs/2301.04246) [find](https://scholar.google.com/scholar?q=Distill%2C+2018%2E+3%283%29%3A+p%2E+e10%2E+Noam+Chomsky%2C+I%2ER%2Ea%2EJ%2EW%2E%2C+The+False+Promise+of+ChatGPT%2C+in+The+New+York+Times%2E+2023%2E+Warren%2C+H%2ES%2E%2C+et+al%2E%2C+Mice+are+not+men%2E+Proceedings+of+the+National+Academy+of+Science)
14. Marr, B., Big data in practice: how 45 successful companies used big data analytics to deliver extraordinary results. 2016: John Wiley & Sons. Kyrylenko, M., Subliminal Advertising: Ethical and Legal Problems of Existence in Ukraine. Journal of Legal Studies “Vasile Goldiş”, 2021. 27(41): p. 59-71. [find](https://scholar.google.com/scholar?q=Marr%2C+B%2E%2C+Big+data+in+practice%3A+how+45+successful+companies+used+big+data+analytics+to+deliver+extraordinary+results%2E+2016%3A+John+Wiley+%26+Sons%2E+Kyrylenko%2C+M%2E%2C+Subliminal+Advertising%3A+Ethical+and+Legal+)

*Updated 30 October 2026*