--- title: "Evaluating Large Language Models in Theory — Kosinski 2023" description: "Theory of mind (ToM) is the ability to attribute hidden mental states, such as beliefs or intentions, to other people." authors: ["Michal Kosinski"] year: 2023 venue: "arXiv:2302.02083" source: "https://arxiv.org/abs/2302.02083" doi: "10.1073/pnas.2405460121" arxiv: "2302.02083" licence: "CC BY-NC-ND 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by-nc-nd/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/kosinski-2023-evaluating-large-language-models-theory/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kosinski-2023-evaluating-large-language-models-theory/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/) / Evaluating Large Language Models in Theory of Mind Tasks # Evaluating Large Language Models in Theory of Mind Tasks **Authors:** Michal Kosinski · **Year:** 2023 · **Venue:** arXiv:2302.02083 · **Licence:** [CC BY-NC-ND 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-nd/4.0/) **Links:** [Publisher's page](https://doi.org/10.1073/pnas.2405460121) · [PDF](https://arxiv.org/pdf/2302.02083) · [DOI 10.1073/pnas.2405460121](https://doi.org/10.1073/pnas.2405460121) · [arXiv 2302.02083](https://arxiv.org/abs/2302.02083) · [OpenAlex](https://openalex.org/W4403863303) **Citations:** 334 citations (openalex-snapshot · 2026-10-14) · [131](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Theory of mind (ToM) is the ability to attribute hidden mental states, such as beliefs or intentions, to other people. It underpins human social behaviour. Earlier language models, including RoBERTa and early GPT-3, failed simple ToM tests. Kosinski asks whether ToM could appear in LLMs without anyone building it in, simply as a side effect of learning language. To avoid contamination from training data, a research assistant who did not know the hypothesis wrote 40 new false-belief tasks: 20 Unexpected Contents and 20 Unexpected Transfer. Each task pairs one false-belief story with three closely matched true-belief controls, plus reversed versions of all four. A model scores a point only if it answers all eight correctly. Eleven models, from GPT-1 to ChatGPT-4 and Bloom, completed sentence-continuation prompts at temperature 0, and the first word of each completion was scored. Accuracy grew with each model generation. ChatGPT-4 solved three quarters of the tasks, about the level reported for six-year-old children in earlier studies. The author argues that machines which act as if they track minds matter socially, however the mechanism is explained. ## Key points - Each task has two prompt types: one asks for the real state of the world (e.g., what the container holds), the other for the protagonist's belief. Stories use the label and content words equally often, so word frequency alone cannot reveal the answer. - Models tested: GPT-1, GPT-2, six GPT-3 family models, ChatGPT-3.5-turbo, ChatGPT-4 (June 2023) and the open-access Bloom. - Results: older models solved 0% of tasks. GPT-3-davinci-003 (Nov 2022) and ChatGPT-3.5-turbo (Mar 2023) each solved 20%, and ChatGPT-4 solved 75%. - Contamination safeguards: the scenarios were drafted in August 2021 and released in February 2023. Only models trained on data from before that release were tested, through APIs that did not keep inputs for training. - Tasks, instructions and code are on OSF (https://osf.io/csdhb/). The 2023 arXiv preprint was later revised and published in PNAS in 2024 under a CC BY-NC-ND licence. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) ## Cited by Cited by 131 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 123 not yet. - [From Persona to Personalization: A Survey on Role-Playing Language Agents](https://md.research.elamur.ai/free/en/papers/chen-2024-persona-personalization-survey-role-playing/) - [AugGPT: Leveraging ChatGPT for Text Data Augmentation](https://md.research.elamur.ai/free/en/papers/dai-2023-auggpt-leveraging-chatgpt-text-data/) - [TrustLLM: Trustworthiness in Large Language Models](https://md.research.elamur.ai/free/en/papers/huang-2024-trustllm-trustworthiness-large-language-models/) - [ChatGPT: Jack of all trades, master of none](https://md.research.elamur.ai/free/en/papers/kocon-2023-chatgpt-jack-all-trades-master/) - [RSA-Control: A Pragmatics-Grounded Lightweight Controllable Text Generation Framework](https://md.research.elamur.ai/free/en/papers/wang-2024-rsa-control-pragmatics-grounded-lightweight/) - [This One or That One? A Study on Accessibility via Demonstratives with Multimodal Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2026-this-one-or-that-one/) - [The Rise and Potential of Large Language Model Based Agents: A Survey](https://md.research.elamur.ai/free/en/papers/xi-2023-rise-potential-large-language/) - [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{kosinski2023evaluating, title = {{Evaluating Large Language Models in Theory of Mind Tasks}}, author = {Kosinski, Michal}, year = {2023}, doi = {10.1073/pnas.2405460121}, eprint = {2302.02083}, archivePrefix = {arXiv}, url = {https://doi.org/10.1073/pnas.2405460121} } ``` - [← previous Evaluating Emotion Arcs Across Languages: Bridging the Global Divide in Sentiment Analysis](https://md.research.elamur.ai/free/en/papers/teodorescu-2023-evaluating-emotion-arcs-across-languages/) - [next → Evaluating Large Language Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/guo-2023-evaluating-large-language-models-comprehensive/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/)