---
title: "R.U.Psycho? A Framework for Robust Unified — Schelb 2026"
description: "Psychometric questionnaires written for people are increasingly given to generative LLMs to profile model traits, benchmark alignment, or replace human…"
authors: ["Julian Schelb", "Orr Borin", "David Garcia", "Andreas Spitz"]
year: 2026
venue: "Proceedings of the Fifteenth Language Resources and Evaluation Conference"
source: "https://aclanthology.org/2026.lrec-1.661/"
doi: "10.63317/4d7ofew6usug"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/schelb-2026-r-u-psycho-framework-robust/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/schelb-2026-r-u-psycho-framework-robust/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/9/) / R.U.Psycho? A Framework for Robust Unified Psychometric Testing of Language Models

# R.U.Psycho? A Framework for Robust Unified Psychometric Testing of Language Models

**Authors:** Julian Schelb, Orr Borin, David Garcia, Andreas Spitz · **Year:** 2026 · **Venue:** Proceedings of the Fifteenth Language Resources and Evaluation Conference · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.lrec-1.661/) · [PDF](https://aclanthology.org/2026.lrec-1.661.pdf) · [DOI 10.63317/4d7ofew6usug](https://doi.org/10.63317/4d7ofew6usug) · [OpenAlex](https://openalex.org/W7160097490)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Psychometric questionnaires written for people are increasingly given to generative LLMs to profile model traits, benchmark alignment, or replace human participants through persona simulation. These studies are hard to reproduce. Model outputs fluctuate, and results shift with prompt wording and ordering, generation parameters, and model version, choices that papers often document poorly. The authors respond with R.U.Psycho, an open-source Python package built on LangChain that lets researchers run such experiments with little programming.

A whole experiment is specified in one JSON configuration file, which also serves as its documentation. The file covers:
- the models and their generation settings,
- prompt templates with placeholders,
- persona attributes,
- the questionnaire, with its instructions and answer options.

The pipeline collects free-text answers, cleans and checks them, maps them onto the allowed options, and exports the results. The authors tested it on four psychometric instruments and one thought experiment, and the outcomes agreed with earlier published findings. The tool's value is that machine-psychology studies become easier to standardize, share, and repeat.

## Key points
- **Pipeline:** four stages run in sequence.
  1. Experiment definition.
  2. Response generation.
  3. Post-processing, with Cleaner, Validator and Judge components.
  4. Export to CSV, DataFrame or JSONL.
- **Model access:** OpenAI-compatible APIs, local Huggingface models (with optional quantization) and the Huggingface Inference API.
  - Both chat-style and single-prompt formats are supported.
  - Temperature, max output tokens and random seeds are fixed in the config.
- **Answer method:** answers are generated as free text rather than scored by token probabilities, because the authors cite known problems with probability-based measurement (Wang et al., 2024).
  - Several responses are sampled per question for each model, seed and persona.
- **Personas:** stored as attribute dictionaries (e.g., title, name, age, ethnicity) and inserted into prompt templates. The attributes stay available for stratified analysis later.
- **Questionnaire adaptation and resources:** human-oriented instructions, such as asking participants to circle an answer, are rewritten into a form suitable for LLMs.
  - The code is available at github.com/julianschelb/rupsycho.
  - An appendix lists 61 strategies for rephrasing answer statements, covering tone, sentence structure, metaphors, and JSON/HTML/XML/YAML formats.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{schelb2026r,
  title = {{R.U.Psycho? A Framework for Robust Unified Psychometric Testing of Language Models}},
  author = {Schelb, Julian and Borin, Orr and Garcia, David and Spitz, Andreas},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4d7ofew6usug},
  url = {https://doi.org/10.63317/4d7ofew6usug}
}
```

- [← previous Qur'anic Readings and Modern Arabic Dialects: A Descriptive-Comparative Linguistic Study](https://md.research.elamur.ai/free/en/papers/sabahi-2026-qur-anic-readings-modern-arabic/)
- [next → RadTimeline: Timeline Summarization for Longitudinal Radiological Lung Findings](https://md.research.elamur.ai/free/en/papers/zhou-2026-radtimeline-timeline-summarization-longitudinal-radiological/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/9/)

## References

50 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Gati V. Aher, Rosa I. Arriaga, and Adam Tauman Kalai. 2023. Using large language models to simulate multiple humans and replicate human subject studies. In International Conference on Machine Learning, ICML. [find](https://scholar.google.com/scholar?q=Gati+V%2E+Aher%2C+Rosa+I%2E+Arriaga%2C+and+Adam+Tauman+Kalai%2E+2023%2E+Using+large+language+models+to+simulate+multiple+humans+and+replicate+human+subject+studies%2E+In+International+Conference+on+Machine+Learning)
2. L. B. Allal, A. Lozhkov, E. Bakouch, L. von Werra, and T. Wolf. 2024. Smollm — blazingly fast and remarkably powerful. Unpublished manuscript. Retrieved from Hugging Face Blog. [find](https://scholar.google.com/scholar?q=L%2E+B%2E+Allal%2C+A%2E+Lozhkov%2C+E%2E+Bakouch%2C+L%2E+von+Werra%2C+and+T%2E+Wolf%2E+2024%2E+Smollm+%E2%80%94+blazingly+fast+and+remarkably+powerful%2E+Unpublished+manuscript%2E+Retrieved+from+Hugging+Face+Blog%2E)
3. Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. 2023. Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3):337–351. [find](https://scholar.google.com/scholar?q=Lisa+P+Argyle%2C+Ethan+C+Busby%2C+Nancy+Fulda%2C+Joshua+R+Gubler%2C+Christopher+Rytting%2C+and+David+Wingate%2E+2023%2E+Out+of+one%2C+many%3A+Using+language+models+to+simulate+human+samples%2E+Political+Analysis%2C+31%283%29%3A3) [our card](https://md.research.elamur.ai/free/en/papers/argyle-2022-out-one-many-using-language/)
4. Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, Kelly Marchisio, Max Bartolo, Sebastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Aidan N. Gomez, Phil Blunsom, Marzieh Fadaee, Ahmet Üstün… [find](https://scholar.google.com/scholar?q=Viraat+Aryabumi%2C+John+Dang%2C+Dwarak+Talupuru%2C+Saurabh+Dash%2C+David+Cairuz%2C+Hangyu+Lin%2C+Bharat+Venkitesh%2C+Madeline+Smith%2C+Jon+Ander+Campos%2C+Yi+Chern+Tan%2C+Kelly+Marchisio%2C+Max+Bartolo%2C+Sebastian+Ruder%2C+Ac)
5. Aya 23: Open weight releases to further multilingual progress. CoRR, abs/2405.15032. [find](https://scholar.google.com/scholar?q=Aya+23%3A+Open+weight+releases+to+further+multilingual+progress%2E+CoRR%2C+abs%2F2405%2E15032%2E) [our card](https://md.research.elamur.ai/free/en/papers/aryabumi-2024-aya-23-open-weight-releases/)
6. Christopher A Bail. 2024. Can generative ai improve social science? Proceedings of the National Academy of Sciences, 121(21):e2314021121. [find](https://scholar.google.com/scholar?q=Christopher+A+Bail%2E+2024%2E+Can+generative+ai+improve+social+science%3F+Proceedings+of+the+National+Academy+of+Sciences%2C+121%2821%29%3Ae2314021121%2E)
7. Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondrej Dusek. 2024. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLMs. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, EACL. [find](https://scholar.google.com/scholar?q=Simone+Balloccu%2C+Patr%C3%ADcia+Schmidtov%C3%A1%2C+Mateusz+Lango%2C+and+Ondrej+Dusek%2E+2024%2E+Leak%2C+cheat%2C+repeat%3A+Data+contamination+and+evaluation+malpractices+in+closed%2Dsource+LLMs%2E+In+Proceedings+of+the+18th+Confe) [our card](https://md.research.elamur.ai/free/en/papers/balloccu-2024-leak-cheat-repeat-data-contamination/)
8. A. T. Beck, C. H. Ward, M. Mendelson, J. Mock, and J. Erbaugh. 1961. An inventory for measuring depression. Archives of General Psychiatry, 4(6):561–571. [find](https://scholar.google.com/scholar?q=A%2E+T%2E+Beck%2C+C%2E+H%2E+Ward%2C+M%2E+Mendelson%2C+J%2E+Mock%2C+and+J%2E+Erbaugh%2E+1961%2E+An+inventory+for+measuring+depression%2E+Archives+of+General+Psychiatry%2C+4%286%29%3A561%E2%80%93571%2E)
9. Aaron T. Beck, Robert A. Steer, and Margery G. Carbin. 1988. Psychometric properties of the beck depression inventory: Twenty-five years of evaluation. Clinical Psychology Review, 8(1):77– 100. [find](https://scholar.google.com/scholar?q=Aaron+T%2E+Beck%2C+Robert+A%2E+Steer%2C+and+Margery+G%2E+Carbin%2E+1988%2E+Psychometric+properties+of+the+beck+depression+inventory%3A+Twenty%2Dfive+years+of+evaluation%2E+Clinical+Psychology+Review%2C+8%281%29%3A77%E2%80%93+100%2E)
10. Marcel Binz and Eric Schulz. 2023. Using cognitive psychology to understand gpt-3. Proceedings of the National Academy of Sciences, 120(6):e2218523120. [find](https://scholar.google.com/scholar?q=Marcel+Binz+and+Eric+Schulz%2E+2023%2E+Using+cognitive+psychology+to+understand+gpt%2D3%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+120%286%29%3Ae2218523120%2E)
11. Bojana Bodroža, Bojana M Dinić, and Ljubiša Bojić. 2024. Personality testing of large language models: limited temporal stability, but highlighted prosociality. Royal Society Open Science, 11(10):240180. [find](https://scholar.google.com/scholar?q=Bojana+Bodro%C5%BEa%2C+Bojana+M+Dini%C4%87%2C+and+Ljubi%C5%A1a+Boji%C4%87%2E+2024%2E+Personality+testing+of+large+language+models%3A+limited+temporal+stability%2C+but+highlighted+prosociality%2E+Royal+Society+Open+Science%2C+11%2810%29%3A2401)
12. Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. 2023. Can ai language models replace human participants? Trends in Cognitive Sciences, 27(7):597–600. [find](https://scholar.google.com/scholar?q=Danica+Dillion%2C+Niket+Tandon%2C+Yuling+Gu%2C+and+Kurt+Gray%2E+2023%2E+Can+ai+language+models+replace+human+participants%3F+Trends+in+Cognitive+Sciences%2C+27%287%29%3A597%E2%80%93600%2E)
13. Florian Dorner, Tom Sühr, Samira Samadi, and Augustin Kelava. 2023. Do personality tests generalize to large language models? In Socially Responsible Language Modelling Research, So-LaR Workshop@NeurIPS’23. [find](https://scholar.google.com/scholar?q=Florian+Dorner%2C+Tom+S%C3%BChr%2C+Samira+Samadi%2C+and+Augustin+Kelava%2E+2023%2E+Do+personality+tests+generalize+to+large+language+models%3F+In+Socially+Responsible+Language+Modelling+Research%2C+So%2DLaR+Workshop%40NeurI)
14. Jinhao Duan, Renming Zhang, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Elias Stengel-Eskin, Mohit Bansal, Tianlong Chen, and Kaidi Xu. 2024. Gtbench: Uncovering the strategic reasoning limitations of llms via game-theoretic evaluations. CoRR, abs/2402.12348. — [arXiv:2402.12348](https://arxiv.org/abs/2402.12348) [find](https://scholar.google.com/scholar?q=Jinhao+Duan%2C+Renming+Zhang%2C+James+Diffenderfer%2C+Bhavya+Kailkhura%2C+Lichao+Sun%2C+Elias+Stengel%2DEskin%2C+Mohit+Bansal%2C+Tianlong+Chen%2C+and+Kaidi+Xu%2E+2024%2E+Gtbench%3A+Uncovering+the+strategic+reasoning+limitati)
15. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, and et al. 2024. The llama 3 herd of models. CoRR, abs/2407.21783. [find](https://scholar.google.com/scholar?q=Abhimanyu+Dubey%2C+Abhinav+Jauhri%2C+Abhinav+Pandey%2C+Abhishek+Kadian%2C+Ahmad+Al%2DDahle%2C+and+et+al%2E+2024%2E+The+llama+3+herd+of+models%2E+CoRR%2C+abs%2F2407%2E21783%2E)
16. Salvatore Giorgi, Tingting Liu, Ankit Aich, Kelsey Jane Isman, Garrick Sherman, Zachary Fried, João Sedoc, Lyle Ungar, and Brenda Curtis. 2024. Modeling human subjectivity in LLMs using explicit and implicit human factors in personas. In Findings of the Association for Computational Linguistics: EMN… [find](https://scholar.google.com/scholar?q=Salvatore+Giorgi%2C+Tingting+Liu%2C+Ankit+Aich%2C+Kelsey+Jane+Isman%2C+Garrick+Sherman%2C+Zachary+Fried%2C+Jo%C3%A3o+Sedoc%2C+Lyle+Ungar%2C+and+Brenda+Curtis%2E+2024%2E+Modeling+human+subjectivity+in+LLMs+using+explicit+and+i)
17. Shashank Gupta, Vaishnavi Shrivastava, Ameet Deshpande, Ashwin Kalyan, Peter Clark, Ashish Sabharwal, and Tushar Khot. 2024. Bias runs deep: Implicit reasoning biases in persona-assigned llms. In The Twelfth International Conference on Learning Representations, ICLR. [find](https://scholar.google.com/scholar?q=Shashank+Gupta%2C+Vaishnavi+Shrivastava%2C+Ameet+Deshpande%2C+Ashwin+Kalyan%2C+Peter+Clark%2C+Ashish+Sabharwal%2C+and+Tushar+Khot%2E+2024%2E+Bias+runs+deep%3A+Implicit+reasoning+biases+in+persona%2Dassigned+llms%2E+In+The+)
18. Thilo Hagendorff. 2023. Machine psychology: Investigating emergent capabilities and behavior in large language models using psychological methods. CoRR, abs/2303.13988. — [arXiv:2303.13988](https://arxiv.org/abs/2303.13988) [find](https://scholar.google.com/scholar?q=Thilo+Hagendorff%2E+2023%2E+Machine+psychology%3A+Investigating+emergent+capabilities+and+behavior+in+large+language+models+using+psychological+methods%2E+CoRR%2C+abs%2F2303%2E13988%2E+%E2%80%94+%5BarXiv%3A2303%2E13988%5D%28https%3A%2F%2Far)
19. E. Tory Higgins, Ronald S. Friedman, Robert E. Harlow, Lorraine Chen Idson, Ozlem N. Ayduk, and Amy Taylor. 2001. Achievement orientations from subjective histories of success: Promotion pride versus prevention pride. European Journal of Social Psychology, 31(1):3–23. [find](https://scholar.google.com/scholar?q=E%2E+Tory+Higgins%2C+Ronald+S%2E+Friedman%2C+Robert+E%2E+Harlow%2C+Lorraine+Chen+Idson%2C+Ozlem+N%2E+Ayduk%2C+and+Amy+Taylor%2E+2001%2E+Achievement+orientations+from+subjective+histories+of+success%3A+Promotion+pride+versus+)
20. John J Horton. 2023. Large language models as simulated economic agents: What can we learn from homo silicus? Working Paper 31122, National Bureau of Economic Research. [find](https://scholar.google.com/scholar?q=John+J+Horton%2E+2023%2E+Large+language+models+as+simulated+economic+agents%3A+What+can+we+learn+from+homo+silicus%3F+Working+Paper+31122%2C+National+Bureau+of+Economic+Research%2E)
21. Tiancheng Hu and Nigel Collier. 2024. Quantifying the persona effect in LLM simulations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ACL. [find](https://scholar.google.com/scholar?q=Tiancheng+Hu+and+Nigel+Collier%2E+2024%2E+Quantifying+the+persona+effect+in+LLM+simulations%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+ACL%2E)
22. Tiancheng Hu, Yara Kyrychenko, Steve Rathje, Nigel Collier, Sander van der Linden, and Jon Roozenbeek. 2025. Generative language models exhibit social identity biases. Nature Computational Science, 5(1):65–75. [find](https://scholar.google.com/scholar?q=Tiancheng+Hu%2C+Yara+Kyrychenko%2C+Steve+Rathje%2C+Nigel+Collier%2C+Sander+van+der+Linden%2C+and+Jon+Roozenbeek%2E+2025%2E+Generative+language+models+exhibit+social+identity+biases%2E+Nature+Computational+Science%2C+5%28)
23. Jen-tse Huang, Wenxiang Jiao, Man Ho Lam, Eric John Li, Wenxuan Wang, and Michael Lyu. 2024. On the reliability of psychological scales on large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Jen%2Dtse+Huang%2C+Wenxiang+Jiao%2C+Man+Ho+Lam%2C+Eric+John+Li%2C+Wenxuan+Wang%2C+and+Michael+Lyu%2E+2024%2E+On+the+reliability+of+psychological+scales+on+large+language+models%2E+In+Proceedings+of+the+2024+Conference+)
24. Silke Husse and Andreas Spitz. 2022. Mind your bias: A critical review of bias detection methods for contextual language models. In Findings of the Association for Computational Linguistics: EMNLP. [find](https://scholar.google.com/scholar?q=Silke+Husse+and+Andreas+Spitz%2E+2022%2E+Mind+your+bias%3A+A+critical+review+of+bias+detection+methods+for+contextual+language+models%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP%2E)
25. Maurice Jakesch, Jeffrey T Hancock, and Mor Naaman. 2023. Human heuristics for ai-generated language are flawed. Proceedings of the National Academy of Sciences, 120(11):e2208839120. [find](https://scholar.google.com/scholar?q=Maurice+Jakesch%2C+Jeffrey+T+Hancock%2C+and+Mor+Naaman%2E+2023%2E+Human+heuristics+for+ai%2Dgenerated+language+are+flawed%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+120%2811%29%3Ae2208839120%2E)
26. Oliver P John, Eileen M Donahue, and Robert L Kentle. 1991. Big Five Inventory (BFI). APA PsycTests. [find](https://scholar.google.com/scholar?q=Oliver+P+John%2C+Eileen+M+Donahue%2C+and+Robert+L+Kentle%2E+1991%2E+Big+Five+Inventory+%28BFI%29%2E+APA+PsycTests%2E)
27. Lea Löhn, Niklas Kiehne, Alexander Ljapunov, and Wolf-Tilo Balke. 2024. Is machine psychology here? on requirements for using human psychological tests on large language models. In Proceedings of the 17th International Natural Language Generation Conference. [find](https://scholar.google.com/scholar?q=Lea+L%C3%B6hn%2C+Niklas+Kiehne%2C+Alexander+Ljapunov%2C+and+Wolf%2DTilo+Balke%2E+2024%2E+Is+machine+psychology+here%3F+on+requirements+for+using+human+psychological+tests+on+large+language+models%2E+In+Proceedings+of+the+)
28. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL. [find](https://scholar.google.com/scholar?q=Yao+Lu%2C+Max+Bartolo%2C+Alastair+Moore%2C+Sebastian+Riedel%2C+and+Pontus+Stenetorp%2E+2022%2E+Fantastically+ordered+prompts+and+where+to+find+them%3A+Overcoming+few%2Dshot+prompt+order+sensitivity%2E+In+Proceedings+of) [our card](https://md.research.elamur.ai/free/en/papers/lu-2022-fantastically-ordered-prompts-where-find/)
29. Inbal Magar and Roy Schwartz. 2022. Data contamination: From memorization to exploitation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL. [find](https://scholar.google.com/scholar?q=Inbal+Magar+and+Roy+Schwartz%2E+2022%2E+Data+contamination%3A+From+memorization+to+exploitation%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+ACL%2E)
30. Francesco Manigrasso, Stefan F. Schouten, Lia Morra, and Peter Bloem. 2024. Probing llms for logical reasoning. In 18th International Conference on Neural-Symbolic Learning and Reasoning, NeSy, pages 257–278. [find](https://scholar.google.com/scholar?q=Francesco+Manigrasso%2C+Stefan+F%2E+Schouten%2C+Lia+Morra%2C+and+Peter+Bloem%2E+2024%2E+Probing+llms+for+logical+reasoning%2E+In+18th+International+Conference+on+Neural%2DSymbolic+Learning+and+Reasoning%2C+NeSy%2C+pages+)
31. Marilù Miotto, Nicola Rossberg, and Bennett Kleinberg. 2022. Who is GPT-3? an exploration of personality, values and demographics. In Proceedings of the Fifth Workshop on Natural Language Processing and Computational Social Science (NLP+CSS). [find](https://scholar.google.com/scholar?q=Maril%C3%B9+Miotto%2C+Nicola+Rossberg%2C+and+Bennett+Kleinberg%2E+2022%2E+Who+is+GPT%2D3%3F+an+exploration+of+personality%2C+values+and+demographics%2E+In+Proceedings+of+the+Fifth+Workshop+on+Natural+Language+Processing+a)
32. OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2023%2E+GPT%2D4+technical+report%2E+CoRR%2C+abs%2F2303%2E08774%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
33. Max Pellert, Clemens M Lechner, Claudia Wagner, Beatrice Rammstedt, and Markus Strohmaier. 2024. AI psychometrics: Assessing the psychological profiles of large language models through psychometric inventories. Perspectives on Psychological Science, 19(5):808–826. [find](https://scholar.google.com/scholar?q=Max+Pellert%2C+Clemens+M+Lechner%2C+Claudia+Wagner%2C+Beatrice+Rammstedt%2C+and+Markus+Strohmaier%2E+2024%2E+AI+psychometrics%3A+Assessing+the+psychological+profiles+of+large+language+models+through+psychometric+in)
34. Nikolay B. Petrov, Gregory Serapio-García, and Jason Rentfrow. 2024. Limited ability of llms to simulate human psychological behaviours: a psychometric analysis. CoRR, abs/2405.07248. — [arXiv:2405.07248](https://arxiv.org/abs/2405.07248) [find](https://scholar.google.com/scholar?q=Nikolay+B%2E+Petrov%2C+Gregory+Serapio%2DGarc%C3%ADa%2C+and+Jason+Rentfrow%2E+2024%2E+Limited+ability+of+llms+to+simulate+human+psychological+behaviours%3A+a+psychometric+analysis%2E+CoRR%2C+abs%2F2405%2E07248%2E+%E2%80%94+%5BarXiv%3A2405%2E07)
35. Pouya Pezeshkpour and Estevam Hruschka. 2024. Large language models sensitivity to the order of options in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024. [find](https://scholar.google.com/scholar?q=Pouya+Pezeshkpour+and+Estevam+Hruschka%2E+2024%2E+Large+language+models+sensitivity+to+the+order+of+options+in+multiple%2Dchoice+questions%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+NAAC) [our card](https://md.research.elamur.ai/free/en/papers/pezeshkpour-2023-large-language-models-sensitivity-order/)
36. ProtectAI.com. 2024. Fine-tuned distilroberta-base for rejection in the output detection. [find](https://scholar.google.com/scholar?q=ProtectAI%2Ecom%2E+2024%2E+Fine%2Dtuned+distilroberta%2Dbase+for+rejection+in+the+output+detection%2E)
37. Yuanyi Ren, Haoran Ye, Hanjun Fang, Xin Zhang, and Guojie Song. 2024. Valuebench: Towards comprehensively evaluating value orientations and understanding of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ACL. [find](https://scholar.google.com/scholar?q=Yuanyi+Ren%2C+Haoran+Ye%2C+Hanjun+Fang%2C+Xin+Zhang%2C+and+Guojie+Song%2E+2024%2E+Valuebench%3A+Towards+comprehensively+evaluating+value+orientations+and+understanding+of+large+language+models%2E+In+Proceedings+of+th)
38. Paul Röttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Kirk, Hinrich Schütze, and Dirk Hovy. 2024. Political compass or spinning arrow? towards more meaningful evaluations for values and opinions in large language models. In Proceedings of the 62nd Annual Meeting of the Association… [find](https://scholar.google.com/scholar?q=Paul+R%C3%B6ttger%2C+Valentin+Hofmann%2C+Valentina+Pyatkin%2C+Musashi+Hinck%2C+Hannah+Kirk%2C+Hinrich+Sch%C3%BCtze%2C+and+Dirk+Hovy%2E+2024%2E+Political+compass+or+spinning+arrow%3F+towards+more+meaningful+evaluations+for+values) [our card](https://md.research.elamur.ai/free/en/papers/rottger-2024-political-compass-or-spinning-arrow/)
39. Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. 2023. Can ai-generated text be reliably detected? CoRR, abs/2303.11156. [find](https://scholar.google.com/scholar?q=Vinu+Sankar+Sadasivan%2C+Aounon+Kumar%2C+Sriram+Balasubramanian%2C+Wenxiao+Wang%2C+and+Soheil+Feizi%2E+2023%2E+Can+ai%2Dgenerated+text+be+reliably+detected%3F+CoRR%2C+abs%2F2303%2E11156%2E) [our card](https://md.research.elamur.ai/free/en/papers/sadasivan-2023-ai-generated-text-reliably-detected/)
40. Zach C Schudson and Sari M van Anders. 2022. Gender/sex diversity beliefs: Scale construction, validation, and links to prejudice. Group Processes & Intergroup Relations, 25(4):1011– 1036. [find](https://scholar.google.com/scholar?q=Zach+C+Schudson+and+Sari+M+van+Anders%2E+2022%2E+Gender%2Fsex+diversity+beliefs%3A+Scale+construction%2C+validation%2C+and+links+to+prejudice%2E+Group+Processes+%26+Intergroup+Relations%2C+25%284%29%3A1011%E2%80%93+1036%2E)
41. Bangzhao Shu, Lechen Zhang, Minje Choi, Lavinia Dunagan, Lajanugen Logeswaran, Moontae Lee, Dallas Card, and David Jurgens. 2024. You don‘t need a personality test to know these models are unreliable: Assessing the reliability of large language models on psychometric instruments. In Proceedings of t… [find](https://scholar.google.com/scholar?q=Bangzhao+Shu%2C+Lechen+Zhang%2C+Minje+Choi%2C+Lavinia+Dunagan%2C+Lajanugen+Logeswaran%2C+Moontae+Lee%2C+Dallas+Card%2C+and+David+Jurgens%2E+2024%2E+You+don%E2%80%98t+need+a+personality+test+to+know+these+models+are+unreliable%3A)
42. Judith Jarvis Thomson. 1984. The trolley problem. Yale LJ, 94:1395. [find](https://scholar.google.com/scholar?q=Judith+Jarvis+Thomson%2E+1984%2E+The+trolley+problem%2E+Yale+LJ%2C+94%3A1395%2E)
43. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, and et al. 2023. Zephyr: Direct distillation of LM alignment. CoRR, abs/2310.16944. [find](https://scholar.google.com/scholar?q=Lewis+Tunstall%2C+Edward+Beeching%2C+Nathan+Lambert%2C+Nazneen+Rajani%2C+Kashif+Rasul%2C+and+et+al%2E+2023%2E+Zephyr%3A+Direct+distillation+of+LM+alignment%2E+CoRR%2C+abs%2F2310%2E16944%2E)
44. Tomer D. Ullman. 2023. Large language models fail on trivial alterations to theory-of-mind tasks. CoRR, abs/2302.08399. [find](https://scholar.google.com/scholar?q=Tomer+D%2E+Ullman%2E+2023%2E+Large+language+models+fail+on+trivial+alterations+to+theory%2Dof%2Dmind+tasks%2E+CoRR%2C+abs%2F2302%2E08399%2E)
45. Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. 2023. Artificial artificial artificial intelligence: Crowd workers widely use large language models for text production tasks. CoRR, abs/2306.07899. [find](https://scholar.google.com/scholar?q=Veniamin+Veselovsky%2C+Manoel+Horta+Ribeiro%2C+and+Robert+West%2E+2023%2E+Artificial+artificial+artificial+intelligence%3A+Crowd+workers+widely+use+large+language+models+for+text+production+tasks%2E+CoRR%2C+abs%2F230)
46. Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. 2024. "My Answer is C": First-token probabilities do not match text answers in instruction-tuned language models. In Findings of the Association for Computational Linguistics, ACL. [find](https://scholar.google.com/scholar?q=Xinpeng+Wang%2C+Bolei+Ma%2C+Chengzhi+Hu%2C+Leon+Weber%2DGenzel%2C+Paul+R%C3%B6ttger%2C+Frauke+Kreuter%2C+Dirk+Hovy%2C+and+Barbara+Plank%2E+2024%2E+%22My+Answer+is+C%22%3A+First%2Dtoken+probabilities+do+not+match+text+answers+in+instr)
47. Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with chatgpt. CoRR, abs/2302.11382. [find](https://scholar.google.com/scholar?q=Jules+White%2C+Quchen+Fu%2C+Sam+Hays%2C+Michael+Sandborn%2C+Carlos+Olea%2C+Henry+Gilbert%2C+Ashraf+Elnashar%2C+Jesse+Spencer%2DSmith%2C+and+Douglas+C%2E+Schmidt%2E+2023%2E+A+prompt+pattern+catalog+to+enhance+prompt+engineeri) [our card](https://md.research.elamur.ai/free/en/papers/white-2023-prompt-pattern-catalog-enhance-prompt/)
48. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, and et al. 2024. Qwen2.5 technical report. CoRR, abs/2412.15115. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+and+et+al%2E+2024%2E+Qwen2%2E5+technical+report%2E+CoRR%2C+abs%2F2412%2E15115%2E)
49. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations, ICLR. [find](https://scholar.google.com/scholar?q=Chujie+Zheng%2C+Hao+Zhou%2C+Fandong+Meng%2C+Jie+Zhou%2C+and+Minlie+Huang%2E+2024%2E+Large+language+models+are+not+robust+multiple+choice+selectors%2E+In+The+Twelfth+International+Conference+on+Learning+Representati)
50. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2024. Can large language models transform computational social science? Comput. Linguistics, 50(1):237– 291. [find](https://scholar.google.com/scholar?q=Caleb+Ziems%2C+William+Held%2C+Omar+Shaikh%2C+Jiaao+Chen%2C+Zhehao+Zhang%2C+and+Diyi+Yang%2E+2024%2E+Can+large+language+models+transform+computational+social+science%3F+Comput%2E+Linguistics%2C+50%281%29%3A237%E2%80%93+291%2E) [our card](https://md.research.elamur.ai/free/en/papers/ziems-2023-large-language-models-transform-computational/)