---
title: "AIn't Nothing But a Survey? Using Large Language Models for"
description: "The paper asks whether generative LLMs can code open-ended survey answers outside the easy setting of earlier studies… 62 references."
authors: ["Leah von der Heyde", "Anna-Carolina Haensch", "Bernd Wei\\ss", "Jessica Daikeler"]
year: 2025
venue: "arXiv:2506.14634 · Survey Research Methods (2025)"
source: "https://arxiv.org/abs/2506.14634"
doi: "10.18148/srm/2025.v19i4.8568"
arxiv: "2506.14634"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/heyde-2025-ain-t-nothing-but-survey/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/heyde-2025-ain-t-nothing-but-survey/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/14/) / AIn't Nothing But a Survey? Using Large Language Models for Coding German Open-Ended Survey Responses on Survey Motivation

# AIn't Nothing But a Survey? Using Large Language Models for Coding German Open-Ended Survey Responses on Survey Motivation

**Authors:** Leah von der Heyde, [Anna-Carolina Haensch](https://md.research.elamur.ai/free/en/authors/anna-carolina-haensch/), Bernd Wei\\ss and Jessica Daikeler · **Year:** 2025 · **Venue:** arXiv:2506.14634 · Survey Research Methods (2025) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://doi.org/10.18148/srm/2025.v19i4.8568) · [PDF](https://arxiv.org/pdf/2506.14634) · [DOI 10.18148/srm/2025.v19i4.8568](https://doi.org/10.18148/srm/2025.v19i4.8568) · [arXiv 2506.14634](https://arxiv.org/abs/2506.14634) · [OpenAlex](https://openalex.org/W4415332850)

**Citations:** 1 citations (OpenAlex, 2026-10-14)

## Summary
The paper asks whether generative LLMs can code open-ended survey answers outside the easy setting of earlier studies, which used English text on familiar political topics. The authors use German free-text answers about why people take part in surveys, coded into a detailed, predefined category scheme. They test three multilingual models, GPT-4o, Llama 3.2 and Mistral NeMo, under several strategies: zero-shot with and without category descriptions, few-shot prompting with example classifications, and fine-tuning on human-coded data. All outputs are scored against codings by human experts, and the authors also look at the reliability of repeated runs.

Results vary widely by model. Whether a prompting strategy helps depends on which model receives it. Only the fine-tuned GPT model reached acceptable quality. Without fine-tuning, the models handled some categories much better than others, so the overall share of answers in each category drifted away from the human distribution.

The study matters because it shows that LLM coding results from simple English tasks do not carry over automatically. Every model and prompt combination has to be validated, and human-coded data is still needed.

## Key points
- The task is German open-ended answers on motivation for survey participation, coded into a complex scheme. Human expert codes serve as the gold standard.
- The models compared are GPT-4o (proprietary), Llama 3.2 and Mistral NeMo (open). These were the strongest multilingual members of their families at the time.
- Four approaches were tested:
  - zero-shot without category descriptions,
  - zero-shot with category descriptions,
  - few-shot prompting,
  - fine-tuning.
- Only fine-tuned GPT performed satisfactorily. Without fine-tuning, quality was below both earlier English studies (e.g. Mellon et al. on the British Election Study, 50 categories) and classic supervised classifiers.
- Uneven accuracy across categories changed the category proportions. This would bias any substantive analysis built on the coded data.

## Interesting ideas
- Beyond per-item accuracy, compare the predicted label distribution with the gold distribution, because a classifier can look acceptable while systematically shifting category shares.
- The authors propose splitting a large label scheme into two steps: first ask, category by category, whether a label fits, then choose the best among the shortlisted ones. This keeps each prompt short and leaves room for more examples per category.
- The finding that prompting gains depend on the specific model warns against reusing a single prompt recipe across models.
- Label uncertainty could be estimated three ways: from token probabilities, from a majority vote over repeated sampled runs, or by asking the model for its confidence.
- High human–model agreement can hide errors that the humans make consistently and the model copies, so agreement alone is not proof of correctness.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Evaluation of silver frame or role labels could report label-frequency drift against gold treebanks, catching a teacher that overuses some labels.
- **I2** (frames): When an LLM teacher assigns VerbNet classes or speech acts, a filter-then-pick chain could replace one huge prompt listing every class.
- **I3** (learn): Each LLM teacher tier should get its own prompt validation on a held-out gold set, not a shared recipe.
- **I4** (learn): Majority over repeated teacher calls could join the witness-parser votes, so low-agreement items are routed to repair or excluded from silver data.
- **I5** (learn): This supports adding negative controls and independent gold checks, rather than trusting agreement between teachers and annotators.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Ludwig-Maximilians-Universität München](https://md.research.elamur.ai/free/en/universities/ludwig-maximilians-universitat-munchen/) — University, Munich, Germany · 107 works · 56,478 citations
- [GESIS - Leibniz Institute for the Social Sciences](https://md.research.elamur.ai/free/en/labs/gesis-leibniz-institute-for-the-social-sciences/) — Research institute, Mannheim, Germany · 10 works · 5 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 577 works · [🤝 General Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/general-social-sciences/) · 227 works

## Cite

Cite as: Leah von der Heyde, Anna-Carolina Haensch, Bernd Wei\\ss and Jessica Daikeler. 2025. AIn't Nothing But a Survey? Using Large Language Models for Coding German Open-Ended Survey Responses on Survey Motivation. arXiv:2506.14634 · Survey Research Methods (2025).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{heyde2025ain,
  title = {{AIn't Nothing But a Survey? Using Large Language Models for Coding German Open-Ended Survey Responses on Survey Motivation}},
  author = {Leah {von der Heyde}, Anna-Carolina Haensch, Bernd Weiss and Jessica Daikeler},
  year = {2025},
  doi = {10.18148/srm/2025.v19i4.8568},
  eprint = {2506.14634},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18148/srm/2025.v19i4.8568}
}
```

- [← previous Advancing Arabic Reverse Dictionary Systems: A Transformer-Based Approach with Dataset Construction Guidelines](https://md.research.elamur.ai/free/en/papers/sibaee-2025-advancing-arabic-reverse-dictionary-systems/)
- [next → ALADAN at IWSLT25 Low-resource Arabic Dialectal Speech Translation Task](https://md.research.elamur.ai/free/en/papers/jon-2025-aladan-iwslt25-low-resource-arabic/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/14/)

## References

62 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ahnert, G., Pellert, M., Garcia, D., & Strohmaier, M. (2025). Extracting Affect Aggregates from Longitudinal Social Media Data with Temporal Adapters for Large Language Models. Proceedings of the International AAAI Conference on Web and Social Media, 19(1), 15–36. https://doi.org/10.1609/icwsm.v19i1.35801 [↗](https://doi.org/10.1609/icwsm.v19i1.35801)
2. Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023). [Out of One, Many: Using Language Models to Simulate Human Samples](https://md.research.elamur.ai/free/en/papers/argyle-2022-out-one-many-using-language/). Political Analysis, 1–15. https://doi.org/10.1017/pan.2023.2 [↗](https://doi.org/10.1017/pan.2023.2)
3. Barrie, C., Palmer, A., & Spirling, A. (2024). Replication for Language Models. [https://drive.google.com/file/d/1wNDIkMZfAGoh4Oaojrgll9SPg3eT-YXz/view](https://drive.google.com/file/d/1wNDIkMZfAGoh4Oaojrgll9SPg3eT-YXz/view) [↗](https://drive.google.com/file/d/1wNDIkMZfAGoh4Oaojrgll9SPg3eT-YXz/view)
4. Bosley, M., Jacobs-Harukawa, M., Licht, H., & Hoyle, A. (2023). Do we still need BERT in the age of GPT? Comparing the benefits of domain-adaptation and in-context-learning approaches to using LLMs for Political Science Research. [find](https://scholar.google.com/scholar?q=Bosley%2C+M%2E%2C+Jacobs%2DHarukawa%2C+M%2E%2C+Licht%2C+H%2E%2C+%26+Hoyle%2C+A%2E+%282023%29%2E+Do+we+still+need+BERT+in+the+age+of+GPT%3F+Comparing+the+benefits+of+domain%2Dadaptation+and+in%2Dcontext%2Dlearning+approaches+to+using+LLMs+fo)
5. Bosnjak, M., Dannwolf, T., Enderle, T., Schaurer, I., Struminskaya, B., Tanner, A., & Weyandt, K. W. (2018). Establishing an Open Probability-Based Mixed-Mode Panel of the General Population in Germany: The GESIS Panel. Social Science Computer Review, 36(1), 103–115. https://doi.org/10.1177/0894439317697949 [↗](https://doi.org/10.1177/0894439317697949)
6. Brand, J., Israeli, A., & Ngwe, D. (2023). Using GPT for Market Research (SSRN Scholarly Paper No. 4395751). [https://papers.ssrn.com/abstract=4395751](https://papers.ssrn.com/abstract=4395751) Casper da Costa-Luis, Stephen Karl Larroque, Kyle Altendorf, Hadrien Mary, richardsheridan, Mikhail Korobov, Noam Yorav-Raphael, Ivan Ivanov, Marcel Bargull, Nishant Rodrigues, Shawn, Mikhail Dektyarev, Michał Górny, mjstevens777, Matthew D. Pagel, Martin Zugnoni, JC, CrazyPython, Charles Newey, … Jack McCracken. [↗](https://papers.ssrn.com/abstract=4395751)
7. (2024). tqdm: A fast, Extensible Progress Bar for Python and CLI (Version v4.67.1) [Computer software]. Zenodo. https://doi.org/10.5281/ZENODO.595120 [↗](https://doi.org/10.5281/zenodo.595120)
8. Cerina, R., & Duch, R. (2023). Artificially Intelligent Opinion Polling (No. arXiv:2309.06029). arXiv. http://arxiv.org/abs/2309.06029 [↗](https://arxiv.org/abs/2309.06029)
9. Cuevas, A., Brown, E. M., Scurrell, J. V., Entenmann, J., & Daepp, M. I. G. (2023). Automated Interviewer or Augmented Survey? Collecting Social Data with Large Language Models (No. arXiv:2309.10187). arXiv. http://arxiv.org/abs/2309.10187 [↗](https://arxiv.org/abs/2309.10187)
10. Durmus, E., Nguyen, K., Liao, T., Schiefer, N., Askell, A., Bakhtin, A., Chen, C., Hatfield-Dodds, Z., Hernandez, D., Joseph, N., Lovitt, L., McCandlish, S., Sikder, O., Tamkin, A., Thamkul, J., Kaplan, J., Clark, J., & Ganguli, D. (2024). [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://md.research.elamur.ai/free/en/papers/durmus-2023-towards-measuring-representation-subjective-global/). First Conference on Language Modeling. https://openreview.net/forum?id=zl16jLb91v [↗](https://doi.org/10.48550/arxiv.2306.16388)
11. Fröhling, L., Demartini, G., & Assenmacher, D. (2024). Personas with Attitudes: Controlling LLMs for Diverse Data Annotation (No. arXiv:2410.11745). arXiv. http://arxiv.org/abs/2410.11745 [↗](https://arxiv.org/abs/2410.11745)
12. Gamer, M., Lemon, J., & Singh, I. F. P. (2019). irr: Various Coefficients of Interrater Reliability and Agreement. [https://CRAN.R-project.org/package=irr](https://CRAN.R-project.org/package=irr) [↗](https://CRAN.R-project.org/package=irr)
13. Gentzkow, M., Kelly, B., & Taddy, M. (2019). Text as Data. Journal of Economic Literature, 57(3), 535–574. https://doi.org/10.1257/jel.20181020 [↗](https://doi.org/10.1257/jel.20181020)
14. GESIS (2024). GESIS Panel - Extended Edition. GESIS, Cologne. ZA5664 Data file Version 52.0.0. https://doi.org/10.4232/1.14284 [↗](https://doi.org/10.4232/1.14284)
15. Grimmer, J., & Stewart, B. M. (2013). Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts. Political Analysis, 21(3), 267–297. https://doi.org/10.1093/pan/mps028 [↗](https://doi.org/10.1093/pan/mps028)
16. Gweon, H., & Schonlau, M. (2024). Automated Classification for Open-Ended Questions with BERT. Journal of Survey Statistics and Methodology, 12(2), 493–504. https://doi.org/10.1093/jssam/smad015 [↗](https://doi.org/10.1093/jssam/smad015)
17. Götz, F. M., Maertens, R., Loomba, S., & Van Der Linden, S. (2023). Let the algorithm speak: How to use neural networks for automatic item generation in psychological scale development. Psychological Methods. https://doi.org/10.1037/met0000540 [↗](https://doi.org/10.1037/met0000540)
18. Haensch, A.-C., Weiß, B., Steins, Patricia, Chyvra, Priscilla, & Bitz, Katja. (2022). The semi-automatic classification of an open-ended question on panel survey motivation and its application in attrition analysis. Frontiers in Big Data, 5:880554. https://doi.org/10.3389/fdata.2022.880554 [↗](https://doi.org/10.3389/fdata.2022.880554)
19. Halterman, A., & Keith, K. A. (2024). Codebook LLMs: Adapting Political Science Codebooks for LLM Use and Adapting LLMs to Follow Codebooks (No. arXiv:2407.10747). arXiv. http://arxiv.org/abs/2407.10747 [↗](https://arxiv.org/abs/2407.10747)
20. Hand, D. J., Christen, P., & Ziyad, S. (2024). Selecting a classification performance measure: Matching the measure to the problem (No. arXiv:2409.12391). arXiv. http://arxiv.org/abs/2409.12391 [↗](https://arxiv.org/abs/2409.12391)
21. Heseltine, M., & Clemm von Hohenberg, B. (2024). Large language models as a substitute for human experts in annotating political text. Research & Politics, 11(1), 20531680241236239. https://doi.org/10.1177/20531680241236239 [↗](https://doi.org/10.1177/20531680241236239)
22. Joachims, T. (2001). A statistical learning model of text classification for support vector machines. Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 128–136. https://doi.org/10.1145/383952.383974 [↗](https://doi.org/10.1145/383952.383974)
23. Johnson, R. L., Pistilli, G., Menédez-González, N., Duran, L. D. D., Panai, E., Kalpokiene, J., & Bertulfo, D. J. (2022). The Ghost in the Machine has an American accent: Value conflict in GPT-3 (No. arXiv:2203.07785). arXiv. http://arxiv.org/abs/2203.07785 [↗](https://arxiv.org/abs/2203.07785)
24. Kim, J., & Lee, B. (2023). AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Prediction (No. arXiv:2305.09620). arXiv. http://arxiv.org/abs/2305.09620 [↗](https://arxiv.org/abs/2305.09620)
25. Krosnick, J. A., & Presser, S. (2010). Question and Questionnaire Design. In P. V. Marsden & J. D. Wright (Eds.), Handbook of Survey Research. 2nd edition (pp. 263–314). Emerald. [find](https://scholar.google.com/scholar?q=Krosnick%2C+J%2E+A%2E%2C+%26+Presser%2C+S%2E+%282010%29%2E+Question+and+Questionnaire+Design%2E+In+P%2E+V%2E+Marsden+%26+J%2E+D%2E+Wright+%28Eds%2E%29%2C+Handbook+of+Survey+Research%2E+2nd+edition+%28pp%2E+263%E2%80%93314%29%2E+Emerald%2E)
26. Kuhn, M. (2008). Building Predictive Models in R Using the caret Package. Journal of Statistical Software, 28(5), 1–26. https://doi.org/10.18637/jss.v028.i05 [↗](https://doi.org/10.18637/jss.v028.i05)
27. Landesvatter, C. (2024). Methods for the classification of data from open-ended questions in surveys [University of Mannheim]. [https://madoc.bib.uni-mannheim.de/67089/](https://madoc.bib.uni-mannheim.de/67089/) [↗](https://madoc.bib.uni-mannheim.de/67089/)
28. Lebrun, B., Temtsin, S., Vonasch, A., & Bartneck, C. (2024). Detecting the corruption of online questionnaires by artificial intelligence. Frontiers in Robotics and AI, Volume 10-2023. https://doi.org/10.3389/frobt.2023.1277635 [↗](https://doi.org/10.3389/frobt.2023.1277635)
29. Li, B., Haider, S., & Callison-Burch, C. (2024). This Land is Your, My Land: Evaluating Geopolitical Bias in Language Models through Territorial Disputes. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 3855–3871. https://doi.org/10.18653/v1/2024.naacl-long.213 [↗](https://doi.org/10.18653/v1/2024.naacl-long.213)
30. McCoy, R. T., Yao, S., Friedman, D., Hardy, M., & Griffiths, T. L. (2023). Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve (No. arXiv:2309.13638). arXiv. http://arxiv.org/abs/2309.13638 [↗](https://arxiv.org/abs/2309.13638)
31. McKinney, W. (2010). Data Structures for Statistical Computing in Python. In S. van der Walt & J. Millman (Eds.), Proceedings of the 9th Python in Science Conference (pp. 56–61). https://doi.org/10.25080/Majora-92bf1922-00a [↗](https://doi.org/10.25080/majora-92bf1922-00a)
32. Meidinger, M., & Aßenmacher, M. (2021). A New Benchmark for NLP in Social Sciences: Evaluating the Usefulness of Pre-trained Language Models for Classifying Open-ended Survey Responses: Proceedings of the 13th International Conference on Agents and Artificial Intelligence, 866–873. https://doi.org/10.5220/0010255108660873 [↗](https://doi.org/10.5220/0010255108660873)
33. Mellon, J., Bailey, J., Scott, R., Breckwoldt, J., Miori, M., & Schmedeman, P. (2024). Do AIs know what the most important issue is? Using language models to code open-text social survey responses at scale. Research & Politics, 11(1). https://doi.org/10.1177/20531680241231468 [↗](https://doi.org/10.1177/20531680241231468)
34. Meta. (2024a). Llama 3.2: Revolutionizing edge AI and vision with open, customizable models. Meta. [https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/](https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/) [↗](https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/)
35. Meta. (2024b). Llama-3.2-3B-Instruct. HuggingFace. [https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct) [↗](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct)
36. MistralAI. (2024a). Mistral NeMo. Mistral AI. [https://mistral.ai/news/mistral-nemo/](https://mistral.ai/news/mistral-nemo/) [↗](https://mistral.ai/news/mistral-nemo/)
37. MistralAI. (2024b). Mistral-Nemo-Instruct-2407. HuggingFace. [https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407](https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407) [↗](https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407)
38. Ooi, H., httr development team, Littlefield, T., Holden, S., Stone, C., & Microsoft. (2019). AzureAuth: Authentication Services for Azure Active Directory (p. 1.3.3) [Computer software]. 10.32614/CRAN.package.AzureAuth [↗](https://doi.org/10.32614/cran.package.azureauth)
39. OpenAI. (2024a). Hello GPT-4o. OpenAI. [https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/) [↗](https://openai.com/index/hello-gpt-4o/)
40. OpenAI. (2024b). [GPT-4o System Card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). [https://cdn.openai.com/gpt-4o-system-card.pdf](https://cdn.openai.com/gpt-4o-system-card.pdf) [↗](https://cdn.openai.com/gpt-4o-system-card.pdf)
41. Ornstein, J. T., Blasingame, E. N., & Truscott, J. S. (2024). How to Train Your Stochastic Parrot: Large Language Models for Political Texts. [https://joeornstein.github.io/publications/ornstein-blasingame-truscott.pdf](https://joeornstein.github.io/publications/ornstein-blasingame-truscott.pdf) [↗](https://joeornstein.github.io/publications/ornstein-blasingame-truscott.pdf)
42. Palmer, A., Smith, N. A., & Spirling, A. (2023). Using proprietary language models in academic research requires explicit justification. Nature Computational Science, 4(1), 2–3. https://doi.org/10.1038/s43588-023-00585-1 [↗](https://doi.org/10.1038/s43588-023-00585-1)
43. Pangakis, N., Wolken, S., & Fasching, N. (2023). [Automated Annotation with Generative AI Requires Validation](https://md.research.elamur.ai/free/en/papers/pangakis-2023-automated-annotation-generative-ai-requires/) (No. arXiv:2306.00176). arXiv. https://doi.org/10.48550/arXiv.2306.00176 Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., … Chintala, S. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32 (pp. 8024–8035). Curran Associates, Inc. http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-de ep-learning-library.pdf Pezeshkpour, P., & Hruschka, E. (2024). Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions. In K. Duh, H. Gomez, & S. Bethard (Eds.), Findings of the Association for Computational Linguistics: NAACL 2024 (pp. 2006–2017). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.findings-naacl.130 [↗](https://doi.org/10.48550/arxiv.2306.00176)
44. Pham, C., Hoyle, A., Sun, S., Resnik, P., & Iyyer, M. (2024). [TopicGPT: A Prompt-based Topic Modeling Framework](https://md.research.elamur.ai/free/en/papers/pham-2024-topicgpt-prompt-based-topic-modeling/). In K. Duh, H. Gomez, & S. Bethard (Eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (pp. 2956–2984). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.naacl-long.164 [↗](https://doi.org/10.18653/v1/2024.naacl-long.164)
45. Porst, R., & von Briel, C. (1995). Wären Sie vielleicht bereit, sich gegebenenfalls noch einmal befragen zu lassen? Oder: Gründe für die Teilnahme an Panelbefragungen. (No. 95/04; ZUMA Arbeitsbericht). Zentrum für Umfragen, Methoden und Analysen. [https://www.gesis.org/fileadmin/upload/forschung/publikationen/gesis_reihen/zuma_a](https://www.gesis.org/fileadmin/upload/forschung/publikationen/gesis_reihen/zuma_a) rbeitsberichte/95_04.pdf R Core Team. (2024, July 2). R: The R Project for Statistical Computing. https://www.r-project.org/Rytting, C. M., Sorensen, T., Argyle, L., Busby, E., Fulda, N., Gubler, J., & Wingate, D. [↗](https://www.gesis.org/fileadmin/upload/forschung/publikationen/gesis_reihen/zuma_a)
46. (2023). Towards Coding Social Science Datasets with Language Models (No. arXiv:2306.02177). arXiv. http://arxiv.org/abs/2306.02177 [↗](https://arxiv.org/abs/2306.02177)
47. Schierholz, M., & Schonlau, M. (2021). Machine Learning for Occupation Coding—A Comparison Study. Journal of Survey Statistics and Methodology, 9(5), 1013–1034. https://doi.org/10.1093/jssam/smaa023 [↗](https://doi.org/10.1093/jssam/smaa023)
48. Schonlau, M., & Couper, M. P. (2016). Semi-automated categorization of open-ended questions. Survey Research Methods, Vol 10, 143-152 Pages. https://doi.org/10.18148/SRM/2016.V10I2.6213 [↗](https://doi.org/10.18148/srm/2016.v10i2.6213)
49. Schonlau, M., Weiß, J., & Marquardt, J. (2023). Multi-label classification of open-ended questions with BERT. 2023 Big Data Meets Survey Science (BigSurv), 1–8. https://doi.org/10.1109/BigSurv59479.2023.10486634 [↗](https://doi.org/10.1109/bigsurv59479.2023.10486634)
50. Schulhoff, S., Ilie, M., Balepur, N., Kahadze, K., Liu, A., Si, C., Li, Y., Gupta, A., Han, H., Schulhoff, S., Dulepet, P. S., Vidyadhara, S., Ki, D., Agrawal, S., Pham, C., Kroiz, G., Li, F., Tao, H., Srivastava, A., Da Costa, H., Gupta, S., Rogers, M. L., Goncearenco, I., Sarli, G., Galynker, I., Peskoff, D., Carpuat, M., White, J., Anadkat, S., Hoyle, A., & Resnik, P. (2024). [The prompt report: A systematic survey of prompting techniques](https://md.research.elamur.ai/free/en/papers/schulhoff-2024-prompt-report-systematic-survey-prompt/). (No. arXiv:2406.06608). arXiv. https://arxiv.org/abs/2406.06608 [↗](https://arxiv.org/abs/2406.06608)
51. Spirling, A. (2023). Why open-source generative AI models are an ethical way forward for science. Nature, 616(7957), 413–413. https://doi.org/10.1038/d41586-023-01295-4 Tian, K., Mitchell, E., Zhou, A., Sharma, A., Rafailov, R., Yao, H., Finn, C., & Manning, C. [↗](https://doi.org/10.1038/d41586-023-01295-4)
52. (2023). Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. In H. Bouamor, J. Pino, & K. Bali (Eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 5433–5442). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.emnlp-main.330 [↗](https://doi.org/10.18653/v1/2023.emnlp-main.330)
53. Törnberg, P. (2024). Large Language Models Outperform Expert Coders and Supervised Classifiers at Annotating Political Social Media Messages. Social Science Computer Review, 08944393241286471. https://doi.org/10.1177/08944393241286471 [↗](https://doi.org/10.1177/08944393241286471)
54. von der Heyde, L., Haensch, A.-C., Wenz, A., & Ma, B. (2025). United in Diversity? [find](https://scholar.google.com/scholar?q=von+der+Heyde%2C+L%2E%2C+Haensch%2C+A%2E%2DC%2E%2C+Wenz%2C+A%2E%2C+%26+Ma%2C+B%2E+%282025%29%2E+United+in+Diversity%3F)
55. Contextual Biases in LLM-Based Predictions of the 2024 European Parliament Elections (No. arXiv: 2409.09045). arXiv. https://arxiv.org/abs/2409.09045 [↗](https://arxiv.org/abs/2409.09045)
56. Wang, W., Jiao, W., Huang, J., Dai, R., Huang, J., Tu, Z., & Lyu, M. (2024). [Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2024-not-all-countries-celebrate-thanksgiving/). In L.-W. Ku, A. Martins, & V. Srikumar (Eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 6349–6384). Association for Computational Linguistics. https://aclanthology.org/2024.acl-long.345 [↗](https://aclanthology.org/2024.acl-long.345/)
57. Weber, M., & Reichardt, M. (2023). Evaluation is all you need. Prompting Generative Large Language Models for Annotation Tasks in the Social Sciences. A Primer using Open Models (No. arXiv:2401.00284). arXiv. http://arxiv.org/abs/2401.00284 Wickham, H., Averick, M., Bryan, J., Chang, W., McGowan, L., François, R., Grolemund, G., Hayes, A., Henry, L., Hester, J., Kuhn, M., Pedersen, T., Miller, E., Bache, S., Müller, K., Ooms, J., Robinson, D., Seidel, D., Spinu, V., … Yutani, H. (2019). Welcome to the Tidyverse. Journal of Open Source Software, 4(43), Article 43. https://doi.org/10.21105/joss.01686 Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., & Rush, A. [↗](https://doi.org/10.21105/joss.01686)
58. M. (2020). Transformers: State-of-the-Art Natural Language Processing (Version v4.25.1) [Computer software]. Zenodo. https://doi.org/10.5281/zenodo.7391177 [↗](https://doi.org/10.5281/zenodo.7391177)
59. Zhang, X., Li, S., Hauer, B., Shi, N., & Kondrak, G. (2023). Don’t Trust ChatGPT when your Question is not in English: A Study of Multilingual Abilities and Types of LLMs. [find](https://scholar.google.com/scholar?q=Zhang%2C+X%2E%2C+Li%2C+S%2E%2C+Hauer%2C+B%2E%2C+Shi%2C+N%2E%2C+%26+Kondrak%2C+G%2E+%282023%29%2E+Don%E2%80%99t+Trust+ChatGPT+when+your+Question+is+not+in+English%3A+A+Study+of+Multilingual+Abilities+and+Types+of+LLMs%2E)
60. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 7915–7927. https://doi.org/10.18653/v1/2023.emnlp-main.491 [↗](https://doi.org/10.18653/v1/2023.emnlp-main.491)
61. Accelerate. (n.d.). HuggingFace. Retrieved January 30, 2025, from [https://huggingface.co/docs/accelerate/index](https://huggingface.co/docs/accelerate/index) [↗](https://huggingface.co/docs/accelerate/index)
62. Hub client library. (n.d.). HuggingFace. Retrieved January 30, 2025, from [https://huggingface.co/docs/huggingface_hub/index](https://huggingface.co/docs/huggingface_hub/index) Appendix I: (see online supplementary materials upon publication) Coding scheme, descriptions and examples in German and English [↗](https://huggingface.co/docs/huggingface_hub/index)

*Updated 1 November 2026*