---
title: "Prompting Instruction-tuned LLMs for Semantic — Snelder 2026"
description: "Gathering human ratings of lexical-semantic properties is expensive, so researchers increasingly ask LLMs to generate such ratings."
authors: ["Xander Akiko Snelder", "Yunchong Huang", "Jelke Bloem"]
year: 2026
venue: "Proceedings of the Fifteenth Language Resources and Evaluation Conference"
source: "https://aclanthology.org/2026.lrec-1.891/"
doi: "10.63317/3kbjxx6989dg"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/snelder-2026-prompting-instruction-tuned-llms-semantic/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/snelder-2026-prompting-instruction-tuned-llms-semantic/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2026](https://md.research.elamur.ai/free/papers/2026/page/4/) / Prompting Instruction-tuned LLMs for Semantic Similarity Values

# Prompting Instruction-tuned LLMs for Semantic Similarity Values

**Authors:** Xander Akiko Snelder, Yunchong Huang, [Jelke Bloem](https://md.research.elamur.ai/free/authors/jelke-bloem/) · **Year:** 2026 · **Venue:** Proceedings of the Fifteenth Language Resources and Evaluation Conference · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.lrec-1.891/) · [PDF](https://aclanthology.org/2026.lrec-1.891.pdf) · [DOI 10.63317/3kbjxx6989dg](https://doi.org/10.63317/3kbjxx6989dg) · [OpenAlex](https://openalex.org/W7160028464)

**Citations:** 0 citations (openalex-snapshot · 2026-10-14) · [2](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Gathering human ratings of lexical-semantic properties is expensive, so researchers increasingly ask LLMs to generate such ratings. This paper tests how closely prompted decoder models match human judgements of word-pair similarity and which prompting set-ups work best. The authors use the SimLex-999 benchmark in English, Dutch and Mandarin Chinese. They compare nine prompt designs: zero-, one- and few-shot variants, different numeric ranges, a categorical scale, an adaptation of the original annotator guidelines, a chat-style format and a one-pair-per-prompt version. They also compare several models: GPT-3.5, GPT-4o, Mistral-7b-v0.3 in base and Instruct versions, gpt-oss-20b, Doubao-pro and Qwen2.5-7b-Instruct. As an embedding baseline, they use cosine scores from OpenAI's text-embedding-3-large. Agreement with human scores is strong across languages and models. Querying a single pair at a time beats batching many pairs, and gpt-oss-20b aligns best with human ratings. Error analysis shows that the generative models still conflate true similarity with mere relatedness, as older static and contextual embeddings did. The findings matter for anyone using LLM scores to build or extend psycholinguistic norms, especially for languages other than English.

## Key points
- The benchmark is SimLex-999: 999 English pairs, each scored 0–10 by roughly 50 US native speakers. The study also uses its Dutch and Mandarin Chinese counterparts. For Chinese, it compares GPT-4o, Doubao-pro and Qwen2.5-7b-Instruct.
- The study defines nine English prompt templates (F-1 to F-9). F-7 reuses the human questionnaire, with synonym examples (cup/mug) contrasted against related-but-dissimilar pairs (car/tyre). F-9 asks about one word pair per prompt.
- The comparison includes instruction-tuned versus non-instruction-tuned Mistral-7b-v0.3, as well as prompting versus text-embedding-3-large embeddings.
- Earlier work reported a correlation of 0.86 for GPT-4 on SimLex-999 (Trott, 2024), against 0.4–0.6 for embedding models. Embeddings extracted from decoders correlate weakly with human ratings: ρ = 0.179 (Arnold et al.), and ρ \< 0.1 for the Dutch models Falcon-7B, BLOOM-560m and Schaapje-2B.
- Code and data are available at github.com/XanderSnelder/semantic-similarity-prompting.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

## Cited by

Cited by

2 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 1 not yet.

- [Lost in Translation: Repurposing semantic similarity benchmarks for evaluating lexical-semantic consistency in LLM-based machine translation](https://md.research.elamur.ai/free/papers/ye-2026-lost-translation-repurposing-semantic-similarity/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{snelder2026prompting,
  title = {{Prompting Instruction-tuned LLMs for Semantic Similarity Values}},
  author = {Snelder, Xander Akiko and Huang, Yunchong and Bloem, Jelke},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3kbjxx6989dg},
  url = {https://doi.org/10.63317/3kbjxx6989dg}
}
```

- [← previous Proficiency-Controlled Text Simplification in European Portuguese: A Preliminary Study using Prompting Approaches](https://md.research.elamur.ai/free/papers/ribeiro-2026-proficiency-controlled-text-simplification-european/)
- [next → Quality and Comprehensibility of Interlingual Subtitles Produced by Humans or with Machines](https://md.research.elamur.ai/free/papers/schluter-2026-quality-comprehensibility-interlingual-subtitles-produced/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/papers/2026/page/4/)

## References

64 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Katherine Abramski, Clara Lavorati, Giulio Rossetti, and Massimo Stella. 2024. LLM-generated word association norms. In HHAI 2024: Hybrid Human AI Systems for the Social Good, pages 3–12. IOS Press. [find](https://scholar.google.com/scholar?q=Katherine+Abramski%2C+Clara+Lavorati%2C+Giulio+Rossetti%2C+and+Massimo+Stella%2E+2024%2E+LLM%2Dgenerated+word+association+norms%2E+In+HHAI+2024%3A+Hybrid+Human+AI+Systems+for+the+Social+Good%2C+pages+3%E2%80%9312%2E+IOS+Press%2E)
2. Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. 2025. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925. [find](https://scholar.google.com/scholar?q=Sandhini+Agarwal%2C+Lama+Ahmad%2C+Jason+Ai%2C+Sam+Altman%2C+Andy+Applebaum%2C+Edwin+Arbus%2C+Rahul+K+Arora%2C+Yu+Bai%2C+Bowen+Baker%2C+Haiming+Bao%2C+et+al%2E+2025%2E+gpt%2Doss%2D120b+%26+gpt%2Doss%2D20b+model+card%2E+arXiv+preprint+arX) [our card](https://md.research.elamur.ai/free/papers/agarwal-2025-gpt-oss-120b-gpt-oss/)
3. Alaa Alzahrani, Wafa Aljuaythin, Hassan Alshumrani, Alaa Mamoun Saleh, and Mohamed M Mostafa. 2025. Kalimah norms: Ratings for 2,467 Modern Standard Arabic words on two scales. Behavior Research Methods, 57(7):1–20. [find](https://scholar.google.com/scholar?q=Alaa+Alzahrani%2C+Wafa+Aljuaythin%2C+Hassan+Alshumrani%2C+Alaa+Mamoun+Saleh%2C+and+Mohamed+M+Mostafa%2E+2025%2E+Kalimah+norms%3A+Ratings+for+2%2C467+Modern+Standard+Arabic+words+on+two+scales%2E+Behavior+Research+Metho)
4. Stefan Arnold, Marian Fietta, and Dilara Yesilbas. 2024. Routing in sparsely-gated language models responds to context. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 15–22. [find](https://scholar.google.com/scholar?q=Stefan+Arnold%2C+Marian+Fietta%2C+and+Dilara+Yesilbas%2E+2024%2E+Routing+in+sparsely%2Dgated+language+models+responds+to+context%2E+In+Proceedings+of+the+7th+BlackboxNLP+Workshop%3A+Analyzing+and+Interpreting+Neura)
5. Mohammad Atari, Mona J Xue, Peter S Park, Damián E Blasi, and Joseph Henrich. 2023. Which humans? [find](https://scholar.google.com/scholar?q=Mohammad+Atari%2C+Mona+J+Xue%2C+Peter+S+Park%2C+Dami%C3%A1n+E+Blasi%2C+and+Joseph+Henrich%2E+2023%2E+Which+humans%3F)
6. Siamak Barzegar, Brian Davis, Manel Zarrouk, Siegfried Handschuh, and Andre Freitas. 2018. [find](https://scholar.google.com/scholar?q=Siamak+Barzegar%2C+Brian+Davis%2C+Manel+Zarrouk%2C+Siegfried+Handschuh%2C+and+Andre+Freitas%2E+2018%2E)
7. SemR-11: A multi-lingual gold-standard for semantic similarity and relatedness for eleven languages. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). [find](https://scholar.google.com/scholar?q=SemR%2D11%3A+A+multi%2Dlingual+gold%2Dstandard+for+semantic+similarity+and+relatedness+for+eleven+languages%2E+Proceedings+of+the+Eleventh+International+Conference+on+Language+Resources+and+Evaluation+%28LREC+201)
8. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. Advances in Neural Information Processing Systems, 3:1137–1155. [find](https://scholar.google.com/scholar?q=Yoshua+Bengio%2C+R%C3%A9jean+Ducharme%2C+Pascal+Vincent%2C+and+Christian+Janvin%2E+2003%2E+A+neural+probabilistic+language+model%2E+Advances+in+Neural+Information+Processing+Systems%2C+3%3A1137%E2%80%931155%2E)
9. Lizzy Brans and Jelke Bloem. 2024. SimLex-999 for Dutch. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 14832–14845. [find](https://scholar.google.com/scholar?q=Lizzy+Brans+and+Jelke+Bloem%2E+2024%2E+SimLex%2D999+for+Dutch%2E+Proceedings+of+the+2024+Joint+International+Conference+on+Computational+Linguistics%2C+Language+Resources+and+Evaluation+%28LREC%2DCOLING+2024%29%2C+page)
10. Lizzy Brans and Jelke Bloem. in press. Multi-SimLex for Dutch: Benchmarking embedding-and prompt-based model performance on semantic similarity. In Proceedings of the Fifteenth International Conference on Language Resources and Evaluation (LREC 2026). [find](https://scholar.google.com/scholar?q=Lizzy+Brans+and+Jelke+Bloem%2E+in+press%2E+Multi%2DSimLex+for+Dutch%3A+Benchmarking+embedding%2Dand+prompt%2Dbased+model+performance+on+semantic+similarity%2E+In+Proceedings+of+the+Fifteenth+International+Conferenc)
11. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… [find](https://scholar.google.com/scholar?q=Tom+B%2E+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche)
12. Marc Brysbaert, Gonzalo Martínez, and Pedro Reviriego. 2025. Moving beyond word frequency based on tally counting: AI-generated familiarity estimates of words and phrases are an interesting additional index of language knowledge. Behavior Research Methods, 57(1):1–15. [find](https://scholar.google.com/scholar?q=Marc+Brysbaert%2C+Gonzalo+Mart%C3%ADnez%2C+and+Pedro+Reviriego%2E+2025%2E+Moving+beyond+word+frequency+based+on+tally+counting%3A+AI%2Dgenerated+familiarity+estimates+of+words+and+phrases+are+an+interesting+additional)
13. Jose Camacho-Collados, Mohammad Taher Pilehvar, Nigel Collier, and Roberto Navigli. 2017. SemEval-2017 task 2: Multilingual and cross-lingual semantic word similarity. Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), page 15–26. [find](https://scholar.google.com/scholar?q=Jose+Camacho%2DCollados%2C+Mohammad+Taher+Pilehvar%2C+Nigel+Collier%2C+and+Roberto+Navigli%2E+2017%2E+SemEval%2D2017+task+2%3A+Multilingual+and+cross%2Dlingual+semantic+word+similarity%2E+Proceedings+of+the+11th+Internat)
14. Billy Chiu, Sampo Pyysalo, Ivan Vulić, and Anna Korhonen. 2018. Bio-SimVerb and Bio-SimLex: wide-coverage evaluation sets of word similarity in biomedicine. BMC Bioinformatics, 19(1). [find](https://scholar.google.com/scholar?q=Billy+Chiu%2C+Sampo+Pyysalo%2C+Ivan+Vuli%C4%87%2C+and+Anna+Korhonen%2E+2018%2E+Bio%2DSimVerb+and+Bio%2DSimLex%3A+wide%2Dcoverage+evaluation+sets+of+word+similarity+in+biomedicine%2E+BMC+Bioinformatics%2C+19%281%29%2E)
15. Gabriella Chronis and Katrin Erk. 2020. When is a bishop not like a rook? when it’s like a rabbi! multi-prototype BERT embeddings for estimating semantic relationships. Proceedings of the 24th Conference on Computational Natural Language Learning, page 227–244. [find](https://scholar.google.com/scholar?q=Gabriella+Chronis+and+Katrin+Erk%2E+2020%2E+When+is+a+bishop+not+like+a+rook%3F+when+it%E2%80%99s+like+a+rabbi%21+multi%2Dprototype+BERT+embeddings+for+estimating+semantic+relationships%2E+Proceedings+of+the+24th+Confere)
16. Simon De Deyne. 2024. Evaluating human-like similarity biases at every scale in large language models: Evidence from remote and basic-level triads. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 46. [find](https://scholar.google.com/scholar?q=Simon+De+Deyne%2E+2024%2E+Evaluating+human%2Dlike+similarity+biases+at+every+scale+in+large+language+models%3A+Evidence+from+remote+and+basic%2Dlevel+triads%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Cognitiv)
17. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1:… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+Proceedings+of+the+2019+Conference+of+the+Nort)
18. Doubao Team. 2024. 8 key moments of Doubao Big Model in 2024. Accessed: 2025-01-15. [find](https://scholar.google.com/scholar?q=Doubao+Team%2E+2024%2E+8+key+moments+of+Doubao+Big+Model+in+2024%2E+Accessed%3A+2025%2D01%2D15%2E)
19. Anton Ehrmanntraut, Thora Hagen, Leonard Konle, and Fotis Jannidis. 2021. Type- and token-based word embeddings in the digital humanities. Workshop on Computational Humanities Research. [find](https://scholar.google.com/scholar?q=Anton+Ehrmanntraut%2C+Thora+Hagen%2C+Leonard+Konle%2C+and+Fotis+Jannidis%2E+2021%2E+Type%2D+and+token%2Dbased+word+embeddings+in+the+digital+humanities%2E+Workshop+on+Computational+Humanities+Research%2E)
20. Lev Finkelstein, Evgeniy Gabrilovich1, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2002. Placing search in context: the concept revisited. ACM Transactions on Information Systems, 20:116–131. [find](https://scholar.google.com/scholar?q=Lev+Finkelstein%2C+Evgeniy+Gabrilovich1%2C+Yossi+Matias%2C+Ehud+Rivlin%2C+Zach+Solan%2C+Gadi+Wolfman%2C+and+Eytan+Ruppin%2E+2002%2E+Placing+search+in+context%3A+the+concept+revisited%2E+ACM+Transactions+on+Information+Sy)
21. Daniela Gerz, Ivan Vulić, Felix Hill, Roi Reichart, and Anna Korhonen. 2016. SimVerb-3500: A large-scale evaluation set of verb similarity. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Daniela+Gerz%2C+Ivan+Vuli%C4%87%2C+Felix+Hill%2C+Roi+Reichart%2C+and+Anna+Korhonen%2E+2016%2E+SimVerb%2D3500%3A+A+large%2Dscale+evaluation+set+of+verb+similarity%2E+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+N)
22. Anna Gladkova and Aleksandr Drozd. 2016. Intrinsic evaluations of word embeddings: What can we do better? In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pages 36–42. [find](https://scholar.google.com/scholar?q=Anna+Gladkova+and+Aleksandr+Drozd%2E+2016%2E+Intrinsic+evaluations+of+word+embeddings%3A+What+can+we+do+better%3F+In+Proceedings+of+the+1st+Workshop+on+Evaluating+Vector%2DSpace+Representations+for+NLP%2C+pages+3)
23. Hila Gonen, Srini Iyer, Terra Blevins, Noah Smith, and Luke Zettlemoyer. 2023. Demystifying prompts in language models via perplexity estimation. Findings of the Association for Computational Linguistics: EMNLP 2023, page 10136–10148. [find](https://scholar.google.com/scholar?q=Hila+Gonen%2C+Srini+Iyer%2C+Terra+Blevins%2C+Noah+Smith%2C+and+Luke+Zettlemoyer%2E+2023%2E+Demystifying+prompts+in+language+models+via+perplexity+estimation%2E+Findings+of+the+Association+for+Computational+Linguist)
24. Jack Grieve, Sara Bartl, Matteo Fuoli, Jason Grafmiller, Weihang Huang, Alejandro Jawerbaum, Akira Murakami, Marcus Perlman, Dana Roemling, and Bodo Winter. 2025. The sociolinguistic foundations of language modeling. Frontiers in Artificial Intelligence, 7:1472411. [find](https://scholar.google.com/scholar?q=Jack+Grieve%2C+Sara+Bartl%2C+Matteo+Fuoli%2C+Jason+Grafmiller%2C+Weihang+Huang%2C+Alejandro+Jawerbaum%2C+Akira+Murakami%2C+Marcus+Perlman%2C+Dana+Roemling%2C+and+Bodo+Winter%2E+2025%2E+The+sociolinguistic+foundations+of+la)
25. Zellig S Harris. 1954. Distributional structure. Word, 10(2-3):146–162. [find](https://scholar.google.com/scholar?q=Zellig+S+Harris%2E+1954%2E+Distributional+structure%2E+Word%2C+10%282%2D3%29%3A146%E2%80%93162%2E)
26. Yoshihiko Hayashi. 2025. Evaluating LLMs’ capability to identify lexical semantic equivalence: Probing with the word-in-context task. In Proceedings of the 31st International Conference on Computational Linguistics, pages 6985–6998, Abu Dhabi, UAE. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yoshihiko+Hayashi%2E+2025%2E+Evaluating+LLMs%E2%80%99+capability+to+identify+lexical+semantic+equivalence%3A+Probing+with+the+word%2Din%2Dcontext+task%2E+In+Proceedings+of+the+31st+International+Conference+on+Computation)
27. Felix Hill, Roi Reichart, and Anna Korhonen. 2015. SimLex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665–695. [find](https://scholar.google.com/scholar?q=Felix+Hill%2C+Roi+Reichart%2C+and+Anna+Korhonen%2E+2015%2E+SimLex%2D999%3A+Evaluating+semantic+models+with+%28genuine%29+similarity+estimation%2E+Computational+Linguistics%2C+41%284%29%3A665%E2%80%93695%2E) [our card](https://md.research.elamur.ai/free/papers/hill-2015-simlex-999-evaluating-semantic-models/)
28. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
29. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423–438. [find](https://scholar.google.com/scholar?q=Zhengbao+Jiang%2C+Frank+F%2E+Xu%2C+Jun+Araki%2C+and+Graham+Neubig%2E+2020%2E+How+can+we+know+what+language+models+know%3F+Transactions+of+the+Association+for+Computational+Linguistics%2C+8%3A423%E2%80%93438%2E) [our card](https://md.research.elamur.ai/free/papers/jiang-2020-how-can-we-know-what/)
30. Jaap Jumelet, Willem Zuidema, and Arabella Sinclair. 2024. Do language models exhibit human-like structural priming effects? In Findings of the Association for Computational Linguistics: ACL 2024, pages 14727–14742, Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jaap+Jumelet%2C+Willem+Zuidema%2C+and+Arabella+Sinclair%2E+2024%2E+Do+language+models+exhibit+human%2Dlike+structural+priming+effects%3F+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2024%2C+pag)
31. Jon A Krosnick and Leandre R Fabrigar. 1997. Designing rating scales for effective measurement in surveys. Survey measurement and process quality, pages 141–164. [find](https://scholar.google.com/scholar?q=Jon+A+Krosnick+and+Leandre+R+Fabrigar%2E+1997%2E+Designing+rating+scales+for+effective+measurement+in+surveys%2E+Survey+measurement+and+process+quality%2C+pages+141%E2%80%93164%2E)
32. Jonghyun Lee, Dojun Park, Jiwoo Lee, Hoekeon Choi, and Sung-Eun Lee. 2025. Exploring multimodal perception in large language models through perceptual strength ratings. IEEE Access, 13:176751–176769. [find](https://scholar.google.com/scholar?q=Jonghyun+Lee%2C+Dojun+Park%2C+Jiwoo+Lee%2C+Hoekeon+Choi%2C+and+Sung%2DEun+Lee%2E+2025%2E+Exploring+multimodal+perception+in+large+language+models+through+perceptual+strength+ratings%2E+IEEE+Access%2C+13%3A176751%E2%80%93176769%2E)
33. Alina Leidinger, Robert van Rooij, and Ekaterina Shutova. 2023. The language of prompting: What linguistic properties make a prompt successful? Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9210– 9232. [find](https://scholar.google.com/scholar?q=Alina+Leidinger%2C+Robert+van+Rooij%2C+and+Ekaterina+Shutova%2E+2023%2E+The+language+of+prompting%3A+What+linguistic+properties+make+a+prompt+successful%3F+Findings+of+the+Association+for+Computational+Linguistic)
34. Alessandro Lenci. 2018. Distributional models of word meaning. Annual review of Linguistics, 4(1):151–171. [find](https://scholar.google.com/scholar?q=Alessandro+Lenci%2E+2018%2E+Distributional+models+of+word+meaning%2E+Annual+review+of+Linguistics%2C+4%281%29%3A151%E2%80%93171%2E)
35. Ira Leviant and Roi Reichart. 2015. Separated by an un-common language: Towards judgment language informed vector space modeling. [find](https://scholar.google.com/scholar?q=Ira+Leviant+and+Roi+Reichart%2E+2015%2E+Separated+by+an+un%2Dcommon+language%3A+Towards+judgment+language+informed+vector+space+modeling%2E)
36. Jeannette Littlemore, Paula Pérez Sobrino, David Houghton, Jinfang Shi, and Bodo Winter. 2018. What makes a good metaphor? A cross-cultural study of computer-generated metaphor appreciation. Metaphor and Symbol, 33(2):101–122. [find](https://scholar.google.com/scholar?q=Jeannette+Littlemore%2C+Paula+P%C3%A9rez+Sobrino%2C+David+Houghton%2C+Jinfang+Shi%2C+and+Bodo+Winter%2E+2018%2E+What+makes+a+good+metaphor%3F+A+cross%2Dcultural+study+of+computer%2Dgenerated+metaphor+appreciation%2E+Metaphor+)
37. Chaoqun Liu, Wenxuan Zhang, Yiran Zhao, Anh Tuan Luu, and Lidong Bing. 2025. Is translation all you need? A study on solving multilingual tasks with large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: H… [find](https://scholar.google.com/scholar?q=Chaoqun+Liu%2C+Wenxuan+Zhang%2C+Yiran+Zhao%2C+Anh+Tuan+Luu%2C+and+Lidong+Bing%2E+2025%2E+Is+translation+all+you+need%3F+A+study+on+solving+multilingual+tasks+with+large+language+models%2E+In+Proceedings+of+the+2025+C)
38. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). [find](https://scholar.google.com/scholar?q=Yao+Lu%2C+Max+Bartolo%2C+Alastair+Moore%2C+Sebastian+Riedel%2C+and+Pontus+Stenetorp%2E+2022%2E+Fantastically+ordered+prompts+and+where+to+find+them%3A+Overcoming+few%2Dshot+prompt+order+sensitivity%2E+Proceedings+of+th) [our card](https://md.research.elamur.ai/free/papers/lu-2022-fantastically-ordered-prompts-where-find/)
39. Bolei Ma. 2024. Evaluating lexical aspect with large language models. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 123–131. [find](https://scholar.google.com/scholar?q=Bolei+Ma%2E+2024%2E+Evaluating+lexical+aspect+with+large+language+models%2E+In+Proceedings+of+the+Workshop+on+Cognitive+Modeling+and+Computational+Linguistics%2C+pages+123%E2%80%93131%2E)
40. Gonzalo Martínez, Javier Conde, Pedro Reviriego, and Marc Brysbaert. 2024. AI-generated estimates of familiarity, concreteness, valence, and arousal for over 100,000 Spanish words. Quarterly Journal of Experimental Psychology, page 17470218241306694. [find](https://scholar.google.com/scholar?q=Gonzalo+Mart%C3%ADnez%2C+Javier+Conde%2C+Pedro+Reviriego%2C+and+Marc+Brysbaert%2E+2024%2E+AI%2Dgenerated+estimates+of+familiarity%2C+concreteness%2C+valence%2C+and+arousal+for+over+100%2C000+Spanish+words%2E+Quarterly+Journal+o)
41. Gonzalo Martínez, Juan Diego Molero, Sandra González, Javier Conde, Marc Brysbaert, and Pedro Reviriego. 2025. Using large language models to estimate features of multi-word expressions: Concreteness, valence, arousal. Behavior Research Methods, 57(1):1–11. [find](https://scholar.google.com/scholar?q=Gonzalo+Mart%C3%ADnez%2C+Juan+Diego+Molero%2C+Sandra+Gonz%C3%A1lez%2C+Javier+Conde%2C+Marc+Brysbaert%2C+and+Pedro+Reviriego%2E+2025%2E+Using+large+language+models+to+estimate+features+of+multi%2Dword+expressions%3A+Concreteness%2C)
42. Natalja Menold. 2020. Rating-scale labeling in online surveys: An experimental comparison of verbal and numeric rating scales with respect to measurement quality and respondents’ cognitive processes. Sociological Methods & Research, 49(1):79–107. [find](https://scholar.google.com/scholar?q=Natalja+Menold%2E+2020%2E+Rating%2Dscale+labeling+in+online+surveys%3A+An+experimental+comparison+of+verbal+and+numeric+rating+scales+with+respect+to+measurement+quality+and+respondents%E2%80%99+cognitive+processes%2E+)
43. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of ICLR (Workshop Track). [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Kai+Chen%2C+Greg+Corrado%2C+and+Jeffrey+Dean%2E+2013%2E+Efficient+estimation+of+word+representations+in+vector+space%2E+In+Proceedings+of+ICLR+%28Workshop+Track%29%2E)
44. Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, Madelei… [find](https://scholar.google.com/scholar?q=Arvind+Neelakantan%2C+Tao+Xu%2C+Raul+Puri%2C+Alec+Radford%2C+Jesse+Michael+Han%2C+Jerry+Tworek%2C+Qiming+Yuan%2C+Nikolas+Tezak%2C+Jong+Wook+Kim%2C+Chris+Hallacy%2C+Johannes+Heidecke%2C+Pranav+Shyam%2C+Boris+Power%2C+Tyna+Eloun) [our card](https://md.research.elamur.ai/free/papers/neelakantan-2022-text-code-embeddings-contrastive-pre/)
45. OpenAI. 2022. Introducing ChatGPT. Retrieved April 8, 2024 from [https://openai.com/blog/chatgpt\#OpenAI](https://openai.com/blog/chatgpt\#OpenAI). [↗](https://openai.com/blog/chatgpt\#OpenAI)
46. OpenAI. 2023. GPT-4 Technical Report. Retrieved April 14, 2024 from https://doi.org/10.48550/arXiv.2303.08774. [↗](https://doi.org/10.48550/arXiv.2303.08774) [our card](https://md.research.elamur.ai/free/papers/openai-2023-gpt-4-technical-report/)
47. OpenAI. 2024a. Hello GPT-4o — OpenAI. Retrieved June 20, 2024 from [https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/). [↗](https://openai.com/index/hello-gpt-4o/)
48. OpenAI. 2024b. Models — OpenAI API. Retrieved April 8, 2024 from [https://platform.openai.com/docs/models](https://platform.openai.com/docs/models). [↗](https://platform.openai.com/docs/models)
49. OpenAI. 2024c. New embedding mod-Sean Trott. 2024. Can large language models help els and API updates. Retrieved April 8, augment English psycholinguistic datasets? Be-2024 from [https://openai.com/blog/new-embedding-models-and-api-updates](https://openai.com/blog/new-embedding-models-and-api-updates). havior Research Methods, pages 1–19. [↗](https://openai.com/blog/new-embedding-models-and-api-updates)
50. OpenAI. 2024d. Pricing. Retrieved April 11, 2024 from [https://openai.com/pricing](https://openai.com/pricing). [↗](https://openai.com/pricing)
51. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2024. Train… [find](https://scholar.google.com/scholar?q=Long+Ouyang%2C+Jeff+Wu%2C+Xu+Jiang%2C+Diogo+Almeida%2C+Carroll+L%2E+Wainwright%2C+Pamela+Mishkin%2C+Chong+Zhang%2C+Sandhini+Agarwal%2C+Katarina+Slama%2C+Alex+Ray%2C+John+Schulman%2C+Jacob+Hilton%2C+Fraser+Kelton%2C+Luke+Miller%2C+)
52. Mohammad Taher Pilehvar, Dimitri Kartsaklis, Victor Prokhorov, and Nigel Collier. 2018. Card-660: Cambridge Rare Word Dataset - a reliable benchmark for infrequent word representation models. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, page 1391–1401. [find](https://scholar.google.com/scholar?q=Mohammad+Taher+Pilehvar%2C+Dimitri+Kartsaklis%2C+Victor+Prokhorov%2C+and+Nigel+Collier%2E+2018%2E+Card%2D660%3A+Cambridge+Rare+Word+Dataset+%2D+a+reliable+benchmark+for+infrequent+word+representation+models%2E+Proceedi)
53. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng… [find](https://scholar.google.com/scholar?q=Qwen%2C+%3A%2C+An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+)
54. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
55. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842– 866. [find](https://scholar.google.com/scholar?q=Anna+Rogers%2C+Olga+Kovaleva%2C+and+Anna+Rumshisky%2E+2020%2E+A+primer+in+BERTology%3A+What+we+know+about+how+BERT+works%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+8%3A842%E2%80%93+866%2E) [our card](https://md.research.elamur.ai/free/papers/rogers-2020-primer-bertology-what-we-know/)
56. Timo Schick and Hinrich Schütze. 2022. True few-shot learning with Prompts—A real-world perspective. Transactions of the Association for Computational Linguistics, 10:716–731. [find](https://scholar.google.com/scholar?q=Timo+Schick+and+Hinrich+Sch%C3%BCtze%2E+2022%2E+True+few%2Dshot+learning+with+Prompts%E2%80%94A+real%2Dworld+perspective%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+10%3A716%E2%80%93731%2E)
57. Tobias Schnabel, Igor Labutov, David Mimno, and Thorsten Joachims. 2015. Evaluation methods for unsupervised word embeddings. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 298– 307, Lisbon, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tobias+Schnabel%2C+Igor+Labutov%2C+David+Mimno%2C+and+Thorsten+Joachims%2E+2015%2E+Evaluation+methods+for+unsupervised+word+embeddings%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Natural+Lang)
58. Ivan Vulić, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020. Probing pretrained language models for lexical semantics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7222–7240, Online. Association for Computational… [find](https://scholar.google.com/scholar?q=Ivan+Vuli%C4%87%2C+Edoardo+Maria+Ponti%2C+Robert+Litschko%2C+Goran+Glava%C5%A1%2C+and+Anna+Korhonen%2E+2020%2E+Probing+pretrained+language+models+for+lexical+semantics%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Me) [our card](https://md.research.elamur.ai/free/papers/vulic-2020-probing-pretrained-language-models-lexical/)
59. Ivan Vulić, Simon Baker, Edoardo Maria Ponti, Ulla Petti, Ira Leviant, Kelly Wing, Olga Majewska, Eden Bar, Matt Malone, Thierry Poibeau, Roi Reichart, and Anna Korhonen. 2020. Multi-SimLex: A large-scale evaluation of multilingual and crosslingual lexical semantic similarity. Computational Linguist… [find](https://scholar.google.com/scholar?q=Ivan+Vuli%C4%87%2C+Simon+Baker%2C+Edoardo+Maria+Ponti%2C+Ulla+Petti%2C+Ira+Leviant%2C+Kelly+Wing%2C+Olga+Majewska%2C+Eden+Bar%2C+Matt+Malone%2C+Thierry+Poibeau%2C+Roi+Reichart%2C+and+Anna+Korhonen%2E+2020%2E+Multi%2DSimLex%3A+A+large%2Ds)
60. Albert Webson and Ellie Pavlick. 2022. Do prompt-based models really understand the meaning of their prompts? Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2300–2344. [find](https://scholar.google.com/scholar?q=Albert+Webson+and+Ellie+Pavlick%2E+2022%2E+Do+prompt%2Dbased+models+really+understand+the+meaning+of+their+prompts%3F+Proceedings+of+the+2022+Conference+of+the+North+American+Chapter+of+the+Association+for+Co) [our card](https://md.research.elamur.ai/free/papers/webson-2022-do-prompt-based-models-really/)
61. Qinyuan Ye, Maxamed Axmed, Reid Pryzant, and Fereshte Khani. 2023. Prompt engineering a prompt engineer. [find](https://scholar.google.com/scholar?q=Qinyuan+Ye%2C+Maxamed+Axmed%2C+Reid+Pryzant%2C+and+Fereshte+Khani%2E+2023%2E+Prompt+engineering+a+prompt+engineer%2E)
62. Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. [find](https://scholar.google.com/scholar?q=Tony+Z%2E+Zhao%2C+Eric+Wallace%2C+Shi+Feng%2C+Dan+Klein%2C+and+Sameer+Singh%2E+2021%2E+Calibrate+before+use%3A+Improving+few%2Dshot+performance+of+language+models%2E) [our card](https://md.research.elamur.ai/free/papers/zhao-2021-calibrate-before-use-improving-few/)
63. Yanlu Zhong, Simon Todd, Nicole Xu, and Laurel Brehm. 2025. Evaluating LLMs as proxies for humans in psycholinguistic ratings: A comparison of statistical knowledge. Research Methods in Applied Linguistics, 4(3):100274. [find](https://scholar.google.com/scholar?q=Yanlu+Zhong%2C+Simon+Todd%2C+Nicole+Xu%2C+and+Laurel+Brehm%2E+2025%2E+Evaluating+LLMs+as+proxies+for+humans+in+psycholinguistic+ratings%3A+A+comparison+of+statistical+knowledge%2E+Research+Methods+in+Applied+Lingui)
64. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. [find](https://scholar.google.com/scholar?q=Yongchao+Zhou%2C+Andrei+Ioan+Muresanu%2C+Ziwen+Han%2C+Keiran+Paster%2C+Silviu+Pitis%2C+Harris+Chan%2C+and+Jimmy+Ba%2E+2022%2E+Large+language+models+are+human%2Dlevel+prompt+engineers%2E) [our card](https://md.research.elamur.ai/free/papers/zhou-2022-large-language-models-human-level/)