---
title: "Rethinking Pragmatics in Large Language Models — Wu 2024"
description: "They therefore propose evaluating free-form answers with the Length-Normalized Relative Score (LN RS) metric. 70 references, cited by 13 papers here."
authors: ["Shengguang Wu", "Shusheng Yang", "Zhenglun Chen", "Qi Su"]
year: 2024
venue: "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2024.emnlp-main.1258/"
doi: "10.18653/v1/2024.emnlp-main.1258"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wu-2024-rethinking-pragmatics-large-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wu-2024-rethinking-pragmatics-large-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / Rethinking Pragmatics in Large Language Models: Towards Open-Ended Evaluation and Preference Tuning

# Rethinking Pragmatics in Large Language Models: Towards Open-Ended Evaluation and Preference Tuning

**Authors:** Shengguang Wu, [Shusheng Yang](https://md.research.elamur.ai/free/en/authors/shusheng-yang/), Zhenglun Chen and Qi Su · **Year:** 2024 · **Venue:** Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.emnlp-main.1258/) · [PDF](https://aclanthology.org/2024.emnlp-main.1258.pdf) · [DOI 10.18653/v1/2024.emnlp-main.1258](https://doi.org/10.18653/v1/2024.emnlp-main.1258) · [OpenAlex](https://openalex.org/W4404782237)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [13](https://md.research.elamur.ai/free/en/papers/wu-2024-rethinking-pragmatics-large-language-models/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
The authors investigate how to evaluate and improve the social-pragmatic reasoning of large language models. This includes understanding implicatures, irony, humor, and metaphors, as well as theory of mind (ToM). In their view, the conventional MCQA format with an accuracy metric poorly reflects a model's actual abilities: a model may select the correct option yet produce a non-pragmatic answer on its own. They therefore propose evaluating free-form answers with the Length-Normalized Relative Score (LN RS) metric. GPT-4 acts as a judge, comparing the model's answer with a reference answer, while a sigmoid correction for the length difference reduces the advantage of verbose answers. For training, the unselected options in MCQA datasets are treated not as wrong but as less pragmatic. This yields pairs for preference optimization (DPO). This approach consistently outperforms SFT across various pragmatic phenomena and barely harms general skills. It also helps in a multimodal image referential game that requires ToM. The analysis shows that the gains are associated primarily with the deeper layers of the transformer. The work motivates a shift toward open-ended evaluation and preference-based training for socially aware models.

## Key points
- For LN RS, GPT-4 (gpt-4-1106-preview) is queried twice with the order of the answers swapped to reduce positional bias. Human evaluation confirmed that LN RS aligns better with human preferences than MCQA-Acc.
- Base models: LLAMA2-7B-Chat, LLAMA2-13B-Chat, and PYTHIA-6.9B-Tulu. They were fine-tuned on SocialIQA and PUB using PO and SFT.
- On general benchmarks (MMLU, ARC-E, ARC-C, AGIEval, GSM8K, OpenBookQA), models after PO generally stay on par with or exceed the base models, whereas SFT sometimes degrades performance. For example, for LLAMA2-7B-Chat trained on SocialIQA, ARC-C is 58.4 with PO versus 52.6 with SFT (53.2 for the base model), and GSM8K drops from 23.2 to 20.2 after SFT.
- Training only the deeper layers noticeably improves pragmatic abilities. The authors see a parallel here with higher-order human cognitive processes.
- Limitations: the evaluation depends on the closed GPT-4 API, and models inherit a tendency toward verbosity from PO.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/)

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 74 works · 31,748 citations
- [Huazhong University of Science and Technology](https://md.research.elamur.ai/free/en/universities/huazhong-university-of-science-and-technology/) — University, Wuhan, China · 15 works · 3,454 citations

## Cite

Cite as: Shengguang Wu, Shusheng Yang, Zhenglun Chen and Qi Su. 2024. Rethinking Pragmatics in Large Language Models: Towards Open-Ended Evaluation and Preference Tuning. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wu2024rethinking,
  title = {{Rethinking Pragmatics in Large Language Models: Towards Open-Ended Evaluation and Preference Tuning}},
  author = {Shengguang Wu, Shusheng Yang, Zhenglun Chen and Qi Su},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.1258},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.1258}
}
```

- [← previous Pragmatic inference of scalar implicature by LLMs](https://md.research.elamur.ai/free/en/papers/cho-2024-pragmatic-inference-scalar-implicature-llms/)
- [next → StyleRemix: Interpretable Authorship Obfuscation via Distillation and Perturbation of Style Elements](https://md.research.elamur.ai/free/en/papers/fisher-2024-styleremix-interpretable-authorship-obfuscation-distillation/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

70 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. John Langshaw Austin. 1962. How to do things with words. William James Lectures. Oxford University Press. [find](https://scholar.google.com/scholar?q=John+Langshaw+Austin%2E+1962%2E+How+to+do+things+with+words%2E+William+James+Lectures%2E+Oxford+University+Press%2E)
2. Bruno G Bara. 2011. Cognitive pragmatics: The mental processes of communication. [find](https://scholar.google.com/scholar?q=Bruno+G+Bara%2E+2011%2E+Cognitive+pragmatics%3A+The+mental+processes+of+communication%2E)
3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam Mc-Candlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. Preprint, arXiv:2005.14165. [↗](https://arxiv.org/abs/2005.14165) [our card](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
4. Robyn Carston. 2004. Stephen c. levinson, presumptive meanings: the theory of generalized conversational implicature. cambridge, ma: Mit press, 2000. pp. xxiii+ 480. Journal of linguistics, 40(1):181–186. [find](https://scholar.google.com/scholar?q=Robyn+Carston%2E+2004%2E+Stephen+c%2E+levinson%2C+presumptive+meanings%3A+the+theory+of+generalized+conversational+implicature%2E+cambridge%2C+ma%3A+Mit+press%2C+2000%2E+pp%2E+xxiii%2B+480%2E+Journal+of+linguistics%2C+40%281%29%3A181%E2%80%93)
5. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chiang%2C+Zhuohan+Li%2C+Zi+Lin%2C+Ying+Sheng%2C+Zhanghao+Wu%2C+Hao+Zhang%2C+Lianmin+Zheng%2C+Siyuan+Zhuang%2C+Yonghao+Zhuang%2C+Joseph+E%2E+Gonzalez%2C+Ion+Stoica%2C+and+Eric+P%2E+Xing%2E+2023%2E+Vicuna%3A+An+open%2Dsource+cha)
6. Paul F Christiano, Jan Leike, Tom Brown, Miljan Mar-tic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30. [find](https://scholar.google.com/scholar?q=Paul+F+Christiano%2C+Jan+Leike%2C+Tom+Brown%2C+Miljan+Mar%2Dtic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+2017%2E+Deep+reinforcement+learning+from+human+preferences%2E+Advances+in+neural+information+processing+systems%2C+30%2E)
7. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. [↗](https://arxiv.org/abs/1803.05457)
8. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168. [↗](https://arxiv.org/abs/2110.14168) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
9. Rodolfo Corona, Stephan Alaniz, and Zeynep Akata. 2019. Modeling conceptual understanding in image reference games. ArXiv, abs/1910.04872. [↗](https://arxiv.org/abs/1910.04872)
10. Yann Dubois. 2024. Length controlled alpacaeval. [find](https://scholar.google.com/scholar?q=Yann+Dubois%2E+2024%2E+Length+controlled+alpacaeval%2E)
11. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. 2024. Length-controlled al-pacaeval: A simple way to debias automatic evaluators. Preprint, arXiv:2404.04475. [↗](https://arxiv.org/abs/2404.04475)
12. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpaca-farm: A simulation framework for methods that learn from human feedback. Preprint, arXiv:2305.14387. [↗](https://arxiv.org/abs/2305.14387) [our card](https://md.research.elamur.ai/free/en/papers/dubois-2023-alpacafarm-simulation-framework-methods-that/)
13. Sammy Floyd. 2022. Pragmega materials. [find](https://scholar.google.com/scholar?q=Sammy+Floyd%2E+2022%2E+Pragmega+materials%2E)
14. Balazs Galambosi. 2024. Advanced length-normalized alpacaeval 2.0. [https://github.com/tatsu-lab/](https://github.com/tatsu-lab/) alpaca_eval/issues/225. [↗](https://github.com/tatsu-lab/)
15. Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gersten-berg, and Noah D. Goodman. 2023. Understanding social reasoning in language models with language models. Preprint, arXiv:2306.15448. [↗](https://arxiv.org/abs/2306.15448)
16. Erving Goffman. 1959. The moral career of the mental patient. Psychiatry, 22(2):123–142. [find](https://scholar.google.com/scholar?q=Erving+Goffman%2E+1959%2E+The+moral+career+of+the+mental+patient%2E+Psychiatry%2C+22%282%29%3A123%E2%80%93142%2E)
17. Mitchell S Green. 1998. Direct reference and implicature. Philosophical Studies: An International Journal for Philosophy in the Analytic Tradition, 91(1):61–90. [find](https://scholar.google.com/scholar?q=Mitchell+S+Green%2E+1998%2E+Direct+reference+and+implicature%2E+Philosophical+Studies%3A+An+International+Journal+for+Philosophy+in+the+Analytic+Tradition%2C+91%281%29%3A61%E2%80%9390%2E)
18. Cameron Robert Jones, Sean Trott, and Ben Bergen. 2023. EPITOME: Experimental protocol inventory for theory of mind evaluation. In First Workshop on Theory of Mind in Communicating Agents. [find](https://scholar.google.com/scholar?q=Cameron+Robert+Jones%2C+Sean+Trott%2C+and+Ben+Bergen%2E+2023%2E+EPITOME%3A+Experimental+protocol+inventory+for+theory+of+mind+evaluation%2E+In+First+Workshop+on+Theory+of+Mind+in+Communicating+Agents%2E)
19. Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41–58. Brill. [↗](https://doi.org/10.1163/9789004368811_003)
20. Hyunwoo Kim, Melanie Sclar, Xuhui Zhou, Ronan Le Bras, Gunhee Kim, Yejin Choi, and Maarten Sap. 2023. Fantom: A benchmark for stress-testing machine theory of mind in interactions. Preprint, arXiv:2310.15421. [↗](https://arxiv.org/abs/2310.15421)
21. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. [↗](https://arxiv.org/abs/2009.03300) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
22. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. CLIPScore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514–7528, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/hessel-2021-clipscore-reference-free-evaluation-metric/)
23. Geoffrey Hinton. 2014. Coursera lecture slides - neural networks for machine learning lecture 6. [find](https://scholar.google.com/scholar?q=Geoffrey+Hinton%2E+2014%2E+Coursera+lecture+slides+%2D+neural+networks+for+machine+learning+lecture+6%2E)
24. Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. Surface form competition: Why the highest probability answer isn’t always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7038–7051. [our card](https://md.research.elamur.ai/free/en/papers/holtzman-2021-surface-form-competition-why-highest/)
25. Laurence R. Horn. 1972. On the semantic properties of logical operators in english’ reproduced by the indiana university lin. [find](https://scholar.google.com/scholar?q=Laurence+R%2E+Horn%2E+1972%2E+On+the+semantic+properties+of+logical+operators+in+english%E2%80%99+reproduced+by+the+indiana+university+lin%2E)
26. Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.02301. [↗](https://arxiv.org/abs/2305.02301) [our card](https://md.research.elamur.ai/free/en/papers/hsieh-2023-distilling-step-step-outperforming-larger/)
27. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. [↗](https://arxiv.org/abs/2106.09685) [our card](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/)
28. Jennifer Hu, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, and Edward Gibson. 2023. A fine-grained comparison of pragmatic language understanding in humans and language models. Preprint, arXiv:2212.06801. [↗](https://arxiv.org/abs/2212.06801) [our card](https://md.research.elamur.ai/free/en/papers/hu-2023-fine-grained-comparison-pragmatic-language/)
29. Jennifer Hu and Roger Levy. 2023. Prompting is not a substitute for probability measurements in large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5040–5060. [our card](https://md.research.elamur.ai/free/en/papers/hu-2023-prompting-is-not-substitute-probability/)
30. Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. 2021. Openclip. If you use this software, please cite it as below. [find](https://scholar.google.com/scholar?q=Gabriel+Ilharco%2C+Mitchell+Wortsman%2C+Ross+Wightman%2C+Cade+Gordon%2C+Nicholas+Carlini%2C+Rohan+Taori%2C+Achal+Dave%2C+Vaishaal+Shankar%2C+Hongseok+Namkoong%2C+John+Miller%2C+Hannaneh+Hajishirzi%2C+Ali+Farhadi%2C+and+Ludwi)
31. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213. [↗](https://doi.org/10.52202/068431-1613) [our card](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/)
32. Michal Kosinski. 2023. Theory of mind might have spontaneously emerged in large language models. arXiv preprint arXiv:2302.02083. [↗](https://arxiv.org/abs/2302.02083) [our card](https://md.research.elamur.ai/free/en/papers/kosinski-2023-evaluating-large-language-models-theory/)
33. Matthew Le, Y-Lan Boureau, and Maximilian Nickel. 2019. Revisiting the evaluation of theory of mind through question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5872–5877. [find](https://scholar.google.com/scholar?q=Matthew+Le%2C+Y%2DLan+Boureau%2C+and+Maximilian+Nickel%2E+2019%2E+Revisiting+the+evaluation+of+theory+of+mind+through+question+answering%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+La)
34. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval). [↗](https://github.com/tatsu-lab/alpaca_eval)
35. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer. [our card](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/)
36. Benjamin Lipkin, Lionel Wong, Gabriel Grand, and Joshua B Tenenbaum. 2023. Evaluating statistical language models as pragmatic reasoners. arXiv preprint arXiv:2305.01020. [↗](https://arxiv.org/abs/2305.01020) [our card](https://md.research.elamur.ai/free/en/papers/lipkin-2023-evaluating-statistical-language-models-pragmatic/)
37. Andy Liu, Hao Zhu, Emmy Liu, Yonatan Bisk, and Graham Neubig. 2023. Computational language acquisition with theory of mind. Preprint, arXiv:2303.01502. [↗](https://arxiv.org/abs/2303.01502) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-computational-language-acquisition-theory-mind/)
38. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024a. Visual instruction tuning. Advances in neural information processing systems, 36. [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/)
39. Jie Liu, Zhanhui Zhou, Jiaheng Liu, Xingyuan Bu, Chao Yang, Han-Sen Zhong, and Wanli Ouyang. 2024b. Iterative length-regularized direct preference optimization: A case study on improving 7b language models to gpt-4 level. arXiv preprint arXiv:2406.11817. [↗](https://arxiv.org/abs/2406.11817)
40. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. [↗](https://arxiv.org/abs/1711.05101)
41. Junru Lu, Jiazheng Li, Siyu An, Meng Zhao, Yulan He, Di Yin, and Xing Sun. 2024. Eliminating biased length reliance of direct preference optimization via down-sampled kl divergence. arXiv preprint arXiv:2406.10957. [↗](https://arxiv.org/abs/2406.10957)
42. Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734. [↗](https://arxiv.org/abs/2405.14734)
43. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789. [↗](https://arxiv.org/abs/1809.02789)
44. Shima Rahimi Moghaddam and Christopher J Honey. 2023. Boosting theory-of-mind performance in large language models via prompting. arXiv preprint arXiv:2304.11490. [↗](https://arxiv.org/abs/2304.11490)
45. OpenAI. 2023. Gpt-4 technical report. arXiv:2303.08774. [↗](https://arxiv.org/abs/2303.08774) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
46. Preprint, Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. Preprint, arXiv:2203.02155. [↗](https://arxiv.org/abs/2203.02155) [our card](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
47. David Premack and Guy Woodruff. 1978. Does the chimpanzee have a theory of mind? Behavioral and brain sciences, 1(4):515–526. [↗](https://doi.org/10.1017/s0140525x00076512)
48. Peng Qi, Nina Du, Christopher Manning, and Jing Huang. 2023. PragmatiCQA: A dataset for pragmatic question answering in conversations. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6175–6191, Toronto, Canada. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/qi-2023-pragmaticqa-dataset-pragmatic-question-answering/)
49. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36. [our card](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/)
50. Joshua Robinson and David Wingate. 2023. Leveraging large language models for multiple choice question answering. In International Conference on Learning Representations. [our card](https://md.research.elamur.ai/free/en/papers/robinson-2022-leveraging-large-language-models-multiple/)
51. Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, and Edward Grefenstette. 2023. The goldilocks of pragmatic understanding: Fine-tuning strategy matters for implicature resolution by llms. Preprint, arXiv:2210.14986. [↗](https://arxiv.org/abs/2210.14986) [our card](https://md.research.elamur.ai/free/en/papers/ruis-2022-goldilocks-pragmatic-understanding-fine-tuning/)
52. Maarten Sap, Ronan LeBras, Daniel Fried, and Yejin Choi. 2023. Neural theory-of-mind? on the limits of social intelligence in large lms. Preprint, arXiv:2210.13312. [↗](https://arxiv.org/abs/2210.13312)
53. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. SocialIQA: Commonsense reasoning about social interactions. In EMNLP. [our card](https://md.research.elamur.ai/free/en/papers/sap-2019-socialiqa-commonsense-reasoning-about-social/)
54. Melanie Sclar, Sachin Kumar, Peter West, Alane Suhr, Yejin Choi, and Yulia Tsvetkov. 2023. Minding language models’ (lack of) theory of mind: A plug-and-play multi-character belief tracker. Preprint, arXiv:2306.00924. [↗](https://arxiv.org/abs/2306.00924)
55. John R Searle. 1975. Indirect speech acts. In Speech acts, pages 59–82. Brill. [find](https://scholar.google.com/scholar?q=John+R+Searle%2E+1975%2E+Indirect+speech+acts%2E+In+Speech+acts%2C+pages+59%E2%80%9382%2E+Brill%2E)
56. Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. 2023. Clever hans or neural theory of mind? stress testing social reasoning in large language models. Preprint, arXiv:2305.14763. [↗](https://arxiv.org/abs/2305.14763)
57. Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2023. Distilling reasoning capabilities into smaller language models. Preprint, arXiv:2212.00193. [↗](https://arxiv.org/abs/2212.00193) [our card](https://md.research.elamur.ai/free/en/papers/shridhar-2023-distilling-reasoning-capabilities-into-smaller/)
58. Dan Sperber and Deirdre Wilson. 1986. Relevance: Communication and cognition, volume 142. Harvard University Press Cambridge, MA. [find](https://scholar.google.com/scholar?q=Dan+Sperber+and+Deirdre+Wilson%2E+1986%2E+Relevance%3A+Communication+and+cognition%2C+volume+142%2E+Harvard+University+Press+Cambridge%2C+MA%2E)
59. Settaluri Lakshmi Sravanthi, Meet Doshi, Tankala Pavan Kalyan, Rudra Murthy, Pushpak Bhattacharyya, and Raj Dabre. 2024. Pub: A pragmatics understanding benchmark for assessing llms’ pragmatics capabilities. Preprint, arXiv:2401.07078. [↗](https://arxiv.org/abs/2401.07078) [our card](https://md.research.elamur.ai/free/en/papers/sravanthi-2024-pub-pragmatics-understanding-benchmark-assessing/)
60. Ece Takmaz, Nicolo’ Brandizzi, Mario Giulianelli, Sandro Pezzelle, and Raquel Fernández. 2023. Speaking the language of your listener: Audience-aware adaptation via plug-and-play theory of mind. Preprint, arXiv:2305.19933. [↗](https://arxiv.org/abs/2305.19933)
61. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Moly-bog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. Preprint, arXiv:2307.09288. [↗](https://arxiv.org/abs/2307.09288)
62. Tomer Ullman. 2023. Large language models fail on trivial alterations to theory-of-mind tasks. Preprint, arXiv:2302.08399. [↗](https://arxiv.org/abs/2302.08399)
63. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
64. Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. 2023a. Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization. arXiv preprint arXiv:2306.05087. [↗](https://arxiv.org/abs/2306.05087) [our card](https://md.research.elamur.ai/free/en/papers/wang-2023-pandalm-automatic-evaluation-benchmark-llm/)
65. Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023b. How far can camels go? exploring the state of instruction tuning on open resources. Preprint, arXiv:2306.04751. [↗](https://arxiv.org/abs/2306.04751) [our card](https://md.research.elamur.ai/free/en/papers/wang-2023-far-camels-go-exploring-state/)
66. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837. [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
67. Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364. [↗](https://arxiv.org/abs/2304.06364) [our card](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/)
68. Pei Zhou, Aman Madaan, Srividya Pranavi Potharaju, Aditya Gupta, Kevin R. McKee, Ari Holtzman, Jay Pujara, Xiang Ren, Swaroop Mishra, Aida Ne-matzadeh, Shyam Upadhyay, and Manaal Faruqui. 2023. How far are large language models from agents with theory-of-mind? Preprint, arXiv:2310.03051. [↗](https://arxiv.org/abs/2310.03051)
69. Hao Zhu, Graham Neubig, and Yonatan Bisk. 2021. Few-shot language coordination by modeling theory of mind. Preprint, arXiv:2107.05697. [↗](https://arxiv.org/abs/2107.05697)
70. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593. [↗](https://arxiv.org/abs/1909.08593)