---
title: "Do Large Language Models Understand Conversational Implicature- A case study — Yue 2024"
description: "The authors test whether large language models (LLMs) understand conversational implicature, that is, the… 50 references, cited by 11 papers here."
authors: ["Shisen Yue", "Siyuan Song", "Xinyuan Cheng", "Hai Hu"]
year: 2024
venue: "Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 1: Main Conference)"
source: "https://aclanthology.org/2024.ccl-1.98/"
arxiv: "2404.19509"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yue-2024-do-large-language-models-understand/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yue-2024-do-large-language-models-understand/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Do Large Language Models Understand Conversational Implicature- A case study with a Chinese sitcom

# Do Large Language Models Understand Conversational Implicature- A case study with a Chinese sitcom

**Authors:** [Shisen Yue](https://md.research.elamur.ai/free/en/authors/shisen-yue/), Siyuan Song, Xinyuan Cheng and [Hai Hu](https://md.research.elamur.ai/free/en/authors/hai-hu/) · **Year:** 2024 · **Venue:** Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 1: Main Conference) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.ccl-1.98/) · [PDF](https://aclanthology.org/2024.ccl-1.98.pdf) · [arXiv 2404.19509](https://arxiv.org/abs/2404.19509)

**Citations:** [11](https://md.research.elamur.ai/free/en/papers/yue-2024-do-large-language-models-understand/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
The authors test whether large language models (LLMs) understand conversational implicature, that is, the indirect, non-literal meaning of an utterance, in Chinese. To this end, they created SwordsmanImp, the first Chinese dataset of this kind, built on multi-turn dialogues from the sitcom My Own Swordsman. Linguists manually selected 200 excerpts. Each contains one utterance with implied meaning, annotated with the Gricean maxims and submaxims it violates. Each excerpt comes with four interpretations: a pragmatic one, a literal one, and two false ones that merely sound plausible. Eight closed and open models were tested on two tasks: choosing the correct interpretation and freely explaining the implicature, with the explanations rated by humans. GPT-4 reached human-level performance, while the remaining models lagged noticeably behind. Most models produce fluent text but explain the implicature unconvincingly. The type of violated maxim had no effect on accuracy. The work fills a gap in the evaluation of pragmatics beyond English. It also shows that multiple-choice tests are insufficient and that free-form generation must be evaluated as well.

## Key points
- **SwordsmanImp**: 200 questions based on dialogues from My Own Swordsman. The annotation follows the four maxims (Quality, Quantity, Relation, Manner) at the submaxim level. The data and code are available on GitHub (`sjtu-compling/llm-pragmatics`).
- **Multiple-choice task** (4 GPT-series models and 4 open models were tested): GPT-4 scores 94%, the same as humans; CausalLM (14B) scores 78.5%; GPT-3.5 and the other open models score between 20% and 60%.
- **Explanation task**: five models explained 32 dialogues, and humans rated the explanations for reasonableness, logic, and fluency. Only GPT-4 received a high reasonableness score. CausalLM (14B), despite its good accuracy on the first task, inserted English words and meaningless character sequences into its explanations.
- Neither for humans nor for models did accuracy depend on which maxim was violated. The authors attribute this to the fact that many dialogues violate several maxims simultaneously.
- On the English dataset from Hu et al. (2023a), text-davinci-002 gets above 60%, yet on SwordsmanImp it stays under 25%. This indicates that the new dataset is more difficult.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [😉 Implicature](https://md.research.elamur.ai/free/en/topics/implicature/) · [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/)

## Cite

Cite as: Shisen Yue, Siyuan Song, Xinyuan Cheng and Hai Hu. 2024. Do Large Language Models Understand Conversational Implicature- A case study with a Chinese sitcom. Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 1: Main Conference).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yue2024do,
  title = {{Do Large Language Models Understand Conversational Implicature- A case study with a Chinese sitcom}},
  author = {Shisen Yue, Siyuan Song, Xinyuan Cheng and Hai Hu},
  booktitle = {Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 1: Main Conference)},
  year = {2024},
  eprint = {2404.19509},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.19509}
}
```

- [← previous Dialogue Discourse Parsing as Generation: A Sequence-to-Sequence LLM-based Approach](https://md.research.elamur.ai/free/en/papers/li-2024-dialogue-discourse-parsing-as-generation/)
- [next → Pragmatic Competence Evaluation of Large Language Models for the Korean Language](https://md.research.elamur.ai/free/en/papers/park-2024-pragmatic-competence-evaluation-large-language/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

50 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, P… [our card](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/)
2. Proceedings of the 23rd China National Conference on Computational Linguistics, pages 1270-1285, Taiyuan, China, July 25 - 28, 2024. Volume 1: Main Conference Papers (c) Technical Committee on Computational Linguistics, Chinese Information Processing Society of China 1279 [find](https://scholar.google.com/scholar?q=Proceedings+of+the+23rd+China+National+Conference+on+Computational+Linguistics%2C+pages+1270%2D1285%2C+Taiyuan%2C+China%2C+July+25+%2D+28%2C+2024%2E+Volume+1%3A+Main+Conference+Papers+%28c%29+Technical+Committee+on+Computa)
3. Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 632–642, 2015. [find](https://scholar.google.com/scholar?q=Samuel+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D+Manning%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+)
4. Penelope Brown and Stephen C Levinson. Politeness: Some universals in language usage, volume 4. Cambridge university press, 1987. [find](https://scholar.google.com/scholar?q=Penelope+Brown+and+Stephen+C+Levinson%2E+Politeness%3A+Some+universals+in+language+usage%2C+volume+4%2E+Cambridge+university+press%2C+1987%2E)
5. Yuyan Chen, Zhixu Li, Jiaqing Liang, Yanghua Xiao, Bang Liu, and Yunwen Chen. Can pre-trained language models understand chinese humor? In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, pp. 465–480, 2023. [find](https://scholar.google.com/scholar?q=Yuyan+Chen%2C+Zhixu+Li%2C+Jiaqing+Liang%2C+Yanghua+Xiao%2C+Bang+Liu%2C+and+Yunwen+Chen%2E+Can+pre%2Dtrained+language+models+understand+chinese+humor%3F+In+Proceedings+of+the+Sixteenth+ACM+International+Conference+on+)
6. Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. MuTual: A dataset for multi-turn dialogue reasoning. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1406–1416, Online, July… [our card](https://md.research.elamur.ai/free/en/papers/cui-2020-mutual-dataset-multi-turn-dialogue/)
7. Yiming Cui, Ziqing Yang, and Xin Yao. Efficient and effective text encoding for chinese llama and alpaca, 2023. [our card](https://md.research.elamur.ai/free/en/papers/cui-2023-efficient-effective-text-encoding-chinese/)
8. Paul E Engelhardt, Karl GD Bailey, and Fernanda Ferreira. Do speakers and listeners observe the gricean maxim of quantity? Journal of memory and language, 54(4):554–573, 2006. [find](https://scholar.google.com/scholar?q=Paul+E+Engelhardt%2C+Karl+GD+Bailey%2C+and+Fernanda+Ferreira%2E+Do+speakers+and+listeners+observe+the+gricean+maxim+of+quantity%3F+Journal+of+memory+and+language%2C+54%284%29%3A554%E2%80%93573%2C+2006%2E)
9. Sammy Floyd, Edward Gibson, and Evelina Fedorenko. Pragmega. Retrieved from [https://osf.io/](https://osf.io/) dpge6, September 2023. [↗](https://osf.io/)
10. Herbert P Grice. Logic and conversation. In Speech acts, pp. 41–58. Brill, 1975. [find](https://scholar.google.com/scholar?q=Herbert+P+Grice%2E+Logic+and+conversation%2E+In+Speech+acts%2C+pp%2E+41%E2%80%9358%2E+Brill%2C+1975%2E)
11. Herbert Paul Grice. Retrospective epilogue. Studies in the Way of Words, 339:386, 1989. [find](https://scholar.google.com/scholar?q=Herbert+Paul+Grice%2E+Retrospective+epilogue%2E+Studies+in+the+Way+of+Words%2C+339%3A386%2C+1989%2E)
12. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020. [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
13. Jack Hessel, Ana Marasovic, Jena D. Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff, and Yejin Choi. Do androids laugh at electric sheep? humor “understanding” benchmarks from the new yorker caption contest. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the… [find](https://scholar.google.com/scholar?q=Jack+Hessel%2C+Ana+Marasovic%2C+Jena+D%2E+Hwang%2C+Lillian+Lee%2C+Jeff+Da%2C+Rowan+Zellers%2C+Robert+Mankoff%2C+and+Yejin+Choi%2E+Do+androids+laugh+at+electric+sheep%3F+humor+%E2%80%9Cunderstanding%E2%80%9D+benchmarks+from+the+new+yorke)
14. J.L.B. Hirschberg. A Theory of Scalar Implicature. University of Pennsylvania, 1985. URL [https://books.google.com/books?id=bvEYAQAAIAAJ](https://books.google.com/books?id=bvEYAQAAIAAJ). [↗](https://books.google.com/books?id=bvEYAQAAIAAJ)
15. Jennifer Hu, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, and Edward Gibson. A fine-grained comparison of pragmatic language understanding in humans and language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for… [our card](https://md.research.elamur.ai/free/en/papers/hu-2023-fine-grained-comparison-pragmatic-language/)
16. Jennifer Hu, Roger Levy, Judith Degen, and Sebastian Schuster. Expectations over unspoken alternatives predict pragmatic inferences. arXiv preprint arXiv:2304.04758, 2023b. [our card](https://md.research.elamur.ai/free/en/papers/hu-2023-expectations-over-unspoken-alternatives-predict/)
17. Sophie Jentzsch and Kristian Kersting. Chatgpt is fun, but it is not funny! humor is still challenging large language models. arXiv preprint arXiv:2306.04563, 2023. [find](https://scholar.google.com/scholar?q=Sophie+Jentzsch+and+Kristian+Kersting%2E+Chatgpt+is+fun%2C+but+it+is+not+funny%21+humor+is+still+challenging+large+language+models%2E+arXiv+preprint+arXiv%3A2306%2E04563%2C+2023%2E)
18. Zae Myung Kim, David E Taylor, and Dongyeop Kang. ” is the pope catholic?” applying chain-of-thought reasoning to understanding conversational implicatures. arXiv preprint arXiv:2305.13826, 2023. [our card](https://md.research.elamur.ai/free/en/papers/kim-2023-pope-catholic-applying-chain-thought/)
19. Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328, 2018. doi: 10.1162/tacl_a_00023. URL [https://aclanthology.o…](https://aclanthology.o…) [↗](https://aclanthology.o…) [our card](https://md.research.elamur.ai/free/en/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/)
20. Proceedings of the 23rd China National Conference on Computational Linguistics, pages 1270-1285, Taiyuan, China, July 25 - 28, 2024. Volume 1: Main Conference Papers (c) Technical Committee on Computational Linguistics, Chinese Information Processing Society of China 1280 [find](https://scholar.google.com/scholar?q=Proceedings+of+the+23rd+China+National+Conference+on+Computational+Linguistics%2C+pages+1270%2D1285%2C+Taiyuan%2C+China%2C+July+25+%2D+28%2C+2024%2E+Volume+1%3A+Main+Conference+Papers+%28c%29+Technical+Committee+on+Computa)
21. Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. CMMLU: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212, 2023a. [our card](https://md.research.elamur.ai/free/en/papers/li-2023-cmmlu-measuring-massive-multitask-language/)
22. Hengli Li, Song-Chun Zhu, and Zilong Zheng. Diplomat: A dialogue dataset for situated pragmatic reasoning, 2023b. [find](https://scholar.google.com/scholar?q=Hengli+Li%2C+Song%2DChun+Zhu%2C+and+Zilong+Zheng%2E+Diplomat%3A+A+dialogue+dataset+for+situated+pragmatic+reasoning%2C+2023b%2E)
23. Benjamin Lipkin, Lionel Wong, Gabriel Grand, and Joshua B Tenenbaum. Evaluating statistical language models as pragmatic reasoners. arXiv preprint arXiv:2305.01020, 2023. [our card](https://md.research.elamur.ai/free/en/papers/lipkin-2023-evaluating-statistical-language-models-pragmatic/)
24. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel.… [our card](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/)
25. Arthur Neidlein, Philip Wiesenbach, and Katja Markert. An analysis of language models for metaphor recognition. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 3722–3736, 2020. [find](https://scholar.google.com/scholar?q=Arthur+Neidlein%2C+Philip+Wiesenbach%2C+and+Katja+Markert%2E+An+analysis+of+language+models+for+metaphor+recognition%2E+In+Proceedings+of+the+28th+International+Conference+on+Computational+Linguistics%2C+pp%2E+37)
26. Mako Okanda, Kosuke Asada, Yusuke Moriguchi, and Shoji Itakura. Understanding violations of gricean maxims in preschoolers and adults. Frontiers in psychology, 6:901, 2015. [find](https://scholar.google.com/scholar?q=Mako+Okanda%2C+Kosuke+Asada%2C+Yusuke+Moriguchi%2C+and+Shoji+Itakura%2E+Understanding+violations+of+gricean+maxims+in+preschoolers+and+adults%2E+Frontiers+in+psychology%2C+6%3A901%2C+2015%2E)
27. Lalchand Pandia, Yan Cong, and Allyson Ettinger. Pragmatic competence of pre-trained language models through the lens of discourse connectives. arXiv preprint arXiv:2109.12951, 2021. [our card](https://md.research.elamur.ai/free/en/papers/pandia-2021-pragmatic-competence-pre-trained-language/)
28. Francesca Panzeri and Francesca Foppolo. Children s and adults sensitivity to gricean maxims and to the maximize presupposition principle. Frontiers in Psychology, 12:624628, 2021. [find](https://scholar.google.com/scholar?q=Francesca+Panzeri+and+Francesca+Foppolo%2E+Children+s+and+adults+sensitivity+to+gricean+maxims+and+to+the+maximize+presupposition+principle%2E+Frontiers+in+Psychology%2C+12%3A624628%2C+2021%2E)
29. Badri N Patro, Mayank Lunayach, Deepankar Srivastava, Hunar Singh, Vinay P Namboodiri, et al. Multimodal humor dataset: Predicting laughter tracks for sitcoms. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 576–585, 2021. [find](https://scholar.google.com/scholar?q=Badri+N+Patro%2C+Mayank+Lunayach%2C+Deepankar+Srivastava%2C+Hunar+Singh%2C+Vinay+P+Namboodiri%2C+et+al%2E+Multimodal+humor+dataset%3A+Predicting+laughter+tracks+for+sitcoms%2E+In+Proceedings+of+the+IEEE%2FCVF+Winter+Co)
30. Zhuang Qiu, Xufeng Duan, and Zhenguang Cai. Does chatgpt resemble humans in processing implicatures? In Proceedings of the 4th Natural Logic Meets Machine Learning Workshop, pp. 25–34, 2023. [find](https://scholar.google.com/scholar?q=Zhuang+Qiu%2C+Xufeng+Duan%2C+and+Zhenguang+Cai%2E+Does+chatgpt+resemble+humans+in+processing+implicatures%3F+In+Proceedings+of+the+4th+Natural+Logic+Meets+Machine+Learning+Workshop%2C+pp%2E+25%E2%80%9334%2C+2023%2E)
31. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+Ilya+Sutskever%2C+et+al%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+2018%2E)
32. Siva Reddy, Danqi Chen, and Christopher D Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266, 2019. [our card](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/)
33. Paula Rubio-Fernandez. Overinformative speakers are cooperative: Revisiting the gricean maxim of quantity. Cognitive science, 43(11):e12797, 2019. [find](https://scholar.google.com/scholar?q=Paula+Rubio%2DFernandez%2E+Overinformative+speakers+are+cooperative%3A+Revisiting+the+gricean+maxim+of+quantity%2E+Cognitive+science%2C+43%2811%29%3Ae12797%2C+2019%2E)
34. Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, and Edward Grefenstette. Large language models are not zero-shot communicators. arXiv preprint arXiv:2210.14986, 2022. [our card](https://md.research.elamur.ai/free/en/papers/ruis-2022-goldilocks-pragmatic-understanding-fine-tuning/)
35. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Int… [find](https://scholar.google.com/scholar?q=Maarten+Sap%2C+Hannah+Rashkin%2C+Derek+Chen%2C+Ronan+Le+Bras%2C+and+Yejin+Choi%2E+Social+IQa%3A+Commonsense+reasoning+about+social+interactions%2E+In+Kentaro+Inui%2C+Jing+Jiang%2C+Vincent+Ng%2C+and+Xiaojun+Wan+%28eds%2E%29%2C+Pr)
36. John R Searle, Ferenc Kiefer, Manfred Bierwisch, et al. Speech act theory and pragmatics, volume 10. Springer, 1980. [find](https://scholar.google.com/scholar?q=John+R+Searle%2C+Ferenc+Kiefer%2C+Manfred+Bierwisch%2C+et+al%2E+Speech+act+theory+and+pragmatics%2C+volume+10%2E+Springer%2C+1980%2E)
37. Proceedings of the 23rd China National Conference on Computational Linguistics, pages 1270-1285, Taiyuan, China, July 25 - 28, 2024. Volume 1: Main Conference Papers (c) Technical Committee on Computational Linguistics, Chinese Information Processing Society of China 1281 [find](https://scholar.google.com/scholar?q=Proceedings+of+the+23rd+China+National+Conference+on+Computational+Linguistics%2C+pages+1270%2D1285%2C+Taiyuan%2C+China%2C+July+25+%2D+28%2C+2024%2E+Volume+1%3A+Main+Conference+Papers+%28c%29+Technical+Committee+on+Computa)
38. Lifeng Shang, Zhengdong Lu, and Hang Li. Neural responding machine for short-text conversation. In Chengqing Zong and Michael Strube (eds.), Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processi… [our card](https://md.research.elamur.ai/free/en/papers/shang-2015-neural-responding-machine-short-text/)
39. Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. DREAM: A challenge data set and models for dialogue-based reading comprehension. Transactions of the Association for Computational Linguistics, 7:217–231, 2019. doi: 10.1162/tacl_a_00264. URL https://aclanthology.org/Q19-1014. [↗](https://aclanthology.org/Q19-1014) [our card](https://md.research.elamur.ai/free/en/papers/sun-2019-dream-challenge-data-set-models/)
40. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+Llama+2%3A+Open+foundation+and+fine%2Dtun)
41. Lennart Wachowiak and Dagmar Gromann. Does gpt-3 grasp metaphors? identifying metaphor mappings with generative language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1018–1032, 2023. [find](https://scholar.google.com/scholar?q=Lennart+Wachowiak+and+Dagmar+Gromann%2E+Does+gpt%2D3+grasp+metaphors%3F+identifying+metaphor+mappings+with+generative+language+models%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Comput)
42. Xiaolong Wang, Rohit Girdhar, and Abhinav Gupta. Binge watching: Scaling affordance learning from sitcoms. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2596–2605, 2017. [find](https://scholar.google.com/scholar?q=Xiaolong+Wang%2C+Rohit+Girdhar%2C+and+Abhinav+Gupta%2E+Binge+watching%3A+Scaling+affordance+learning+from+sitcoms%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+pp%2E+2596%E2%80%9326)
43. Deirdre Wilson and Dan Sperber. Relevance theory. The handbook of pragmatics, pp. 606–632, 2006. [find](https://scholar.google.com/scholar?q=Deirdre+Wilson+and+Dan+Sperber%2E+Relevance+theory%2E+The+handbook+of+pragmatics%2C+pp%2E+606%E2%80%93632%2C+2006%2E)
44. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
45. Jiaming Wu, Hongfei Lin, Liang Yang, and Bo Xu. Mumor: A multimodal dataset for humor detection in conversations. In Natural Language Processing and Chinese Computing: 10th CCF International Conference, NLPCC 2021, Qingdao, China, October 13–17, 2021, Proceedings, Part I 10, pp. 619– 627. [find](https://scholar.google.com/scholar?q=Jiaming+Wu%2C+Hongfei+Lin%2C+Liang+Yang%2C+and+Bo+Xu%2E+Mumor%3A+A+multimodal+dataset+for+humor+detection+in+conversations%2E+In+Natural+Language+Processing+and+Chinese+Computing%3A+10th+CCF+International+Conferenc)
46. Yu Wu, Wei Wu, Chen Xing, Ming Zhou, and Zhoujun Li. Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatbots. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volu… [our card](https://md.research.elamur.ai/free/en/papers/wu-2017-sequential-matching-network-new-architecture/)
47. Zhuosheng Zhang, Jiangtong Li, Pengfei Zhu, Hai Zhao, and Gongshen Liu. Modeling multi-turn conversation with deep utterance aggregation. In Emily M. Bender, Leon Derczynski, and Pierre Isabelle (eds.), Proceedings of the 27th International Conference on Computational Linguistics, pp. 3740– 3752, Sa… [find](https://scholar.google.com/scholar?q=Zhuosheng+Zhang%2C+Jiangtong+Li%2C+Pengfei+Zhu%2C+Hai+Zhao%2C+and+Gongshen+Liu%2E+Modeling+multi%2Dturn+conversation+with+deep+utterance+aggregation%2E+In+Emily+M%2E+Bender%2C+Leon+Derczynski%2C+and+Pierre+Isabelle+%28eds%2E)
48. Zilong Zheng, Shuwen Qiu, Lifeng Fan, Yixin Zhu, and Song-Chun Zhu. Grice: A grammar-based dataset for recovering implicature and conversational reasoning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 2074–2085, 2021. [our card](https://md.research.elamur.ai/free/en/papers/zheng-2021-grice-grammar-based-dataset-recovering/)
49. A Evaluation paradigms and differences Models’ performance might vary from how their answers are estimated and collected. Next token prediction and free generation are two paradigms separately used to estimate the performance of open-source Proceedings of the 23rd China National Conference on Comput… [find](https://scholar.google.com/scholar?q=A+Evaluation+paradigms+and+differences+Models%E2%80%99+performance+might+vary+from+how+their+answers+are+estimated+and+collected%2E+Next+token+prediction+and+free+generation+are+two+paradigms+separately+used+to)
50. Model BlOOMZ OpenBuddy-Llama2 Chinese-Alpaca-2 CausalLM Total Next Gen Quantity Next Gen Accuracy (%) Quality Next Gen Relevance Next Gen Manner Next Gen 52.50 42.50 41.50 78.50 63.64 36.36 42.42 81.82 50.00 43.42 39.47 80.26 54.93 45.07 45.07 76.06 53.23 46.77 45.16 88.71 35.50 21.50 42.00 31.50 24… [find](https://scholar.google.com/scholar?q=Model+BlOOMZ+OpenBuddy%2DLlama2+Chinese%2DAlpaca%2D2+CausalLM+Total+Next+Gen+Quantity+Next+Gen+Accuracy+%28%25%29+Quality+Next+Gen+Relevance+Next+Gen+Manner+Next+Gen+52%2E50+42%2E50+41%2E50+78%2E50+63%2E64+36%2E36+42%2E42+81%2E8)