---
title: "Talk Less, Interact Better — paper by Hua and Artzi (2024)"
description: "The authors build ICCA, an automated evaluation framework based on repeated reference games. 43 references, cited by 5 papers here."
authors: ["Yilun Hua", "Yoav Artzi"]
year: 2024
venue: "arXiv:2408.01417"
source: "https://arxiv.org/abs/2408.01417"
doi: "10.48550/arxiv.2408.01417"
arxiv: "2408.01417"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hua-2024-talk-less-interact-better-evaluating/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hua-2024-talk-less-interact-better-evaluating/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/11/) / Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs

# Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs

**Authors:** [Yilun Hua](https://md.research.elamur.ai/free/en/authors/yilun-hua/) and [Yoav Artzi](https://md.research.elamur.ai/free/en/authors/yoav-artzi/) · **Year:** 2024 · **Venue:** arXiv:2408.01417 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2408.01417) · [PDF](https://arxiv.org/pdf/2408.01417) · [DOI 10.48550/arxiv.2408.01417](https://doi.org/10.48550/arxiv.2408.01417) · [arXiv 2408.01417](https://arxiv.org/abs/2408.01417) · [OpenAlex](https://openalex.org/W4406302072)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [5](https://md.research.elamur.ai/free/en/papers/hua-2024-talk-less-interact-better-evaluating/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
When people refer to the same things repeatedly, they gradually shorten their descriptions and settle on shared, ad-hoc conventions. This paper asks whether multimodal LLMs show the same drive toward efficiency when the entire conversation stays in their context. The authors build ICCA, an automated evaluation framework based on repeated reference games. In each game, a speaker describes one of four similar images and a listener must identify it, over six rounds. ICCA reuses recorded human-human games from Hawkins et al. (2020b), so no new human studies are needed. Models are tested both as speakers, paired with a GPT4 listener, and as listeners, fed recorded human messages. The framework tracks message length, lexical convergence and selection accuracy. None of the five tested MLLMs spontaneously shortens its own messages. Only the strongest ones do so under heavily engineered prompts, and even then their wording keeps drifting instead of stabilizing. As listeners, models cope better with increasingly terse human messages, and GPT4 is the most human-like. The results expose a gap in current training that matters for cheaper, more natural dialogue systems.

## Key points
- **Data and setup:** ICCA (In-context Conversational Adaptation) uses 54 human-human interactions from Hawkins et al. (2020b). Each game has 4 images and 6 repetitions, giving 24 trials. ICCA also supports variants, such as showing the image context only once.
- **Models evaluated:** The five models tested are IDEFICS, LLaVa-1.5, GPT4-vision, Claude 3 opus, plus Gemini 1.0 Pro Vision.
- **Speaker results:** GPT4, Gemini and Claude shorten messages only with explicit, game-specific instructions. They often introduce new words instead of converging on a stable label, so true conventions do not form.
- **Repetition bias:** IDEFICS and LLaVa assign higher likelihood to artificial transcripts that repeat the first-round messages than to the real, evolving human transcripts. This held in all 54 interactions.
- **Multi-image weakness and release:** IDEFICS loses accuracy as the number of images in its input grows beyond about four, which the authors partly attribute to its Flamingo-style cross-attention. ICCA is released under the MIT license at github.com/lil-lab/ICCA.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Cornell University](https://md.research.elamur.ai/free/en/universities/cornell-university/) — University, Ithaca, United States · 120 works · 59,940 citations
- [Cornell Tech](https://md.research.elamur.ai/free/en/universities/cornell-tech/) — University, New York, United States · 7 works · 1,320 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 624 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 657 works

## Cite

Cite as: Yilun Hua and Yoav Artzi. 2024. Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs. arXiv:2408.01417.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{hua2024talk,
  title = {{Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs}},
  author = {Yilun Hua and Yoav Artzi},
  year = {2024},
  doi = {10.48550/arxiv.2408.01417},
  eprint = {2408.01417},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2408.01417}
}
```

- [← previous Syntactic Language Change in English and German: Metrics, Parsers, and Convergences](https://md.research.elamur.ai/free/en/papers/chen-2024-syntactic-language-change-english-german/)
- [next → The Evolution of Darija Open Dataset: Introducing Version 2](https://md.research.elamur.ai/free/en/papers/outchakoucht-2024-evolution-darija-open-dataset-introducing/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/11/)

## References

43 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew B… — [Flamingo: a Visual Language Model for Few-Shot Learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/)
2. Jacob Andreas and Dan Klein. [Reasoning about pragmatics with neural listeners and speakers](https://md.research.elamur.ai/free/en/papers/andreas-2016-reasoning-about-pragmatics-neural-listeners/). In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2016. URL [https://www.aclweb.org/anthology/D16-1125](https://www.aclweb.org/anthology/D16-1125). [↗](https://www.aclweb.org/anthology/D16-1125)
3. Chinmaya Andukuri, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D. Goodman. Star-gate: Teaching language models to ask clarifying questions. arXiv, 2024. URL https://arxiv.org/abs/2403.19154. [↗](https://arxiv.org/abs/2403.19154)
4. Anthropic. Introducing the next generation of Claude, 2024. URL [https://www.anthropic.com/news/claude-3-family](https://www.anthropic.com/news/claude-3-family). [↗](https://www.anthropic.com/news/claude-3-family)
5. Susan E. Brennan and Herbert H. Clark. Conceptual pacts and lexical choice in conversation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22, 1996. [find](https://scholar.google.com/scholar?q=Susan+E%2E+Brennan+and+Herbert+H%2E+Clark%2E+Conceptual+pacts+and+lexical+choice+in+conversation%2E+Journal+of+Experimental+Psychology%3A+Learning%2C+Memory%2C+and+Cognition%2C+22%2C+1996%2E)
6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
7. Kranti Chalamalasetti, Jana Götze, Sherzod Hakimov, Brielen Madureira, Philipp Sadler, and David Schlangen. Clembench: Using game play to evaluate chat-optimized language models as conversational agents. arXiv, 2023. URL https://arxiv.org/abs/2305.13455. [↗](https://arxiv.org/abs/2305.13455)
8. Herbert H. Clark. Common ground. Cambridge University Press, 1996. [find](https://scholar.google.com/scholar?q=Herbert+H%2E+Clark%2E+Common+ground%2E+Cambridge+University+Press%2C+1996%2E)
9. Herbert H. Clark and Susan E. Brennan. Grounding in communication. In Perspectives on Socially Shared Cognition. American Psychological Association, 1991. [find](https://scholar.google.com/scholar?q=Herbert+H%2E+Clark+and+Susan+E%2E+Brennan%2E+Grounding+in+communication%2E+In+Perspectives+on+Socially+Shared+Cognition%2E+American+Psychological+Association%2C+1991%2E)
10. Herbert H. Clark and Deanna Wilkes-Gibbs. Referring as a collaborative process. Cognition, 22, 1986. URL [https://www.sciencedirect.com/science/article/pii/](https://www.sciencedirect.com/science/article/pii/) 0010027786900107. [↗](https://www.sciencedirect.com/science/article/pii/)
11. Philip R. Cohen and Hector J. Levesque. Rational Interaction as the Basis for Communication. The MIT Press. URL https://doi.org/10.7551/mitpress/3839.003.0014. Marco Del Tredici, Xiaoyu Shen, Gianni Barlacchi, Bill Byrne, and Adrià de Gispert. From rewriting to remembering: Common ground for conver… [↗](https://doi.org/10.7551/mitpress/3839.003.0014)
12. Anna Effenberger, Rhia Singh, Eva Yan, Alane Suhr, and Yoav Artzi. Analysis of language change in collaborative instruction following. In Findings of the Association for Computational Linguistics: EMNLP, 2021. URL https://aclanthology.org/2021.findings-emnlp. 239.pdf. [↗](https://aclanthology.org/2021.findings-emnlp)
13. Ron Eliav, Anya Ji, Yoav Artzi, and Robert Hawkins. Semantic uncertainty guides the extension of conventions to new referents. arXiv, 2023. URL http://arxiv.org/abs/2305. 06539. [↗](http://arxiv.org/abs/2305)
14. Daniel Fried, Jacob Andreas, and Dan Klein. [Unified pragmatic models for generating and following instructions](https://md.research.elamur.ai/free/en/papers/fried-2018-unified-pragmatic-models-generating-following/). In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018. URL [https://www.aclweb.org/anthology/N18…](https://www.aclweb.org/anthology/N18…) [↗](https://www.aclweb.org/anthology/N18…)
15. Edward Gibson, Richard Futrell, Steven P. Piantadosi, Isabelle Dautriche, Kyle Mahowald, Leon Bergen, and Roger Levy. How efficiency shapes human language. Trends in Cognitive Sciences, 23, 2019. URL [https://www.sciencedirect.com/science/article/pii/](https://www.sciencedirect.com/science/article/pii/) S1364661319300580. [↗](https://www.sciencedirect.com/science/article/pii/)
16. Noah D. Goodman and Michael C. Frank. Pragmatic language interpretation as probabilistic inference. Trends in Cognitive Sciences, 20, 2016. URL [https://api.semanticscholar.org/](https://api.semanticscholar.org/) CorpusID:3632786. [↗](https://api.semanticscholar.org/)
17. Google. Introducing Gemini: Our largest and most capable AI model, 2023. URL [https://blog.google/technology/ai/google-gemini-ai/](https://blog.google/technology/ai/google-gemini-ai/). [↗](https://blog.google/technology/ai/google-gemini-ai/)
18. Barbara J. Grosz and Candace L. Sidner. Plans for discourse. In Intentions in Communication. The MIT Press. URL https://doi.org/10.7551[/mitpress/3839.003.0022](https://md.research.elamur.ai/free/en/papers/haber-2019-photobook-dataset-building-common-ground/). Janosch Haber, Tim Baumgärtner, Ece Takmaz, Lieke Gelderloos, Elia Bruni, and Raquel Fernández. The PhotoBook dataset: Building common gro… [↗](https://doi.org/10.7551/mitpress/3839.003.0022)
19. Robert Hawkins, Michael C. Frank, and Noah D. Goodman. [Characterizing the dynamics of learning in repeated reference games](https://md.research.elamur.ai/free/en/papers/hawkins-2019-characterizing-dynamics-learning-repeated-reference/). Cognitive Science, 44, 2020a. URL [https://onlinelibrary.wiley.com/doi/abs/10.1111/cogs.12845](https://onlinelibrary.wiley.com/doi/abs/10.1111/cogs.12845). [↗](https://onlinelibrary.wiley.com/doi/abs/10.1111/cogs.12845)
20. Robert Hawkins, Minae Kwon, Dorsa Sadigh, and Noah Goodman. [Continual adaptation for efficient machine communication](https://md.research.elamur.ai/free/en/papers/hawkins-2020-continual-adaptation-efficient-machine-communication/). In Proceedings of the Conference on Computational Natural Language Learning, November 2020b.
21. William S. Horton and Richard J. Gerrig. Revisiting the memory-based processing approach to common ground. Topics in Cognitive Science, 8, 2016. URL [https://onlinelibrary.wiley.com/doi/full/10.1111/tops.12216](https://onlinelibrary.wiley.com/doi/full/10.1111/tops.12216). — [doi:10.1111/tops.12216](https://doi.org/10.1111/tops.12216) [↗](https://onlinelibrary.wiley.com/doi/full/10.1111/tops.12216)
22. Huggingface. HuggingFaceM4/idefics-80b-instruct, 2023. URL [https://huggingface.co/](https://huggingface.co/) HuggingFaceM4/idefics-80b-instruct. [↗](https://huggingface.co/)
23. Anya Ji, Noriyuki Kojima, Noah Rush, Alane Suhr, Wai Keen Vong, Robert Hawkins, and Yoav Artzi. Abstract visual reasoning with tangram shapes. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, December 2022. URL https://aclanthology.org/2022.emnlp-main.38. [↗](https://aclanthology.org/2022.emnlp-main.38)
24. Robert M. Krauss and Sidney Weinheimer. Changes in reference phrases as a function of frequency of usage in social interaction: A preliminary study. Psychonomic Science, 1, 1964. URL [https://link.springer.com/article/10.3758/BF03342817](https://link.springer.com/article/10.3758/BF03342817). — [doi:10.3758/bf03342817](https://doi.org/10.3758/bf03342817) [↗](https://link.springer.com/article/10.3758/BF03342817)
25. Robert M. Krauss and Sidney Weinheimer. Concurrent feedback, confirmation, and the encoding of referents in verbal communication. Journal of Personality and Social Psychology, 4, 1966. [find](https://scholar.google.com/scholar?q=Robert+M%2E+Krauss+and+Sidney+Weinheimer%2E+Concurrent+feedback%2C+confirmation%2C+and+the+encoding+of+referents+in+verbal+communication%2E+Journal+of+Personality+and+Social+Psychology%2C+4%2C+1966%2E)
26. David Kellogg Lewis. Convention: A Philosophical Study. Wiley-Blackwell, 1969. [find](https://scholar.google.com/scholar?q=David+Kellogg+Lewis%2E+Convention%3A+A+Philosophical+Study%2E+Wiley%2DBlackwell%2C+1969%2E)
27. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. [Improved baselines with visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/). arXiv, 2023a. URL [https://arxiv.org/pdf/2310.03744](https://arxiv.org/pdf/2310.03744). [↗](https://arxiv.org/pdf/2310.03744)
28. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). Transactions of the Association for Computational Linguistics, 12, 2023b. URL https://aclanthology.org/ 2024.tacl-1.9.pdf. [↗](https://aclanthology.org/)
29. Arthur B. Markman and Valerie S. Makin. Referential communication and category acquisition. Journal of Experimental Psychology: General, 127, 1998. [find](https://scholar.google.com/scholar?q=Arthur+B%2E+Markman+and+Valerie+S%2E+Makin%2E+Referential+communication+and+category+acquisition%2E+Journal+of+Experimental+Psychology%3A+General%2C+127%2C+1998%2E)
30. Bill McDowell and Noah D. Goodman. Learning from omission. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2019. URL https://aclanthology.org/P19-1059.pdf. [↗](https://aclanthology.org/P19-1059.pdf)
31. Charles Metzing and Susan E. Brennan. When conceptual pacts are broken: Partner-specific effects on the comprehension of referring expressions. Journal of Memory and Language, 49, 2003. URL [https://www.sciencedirect.com/science/article/pii/S0749596X03000287](https://www.sciencedirect.com/science/article/pii/S0749596X03000287). [↗](https://www.sciencedirect.com/science/article/pii/S0749596X03000287)
32. Will Monroe, Robert Hawkins, Noah D. Goodman, and Christopher Potts. [Colors in context: A pragmatic neural model for grounded language understanding](https://md.research.elamur.ai/free/en/papers/monroe-2017-colors-context-pragmatic-neural-model/). Transactions of the Association for Computational Linguistics, 5, 2017. URL https://aclanthology.org/ Q17-1023.pdf. [↗](https://aclanthology.org/)
33. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… — [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
34. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Low… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
35. Jeffrey Pennington, Richard Socher, and Christopher Manning. [Glove: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2014. URL [http://www.aclweb.org/anthology/D14-1162](http://www.aclweb.org/anthology/D14-1162). [↗](http://www.aclweb.org/anthology/D14-1162)
36. Omar Shaikh, Kristina Gligoric, Ashna Khetan, Matthias Gerstgrasser, Diyi Yang, and Dan Jurafsky. Grounding gaps in language model generations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, June 2024. URL… [find](https://scholar.google.com/scholar?q=Omar+Shaikh%2C+Kristina+Gligoric%2C+Ashna+Khetan%2C+Matthias+Gerstgrasser%2C+Diyi+Yang%2C+and+Dan+Jurafsky%2E+Grounding+gaps+in+language+model+generations%2E+In+Proceedings+of+the+Conference+of+the+North+American+C)
37. Robert Stalnaker. Common ground. Linguistics and Philosophy, 25, 2002. URL [https://link.springer.com/article/10.1023/A:1020867916902](https://link.springer.com/article/10.1023/A:1020867916902). [↗](https://link.springer.com/article/10.1023/A:1020867916902)
38. Alberto Testoni and Raquel Fernández. Asking the right question at the right time: Human and model uncertainty guidance to ask clarification questions. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics, 2024. URL https://aclanthology.org/2024.… [↗](https://aclanthology.org/2024.…)
39. David Traum. A Computational Theory of Grounding in Natural Language Conversation. PhD thesis, University of Rochester, 1994. [find](https://scholar.google.com/scholar?q=David+Traum%2E+A+Computational+Theory+of+Grounding+in+Natural+Language+Conversation%2E+PhD+thesis%2C+University+of+Rochester%2C+1994%2E)
40. Julia White, Jesse Mu, and Noah D. Goodman. Learning to refer informatively by amortizing pragmatic reasoning. In Proceedings of the Annual Meeting of the Cognitive Science Society, 2020. URL [https://cognitivesciencesociety.org/cogsci20/papers/0175/0175.pdf](https://cognitivesciencesociety.org/cogsci20/papers/0175/0175.pdf). [↗](https://cognitivesciencesociety.org/cogsci20/papers/0175/0175.pdf)
41. Kayo Yin, Terry Regier, and Dan Klein. American sign language handshapes reflect pressures for communicative efficiency. arXiv, 2024. URL https://arxiv.org/abs/2406.04024. [↗](https://arxiv.org/abs/2406.04024)
42. Hao Zhu, Graham Neubig, and Yonatan Bisk. Few-shot language coordination by modeling theory of mind. In Proceedings of the International Conference on Machine Learning, 2021. URL [https://proceedings.mlr.press/v139/zhu21d/zhu21d.pdf](https://proceedings.mlr.press/v139/zhu21d/zhu21d.pdf). [↗](https://proceedings.mlr.press/v139/zhu21d/zhu21d.pdf)
43. George Kingsley Zipf. Human Behavior and the Principle of Least Effort: An Introduction to Human Ecology. Addison-Wesley Press, 1949. [find](https://scholar.google.com/scholar?q=George+Kingsley+Zipf%2E+Human+Behavior+and+the+Principle+of+Least+Effort%3A+An+Introduction+to+Human+Ecology%2E+Addison%2DWesley+Press%2C+1949%2E)

*Updated 1 November 2026*