---
title: "LLMs Get Lost In Multi-Turn Conversation — Laban 2025"
description: "They build a \"sharded simulation\" environment from existing high-quality single-turn benchmarks such… 91 references, cited by 66 papers here."
authors: ["Philippe Laban", "Hiroaki Hayashi", "Yingbo Zhou", "Jennifer Neville"]
year: 2025
venue: "arXiv:2505.06120"
source: "https://arxiv.org/abs/2505.06120"
doi: "10.48550/arxiv.2505.06120"
arxiv: "2505.06120"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/laban-2025-llms-get-lost-multi-turn/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/laban-2025-llms-get-lost-multi-turn/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/) / LLMs Get Lost In Multi-Turn Conversation

# LLMs Get Lost In Multi-Turn Conversation

**Authors:** [Philippe Laban](https://md.research.elamur.ai/free/en/authors/philippe-laban/), [Hiroaki Hayashi](https://md.research.elamur.ai/free/en/authors/hiroaki-hayashi/), Yingbo Zhou and Jennifer Neville · **Year:** 2025 · **Venue:** arXiv:2505.06120 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2505.06120) · [PDF](https://arxiv.org/pdf/2505.06120) · [DOI 10.48550/arxiv.2505.06120](https://doi.org/10.48550/arxiv.2505.06120) · [arXiv 2505.06120](https://arxiv.org/abs/2505.06120) · [OpenAlex](https://openalex.org/W4417086184)

**Citations:** 11 citations (OpenAlex, 2026-10-14) · 66 in our library, of the 141,840 papers whose bibliographies we read

## Summary
Most LLM benchmarks give the model a complete instruction in a single turn. Real users, however, often begin vaguely and add details as the conversation goes on. Laban et al. measure how much this gap matters. They build a "sharded simulation" environment from existing high-quality single-turn benchmarks such as GSM8K. Each fully-specified instruction is split into shards: the first states the overall goal, and the others add one clarification each. An LLM-driven user simulator then reveals at most one shard per turn. The authors run more than 200,000 simulated conversations across six generation tasks covering both programming and natural language, testing 15 open- and closed-weight models. Every model scores much worse when information arrives gradually. Average performance falls from roughly 90% to 65%. The authors trace most of this decline to a sharp rise in unreliability rather than a loss of core ability. Models tend to commit early to assumptions and attempt answers too soon, and they then fail to correct course. The study argues that multi-turn reliability should be evaluated and optimized alongside raw capability.

## Key points
- The performance drop appears for all 15 models tested, from Llama3.1-8B-Instruct to Gemini 2.5 Pro, GPT-4.1, Claude 3.7 Sonnet, o3 and Deepseek-R1. The abstract reports an average decline of about 39%, and the drop appears even in two-turn conversations.
- The authors split performance into aptitude (best-case results) and unreliability (the gap between best and worst runs). In multi-turn settings, aptitude falls by about 15%, while unreliability rises by about 112%.
- They identify four failure behaviors:
  - overly long responses;
  - premature final answers;
  - wrong guesses about unstated details;
  - over-reliance on the model's own earlier, flawed attempts.
- Two fixes that help in simpler settings did not repair multi-turn behavior: agent-style concatenation of the conversation and lowering the generation temperature. As practical advice, the authors suggest that users restart the chat and merge all their requirements into a single prompt.
- The code and data are released as Microsoft/lost_in_conversation. The study covers only English, text-only, analytical tasks. The authors consider their idealized simulation likely to understate how often this degradation occurs in real use.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works

## Cite

Cite as: Philippe Laban, Hiroaki Hayashi, Yingbo Zhou and Jennifer Neville. 2025. LLMs Get Lost In Multi-Turn Conversation. arXiv:2505.06120.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{laban2025llms,
  title = {{LLMs Get Lost In Multi-Turn Conversation}},
  author = {Philippe Laban, Hiroaki Hayashi, Yingbo Zhou and Jennifer Neville},
  year = {2025},
  doi = {10.48550/arxiv.2505.06120},
  eprint = {2505.06120},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.06120}
}
```

- [← previous GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/team-2025-v-glm-4-5v-glm/)
- [next → Style Over Substance: Evaluation Biases for Large Language Models](https://md.research.elamur.ai/free/en/papers/wu-2025-style-over-substance-evaluation-biases/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/)

## References

91 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gunasekar, M. Harrison, R. J. Hewett, M. Javaheripi, P. Kauffmann, et al. [Phi-4 technical report](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-4-technical-report/). arXiv preprint arXiv:2412.08905, 2024.
2. G. Bai, J. Liu, X. Bu, Y. He, J. Liu, Z. Zhou, Z. Lin, W. Su, T. Ge, B. Zheng, et al. [Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues](https://md.research.elamur.ai/free/en/papers/bai-2024-mt-bench-101-fine-grained/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers…
3. C. G. Belem, P. Pezeskhpour, H. Iso, S. Maekawa, N. Bhutani, and E. Hruschka. From single to multi: How llms hallucinate in multi-document summarization. arXiv preprint arXiv:2410.13961, 2024. [find](https://scholar.google.com/scholar?q=C%2E+G%2E+Belem%2C+P%2E+Pezeskhpour%2C+H%2E+Iso%2C+S%2E+Maekawa%2C+N%2E+Bhutani%2C+and+E%2E+Hruschka%2E+From+single+to+multi%3A+How+llms+hallucinate+in+multi%2Ddocument+summarization%2E+arXiv+preprint+arXiv%3A2410%2E13961%2C+2024%2E)
4. P. Brauner, A. Hick, R. Philipsen, and M. Ziefle. What does the public think about artificial intelligence?—a criticality map to understand bias in the public perception of ai. In Frontiers of Computer Science, 2023. URL [https://api.semanticscholar.org/CorpusID:257598212](https://api.semanticscholar.org/CorpusID:257598212) . [↗](https://api.semanticscholar.org/CorpusID:257598212)
5. T. Chakrabarty, P. Laban, D. Agarwal, S. Muresan, and C.-S. Wu. Art or artifice? large language models and the false promise of creativity. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pages 1–34, 2024. [find](https://scholar.google.com/scholar?q=T%2E+Chakrabarty%2C+P%2E+Laban%2C+D%2E+Agarwal%2C+S%2E+Muresan%2C+and+C%2E%2DS%2E+Wu%2E+Art+or+artifice%3F+large+language+models+and+the+false+promise+of+creativity%2E+In+Proceedings+of+the+2024+CHI+Conference+on+Human+Factors+i)
6. T. Chakrabarty, P. Laban, and C.-S. Wu. Ai-slop to ai-polish? aligning language models through edit-based writing rewards and test-time computation. arXiv preprint arXiv:2504.07532, 2025. — [arXiv:2504.07532](https://arxiv.org/abs/2504.07532) [find](https://scholar.google.com/scholar?q=T%2E+Chakrabarty%2C+P%2E+Laban%2C+and+C%2E%2DS%2E+Wu%2E+Ai%2Dslop+to+ai%2Dpolish%3F+aligning+language+models+through+edit%2Dbased+writing+rewards+and+test%2Dtime+computation%2E+arXiv+preprint+arXiv%3A2504%2E07532%2C+2025%2E+%E2%80%94+%5BarXiv%3A250)
7. S. Chang, A. Anderson, and J. M. Hofman. Chatbench: From static benchmarks to human-ai evaluation. arXiv preprint arXiv:2504.07114, 2025. [find](https://scholar.google.com/scholar?q=S%2E+Chang%2C+A%2E+Anderson%2C+and+J%2E+M%2E+Hofman%2E+Chatbench%3A+From+static+benchmarks+to+human%2Dai+evaluation%2E+arXiv+preprint+arXiv%3A2504%2E07114%2C+2025%2E)
8. H. Chase. Langchain, October 2022. URL [https://github.com/langchain-ai/langchain](https://github.com/langchain-ai/langchain) . [↗](https://github.com/langchain-ai/langchain)
9. A. Chaturvedi, K. Thompson, and N. Asher. Nebula: A discourse aware minecraft builder. ArXiv, abs/2406.18164, 2024. URL [https://api.semanticscholar.org/CorpusID:270738020](https://api.semanticscholar.org/CorpusID:270738020) . [↗](https://api.semanticscholar.org/CorpusID:270738020)
10. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374, 2021.
11. W.-L. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, B. Zhu, H. Zhang, M. Jordan, J. E. Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conference on Machine Learning, 2024. [find](https://scholar.google.com/scholar?q=W%2E%2DL%2E+Chiang%2C+L%2E+Zheng%2C+Y%2E+Sheng%2C+A%2E+N%2E+Angelopoulos%2C+T%2E+Li%2C+D%2E+Li%2C+B%2E+Zhu%2C+H%2E+Zhang%2C+M%2E+Jordan%2C+J%2E+E%2E+Gonzalez%2C+et+al%2E+Chatbot+arena%3A+An+open+platform+for+evaluating+llms+by+human+preference%2E+In+Fort)
12. E. Choi, H. He, M. Iyyer, M. Yatskar, W.-t. Yih, Y. Choi, P. Liang, and L. Zettlemoyer. [Quac: Question answering in context](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/). arXiv preprint arXiv:1808.07036, 2018.
13. E. Choi, J. Palomaki, M. Lamm, T. Kwiatkowski, D. Das, and M. Collins. [Decontextualization: Making sentences stand-alone](https://md.research.elamur.ai/free/en/papers/choi-2021-decontextualization-making-sentences-stand-alone/). Transactions of the Association for Computational Linguistics, 9:447–461, 2021.
14. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
15. T. Cohere, A. Ahmadian, M. Ahmed, J. Alammar, Y. Alnumay, S. Althammer, A. Arkhangorodsky, V. Aryabumi, D. Aumiller, R. Avalos, et al. Command a: An enterprise-ready large language model. arXiv preprint arXiv:2504.00698, 2025. — [arXiv:2504.00698](https://arxiv.org/abs/2504.00698) [find](https://scholar.google.com/scholar?q=T%2E+Cohere%2C+A%2E+Ahmadian%2C+M%2E+Ahmed%2C+J%2E+Alammar%2C+Y%2E+Alnumay%2C+S%2E+Althammer%2C+A%2E+Arkhangorodsky%2C+V%2E+Aryabumi%2C+D%2E+Aumiller%2C+R%2E+Avalos%2C+et+al%2E+Command+a%3A+An+enterprise%2Dready+large+language+model%2E+arXiv+prepri)
16. Y. Deng, X. Zhang, W. Zhang, Y. Yuan, S.-K. Ng, and T.-S. Chua. On the multi-turn instruction following for conversational web agents. arXiv preprint arXiv:2402.15057, 2024. [find](https://scholar.google.com/scholar?q=Y%2E+Deng%2C+X%2E+Zhang%2C+W%2E+Zhang%2C+Y%2E+Yuan%2C+S%2E%2DK%2E+Ng%2C+and+T%2E%2DS%2E+Chua%2E+On+the+multi%2Dturn+instruction+following+for+conversational+web+agents%2E+arXiv+preprint+arXiv%3A2402%2E15057%2C+2024%2E)
17. J. Deriu, A. Rodrigo, A. Otegi, G. Echegoyen, S. Rosset, E. Agirre, and M. Cieliebak. Survey on evaluation methods for dialogue systems. Artificial Intelligence Review, 54:755–810, 2021. [find](https://scholar.google.com/scholar?q=J%2E+Deriu%2C+A%2E+Rodrigo%2C+A%2E+Otegi%2C+G%2E+Echegoyen%2C+S%2E+Rosset%2C+E%2E+Agirre%2C+and+M%2E+Cieliebak%2E+Survey+on+evaluation+methods+for+dialogue+systems%2E+Artificial+Intelligence+Review%2C+54%3A755%E2%80%93810%2C+2021%2E)
18. H. Duan, J. Wei, C. Wang, H. Liu, Y. Fang, S. Zhang, D. Lin, and K. Chen. Botchat: Evaluating llms’ capabilities of having multi-turn dialogues. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3184–3200, 2024. [find](https://scholar.google.com/scholar?q=H%2E+Duan%2C+J%2E+Wei%2C+C%2E+Wang%2C+H%2E+Liu%2C+Y%2E+Fang%2C+S%2E+Zhang%2C+D%2E+Lin%2C+and+K%2E+Chen%2E+Botchat%3A+Evaluating+llms%E2%80%99+capabilities+of+having+multi%2Dturn+dialogues%2E+In+Findings+of+the+Association+for+Computational+Lingui)
19. Z. Fan, R. Chen, T. Hu, and Z. Liu. Fairmt-bench: Benchmarking fairness for multi-turn dialogue in conversational llms. arXiv preprint arXiv:2410.19317, 2024. — [arXiv:2410.19317](https://arxiv.org/abs/2410.19317) [find](https://scholar.google.com/scholar?q=Z%2E+Fan%2C+R%2E+Chen%2C+T%2E+Hu%2C+and+Z%2E+Liu%2E+Fairmt%2Dbench%3A+Benchmarking+fairness+for+multi%2Dturn+dialogue+in+conversational+llms%2E+arXiv+preprint+arXiv%3A2410%2E19317%2C+2024%2E+%E2%80%94+%5BarXiv%3A2410%2E19317%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
20. V. S. Ferreira. Ambiguity, accessibility, and a division of labor for communicative success. Psychology of Learning and motivation, 49:209–246, 2008. [find](https://scholar.google.com/scholar?q=V%2E+S%2E+Ferreira%2E+Ambiguity%2C+accessibility%2C+and+a+division+of+labor+for+communicative+success%2E+Psychology+of+Learning+and+motivation%2C+49%3A209%E2%80%93246%2C+2008%2E)
21. S. E. Finch, J. D. Finch, and J. D. Choi. Don’t forget your abc’s: Evaluating the state-of-the-art in chat-oriented dialogue systems. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023. [find](https://scholar.google.com/scholar?q=S%2E+E%2E+Finch%2C+J%2E+D%2E+Finch%2C+and+J%2E+D%2E+Choi%2E+Don%E2%80%99t+forget+your+abc%E2%80%99s%3A+Evaluating+the+state%2Dof%2Dthe%2Dart+in+chat%2Doriented+dialogue+systems%2E+In+The+61st+Annual+Meeting+Of+The+Association+For+Computational+Li)
22. S. Frisson. Semantic underspecification in language processing. Lang. Linguistics Compass, 3:111–127, 2009. URL [https://api.semanticscholar.org/CorpusID:13384476](https://api.semanticscholar.org/CorpusID:13384476) . [↗](https://api.semanticscholar.org/CorpusID:13384476)
23. A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. [find](https://scholar.google.com/scholar?q=A%2E+Grattafiori%2C+A%2E+Dubey%2C+A%2E+Jauhri%2C+A%2E+Pandey%2C+A%2E+Kadian%2C+A%2E+Al%2DDahle%2C+A%2E+Letman%2C+A%2E+Mathur%2C+A%2E+Schelten%2C+A%2E+Vaughan%2C+et+al%2E+The+llama+3+herd+of+models%2E+arXiv+preprint+arXiv%3A2407%2E21783%2C+2024%2E)
24. D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948, 2025.
25. C. Han. Can language models follow multiple turns of entangled instructions? arXiv preprint arXiv:2503.13222, 2025. [find](https://scholar.google.com/scholar?q=C%2E+Han%2E+Can+language+models+follow+multiple+turns+of+entangled+instructions%3F+arXiv+preprint+arXiv%3A2503%2E13222%2C+2025%2E)
26. K. Handa, A. Tamkin, M. McCain, S. Huang, E. Durmus, S. Heck, J. Mueller, J. Hong, S. Ritchie, T. Belonax, et al. Which economic tasks are performed with ai? evidence from millions of claude conversations. arXiv preprint arXiv:2503.04761, 2025. — [arXiv:2503.04761](https://arxiv.org/abs/2503.04761) [find](https://scholar.google.com/scholar?q=K%2E+Handa%2C+A%2E+Tamkin%2C+M%2E+McCain%2C+S%2E+Huang%2C+E%2E+Durmus%2C+S%2E+Heck%2C+J%2E+Mueller%2C+J%2E+Hong%2C+S%2E+Ritchie%2C+T%2E+Belonax%2C+et+al%2E+Which+economic+tasks+are+performed+with+ai%3F+evidence+from+millions+of+claude+conversat)
27. C. Herlihy, J. Neville, T. Schnabel, and A. Swaminathan. On overcoming miscalibrated conversational priors in llm-based chatbots. arXiv preprint arXiv:2406.01633, 2024. — [arXiv:2406.01633](https://arxiv.org/abs/2406.01633) [find](https://scholar.google.com/scholar?q=C%2E+Herlihy%2C+J%2E+Neville%2C+T%2E+Schnabel%2C+and+A%2E+Swaminathan%2E+On+overcoming+miscalibrated+conversational+priors+in+llm%2Dbased+chatbots%2E+arXiv+preprint+arXiv%3A2406%2E01633%2C+2024%2E+%E2%80%94+%5BarXiv%3A2406%2E01633%5D%28https%3A%2F%2Far)
28. M. C. Horowitz, L. Kahn, J. Macdonald, and J. Schneider. Adopting ai: how familiarity breeds both trust and contempt. AI & society, 39(4):1721–1735, 2024. [find](https://scholar.google.com/scholar?q=M%2E+C%2E+Horowitz%2C+L%2E+Kahn%2C+J%2E+Macdonald%2C+and+J%2E+Schneider%2E+Adopting+ai%3A+how+familiarity+breeds+both+trust+and+contempt%2E+AI+%26+society%2C+39%284%29%3A1721%E2%80%931735%2C+2024%2E)
29. K.-H. Huang, P. Laban, A. R. Fabbri, P. K. Choubey, S. Joty, C. Xiong, and C.-S. Wu. Embrace divergence for richer insights: A multi-document summarization benchmark and a case study on summarizing diverse information from news articles. arXiv preprint arXiv:2309.09369, 2023. [find](https://scholar.google.com/scholar?q=K%2E%2DH%2E+Huang%2C+P%2E+Laban%2C+A%2E+R%2E+Fabbri%2C+P%2E+K%2E+Choubey%2C+S%2E+Joty%2C+C%2E+Xiong%2C+and+C%2E%2DS%2E+Wu%2E+Embrace+divergence+for+richer+insights%3A+A+multi%2Ddocument+summarization+benchmark+and+a+case+study+on+summarizing+di)
30. A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. [Gpt-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). arXiv preprint arXiv:2410.21276, 2024.
31. N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. [Livecodebench: Holistic and contamination free evaluation of large language models for code](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/). arXiv preprint arXiv:2403.07974, 2024.
32. M. Karpinska, K. Thai, K. Lo, T. Goyal, and M. Iyyer. One thousand and one pairs: A" novel" challenge for long-context language models. arXiv preprint arXiv:2406.16264, 2024. [find](https://scholar.google.com/scholar?q=M%2E+Karpinska%2C+K%2E+Thai%2C+K%2E+Lo%2C+T%2E+Goyal%2C+and+M%2E+Iyyer%2E+One+thousand+and+one+pairs%3A+A%22+novel%22+challenge+for+long%2Dcontext+language+models%2E+arXiv+preprint+arXiv%3A2406%2E16264%2C+2024%2E)
33. Y. Kim, Y. Chang, M. Karpinska, A. Garimella, V. Manjunatha, K. Lo, T. Goyal, and M. Iyyer. Fables: Evaluating faithfulness and content selection in book-length summarization. arXiv preprint arXiv:2404.01261, 2024. — [arXiv:2404.01261](https://arxiv.org/abs/2404.01261) [find](https://scholar.google.com/scholar?q=Y%2E+Kim%2C+Y%2E+Chang%2C+M%2E+Karpinska%2C+A%2E+Garimella%2C+V%2E+Manjunatha%2C+K%2E+Lo%2C+T%2E+Goyal%2C+and+M%2E+Iyyer%2E+Fables%3A+Evaluating+faithfulness+and+content+selection+in+book%2Dlength+summarization%2E+arXiv+preprint+arXiv%3A240)
34. Y. Kim, K. Son, S. Kim, and J. Kim. Beyond prompts: Learning from human communication for enhanced ai intent alignment. ArXiv, abs/2405.05678, 2024. URL [https://api.semanticscholar.org/CorpusID:269635257](https://api.semanticscholar.org/CorpusID:269635257) . — [arXiv:2405.05678](https://arxiv.org/abs/2405.05678) [↗](https://api.semanticscholar.org/CorpusID:269635257)
35. N. Knoth, A. Tolzin, A. Janson, and J. M. Leimeister. Ai literacy and its implications for prompt engineering strategies. Comput. Educ. Artif. Intell., 6:100225, 2024. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 269273689 . [↗](https://api.semanticscholar.org/CorpusID:)
36. J. Konrád, J. Pichl, P. Marek, P. Lorenc, V. D. Ta, O. Kobza, L. Hỳlová, and J. Šedivỳ. Alquist 4.0: Towards social intelligence using generative models and dialogue personalization. arXiv preprint arXiv:2109.07968, 2021. — [arXiv:2109.07968](https://arxiv.org/abs/2109.07968) [find](https://scholar.google.com/scholar?q=J%2E+Konr%C3%A1d%2C+J%2E+Pichl%2C+P%2E+Marek%2C+P%2E+Lorenc%2C+V%2E+D%2E+Ta%2C+O%2E+Kobza%2C+L%2E+Hy%CC%80lov%C3%A1%2C+and+J%2E+%C5%A0edivy%CC%80%2E+Alquist+4%2E0%3A+Towards+social+intelligence+using+generative+models+and+dialogue+personalization%2E+arXiv+preprint+)
37. W.-C. Kwan, X. Zeng, Y. Jiang, Y. Wang, L. Li, L. Shang, X. Jiang, Q. Liu, and K.-F. Wong. [Mt-eval: A multi-turn capabilities evaluation benchmark for large language models](https://md.research.elamur.ai/free/en/papers/kwan-2024-mt-eval-multi-turn-capabilities/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20153–20177, 2024.
38. P. Laban, J. Canny, and M. A. Hearst. What’s the latest? a question-driven news chatbot. arXiv preprint arXiv:2105.05392, 2021. [find](https://scholar.google.com/scholar?q=P%2E+Laban%2C+J%2E+Canny%2C+and+M%2E+A%2E+Hearst%2E+What%E2%80%99s+the+latest%3F+a+question%2Ddriven+news+chatbot%2E+arXiv+preprint+arXiv%3A2105%2E05392%2C+2021%2E)
39. P. Laban, L. Murakhovs’ ka, C. Xiong, and C.-S. Wu. Are you sure? challenging llms leads to performance drops in the flipflop experiment. arXiv preprint arXiv:2311.08596, 2023. — [arXiv:2311.08596](https://arxiv.org/abs/2311.08596) [find](https://scholar.google.com/scholar?q=P%2E+Laban%2C+L%2E+Murakhovs%E2%80%99+ka%2C+C%2E+Xiong%2C+and+C%2E%2DS%2E+Wu%2E+Are+you+sure%3F+challenging+llms+leads+to+performance+drops+in+the+flipflop+experiment%2E+arXiv+preprint+arXiv%3A2311%2E08596%2C+2023%2E+%E2%80%94+%5BarXiv%3A2311%2E08596%5D%28ht)
40. P. Laban, A. R. Fabbri, C. Xiong, and C.-S. Wu. Summary of a haystack: A challenge to long-context llms and rag systems. arXiv preprint arXiv:2407.01370, 2024. [find](https://scholar.google.com/scholar?q=P%2E+Laban%2C+A%2E+R%2E+Fabbri%2C+C%2E+Xiong%2C+and+C%2E%2DS%2E+Wu%2E+Summary+of+a+haystack%3A+A+challenge+to+long%2Dcontext+llms+and+rag+systems%2E+arXiv+preprint+arXiv%3A2407%2E01370%2C+2024%2E)
41. S. Lappin. An intensional parametric semantics for vague quantifiers. Linguistics and Philosophy, 23:599–620, 2000. URL [https://api.semanticscholar.org/CorpusID:170154611](https://api.semanticscholar.org/CorpusID:170154611) . [↗](https://api.semanticscholar.org/CorpusID:170154611)
42. M. Lee, M. Srivastava, A. Hardy, J. Thickstun, E. Durmus, A. Paranjape, I. Gerard-Ursin, X. L. Li, F. Ladhak, F. Rong, et al. Evaluating human-language model interaction. arXiv preprint arXiv:2212.09746, 2022. — [arXiv:2212.09746](https://arxiv.org/abs/2212.09746) [find](https://scholar.google.com/scholar?q=M%2E+Lee%2C+M%2E+Srivastava%2C+A%2E+Hardy%2C+J%2E+Thickstun%2C+E%2E+Durmus%2C+A%2E+Paranjape%2C+I%2E+Gerard%2DUrsin%2C+X%2E+L%2E+Li%2C+F%2E+Ladhak%2C+F%2E+Rong%2C+et+al%2E+Evaluating+human%2Dlanguage+model+interaction%2E+arXiv+preprint+arXiv%3A2212%2E097)
43. Y. Lee, K. Son, T. S. Kim, J. Kim, J. J. Y. Chung, E. Adar, and J. Kim. One vs. many: Comprehending accurate information from multiple erroneous and inconsistent ai generations. Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, 2024. URL [https://api.semanticschola…](https://api.semanticschola…) [↗](https://api.semanticschola…)
44. F. Lei, J. Chen, Y. Ye, R. Cao, D. Shin, H. Su, Z. Suo, H. Gao, W. Hu, P. Yin, et al. Spider 2.0: Evaluating language models on real-world enterprise text-to-sql workflows. arXiv preprint arXiv:2411.07763, 2024. — [arXiv:2411.07763](https://arxiv.org/abs/2411.07763) [find](https://scholar.google.com/scholar?q=F%2E+Lei%2C+J%2E+Chen%2C+Y%2E+Ye%2C+R%2E+Cao%2C+D%2E+Shin%2C+H%2E+Su%2C+Z%2E+Suo%2C+H%2E+Gao%2C+W%2E+Hu%2C+P%2E+Yin%2C+et+al%2E+Spider+2%2E0%3A+Evaluating+language+models+on+real%2Dworld+enterprise+text%2Dto%2Dsql+workflows%2E+arXiv+preprint+arXiv%3A2411%2E0)
45. M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer. [Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). arXiv preprint arXiv:1910.13461, 2019.
46. R. Li, R. Li, B. Wang, and X. Du. Iqa-eval: Automatic evaluation of human-model interactive question answering. Advances in Neural Information Processing Systems, 37:109894–109921, 2024. [find](https://scholar.google.com/scholar?q=R%2E+Li%2C+R%2E+Li%2C+B%2E+Wang%2C+and+X%2E+Du%2E+Iqa%2Deval%3A+Automatic+evaluation+of+human%2Dmodel+interactive+question+answering%2E+Advances+in+Neural+Information+Processing+Systems%2C+37%3A109894%E2%80%93109921%2C+2024%2E)
47. S. Li, J. Yan, H. Wang, Z. Tang, X. Ren, V. Srinivasan, and H. Jin. Instruction-following evaluation through verbalizer manipulation. arXiv preprint arXiv:2307.10558, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Li%2C+J%2E+Yan%2C+H%2E+Wang%2C+Z%2E+Tang%2C+X%2E+Ren%2C+V%2E+Srinivasan%2C+and+H%2E+Jin%2E+Instruction%2Dfollowing+evaluation+through+verbalizer+manipulation%2E+arXiv+preprint+arXiv%3A2307%2E10558%2C+2023%2E)
48. Z. Liang, D. Yu, W. Yu, W. Yao, Z. Zhang, X. Zhang, and D. Yu. Mathchat: Benchmarking mathematical reasoning and instruction following in multi-turn interactions. arXiv preprint arXiv:2405.19444, 2024. [find](https://scholar.google.com/scholar?q=Z%2E+Liang%2C+D%2E+Yu%2C+W%2E+Yu%2C+W%2E+Yao%2C+Z%2E+Zhang%2C+X%2E+Zhang%2C+and+D%2E+Yu%2E+Mathchat%3A+Benchmarking+mathematical+reasoning+and+instruction+following+in+multi%2Dturn+interactions%2E+arXiv+preprint+arXiv%3A2405%2E19444%2C+2024)
49. A. Liu, Z. Wu, J. Michael, A. Suhr, P. West, A. Koller, S. Swayamdipta, N. A. Smith, and Y. Choi. We’re afraid language models aren’t modeling ambiguity. In Proceedings of the 2023 [Conference on Empirical Methods in Natural Language Processing, pages 790–807, 2023](https://md.research.elamur.ai/free/en/papers/liu-2023-we-re-afraid-language-models/).
50. N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). Transactions of the Association for Computational Linguistics, 12:157–173, 2024.
51. Y. Liu, A. R. Fabbri, P. Liu, Y. Zhao, L. Nan, R. Han, S. Han, S. Joty, C.-S. Wu, C. Xiong, et al. [Revisiting the gold standard: Grounding summarization evaluation with robust human evaluation](https://md.research.elamur.ai/free/en/papers/liu-2022-revisiting-gold-standard-grounding-summarization/). arXiv preprint arXiv:2212.07981, 2022.
52. Z. Ma, S. Edunov, and M. Auli. A comparison of approaches to document-level machine translation. arXiv preprint arXiv:2101.11040, 2021. — [arXiv:2101.11040](https://arxiv.org/abs/2101.11040) [find](https://scholar.google.com/scholar?q=Z%2E+Ma%2C+S%2E+Edunov%2C+and+M%2E+Auli%2E+A+comparison+of+approaches+to+document%2Dlevel+machine+translation%2E+arXiv+preprint+arXiv%3A2101%2E11040%2C+2021%2E+%E2%80%94+%5BarXiv%3A2101%2E11040%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2101%2E11040%29)
53. C. Malaviya, J. C. Chang, D. Roth, M. Iyyer, M. Yatskar, and K. Lo. Contextualized evaluations: Taking the guesswork out of language model evaluations. arXiv preprint arXiv:2411.07237, 2024. — [arXiv:2411.07237](https://arxiv.org/abs/2411.07237) [find](https://scholar.google.com/scholar?q=C%2E+Malaviya%2C+J%2E+C%2E+Chang%2C+D%2E+Roth%2C+M%2E+Iyyer%2C+M%2E+Yatskar%2C+and+K%2E+Lo%2E+Contextualized+evaluations%3A+Taking+the+guesswork+out+of+language+model+evaluations%2E+arXiv+preprint+arXiv%3A2411%2E07237%2C+2024%2E+%E2%80%94+%5BarXiv%3A)
54. L. Murakhovs’ ka, P. Laban, T. Xie, C. Xiong, and C.-S. Wu. Salespeople vs salesbot: Exploring the role of educational value in conversational recommender systems. arXiv preprint arXiv:2310.17749, 2023. [find](https://scholar.google.com/scholar?q=L%2E+Murakhovs%E2%80%99+ka%2C+P%2E+Laban%2C+T%2E+Xie%2C+C%2E+Xiong%2C+and+C%2E%2DS%2E+Wu%2E+Salespeople+vs+salesbot%3A+Exploring+the+role+of+educational+value+in+conversational+recommender+systems%2E+arXiv+preprint+arXiv%3A2310%2E17749%2C+202)
55. M. Mylrea and N. Robinson. Artificial intelligence (ai) trust framework and maturity model: Applying an entropy lens to improve security, privacy, and ethical ai. Entropy, 25, 2023. URL [https://api.semanticscholar.org/](https://api.semanticscholar.org/) CorpusID:263840323 . [↗](https://api.semanticscholar.org/)
56. T. OLMo, P. Walsh, L. Soldaini, D. Groeneveld, K. Lo, S. Arora, A. Bhagia, Y. Gu, S. Huang, M. Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656, 2024. [find](https://scholar.google.com/scholar?q=T%2E+OLMo%2C+P%2E+Walsh%2C+L%2E+Soldaini%2C+D%2E+Groeneveld%2C+K%2E+Lo%2C+S%2E+Arora%2C+A%2E+Bhagia%2C+Y%2E+Gu%2C+S%2E+Huang%2C+M%2E+Jordan%2C+et+al%2E+2+olmo+2+furious%2E+arXiv+preprint+arXiv%3A2501%2E00656%2C+2024%2E)
57. OpenAI. OpenAI o3 and o4-mini System Card — openai.com. [https://openai.com/index/o3-o4-mini-system-card/](https://openai.com/index/o3-o4-mini-system-card/) , 2025. [Accessed 08-05-2025]. [↗](https://openai.com/index/o3-o4-mini-system-card/)
58. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002.
59. A. P. Parikh, X. Wang, S. Gehrmann, M. Faruqui, B. Dhingra, D. Yang, and D. Das. [Totto: A controlled table-to-text generation dataset](https://md.research.elamur.ai/free/en/papers/parikh-2020-totto-controlled-table-text-generation/). arXiv preprint arXiv:2004.14373, 2020.
60. H. Peng, X. Wang, J. Chen, W. Li, Y. P. Qi, Z. Wang, Z. Wu, K. Zeng, B. Xu, L. Hou, and J. Li. When does in-context learning fall short and why? a study on specification-heavy tasks. ArXiv, abs/2311.08993, 2023. URL [https://api.semanticscholar.org/CorpusID:265212914](https://api.semanticscholar.org/CorpusID:265212914) . [↗](https://api.semanticscholar.org/CorpusID:265212914)
61. S. Pezzelle. Dealing with semantic underspecification in multimodal nlp. arXiv preprint arXiv:2306.05240, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Pezzelle%2E+Dealing+with+semantic+underspecification+in+multimodal+nlp%2E+arXiv+preprint+arXiv%3A2306%2E05240%2C+2023%2E)
62. L. Phan, A. Gatti, Z. Han, N. Li, J. Hu, H. Zhang, C. B. C. Zhang, M. Shaaban, J. Ling, S. Shi, et al. Humanity’s last exam. arXiv preprint arXiv:2501.[14249, 2025](https://md.research.elamur.ai/free/en/papers/phan-2025-humanity-s-last-exam/).
63. C. Poelitz and N. McKenna. Synthetic clarification and correction dialogues about data-centric tasks–a teacher-student approach. arXiv preprint arXiv:2503.14167, 2025. — [arXiv:2503.14167](https://arxiv.org/abs/2503.14167) [find](https://scholar.google.com/scholar?q=C%2E+Poelitz+and+N%2E+McKenna%2E+Synthetic+clarification+and+correction+dialogues+about+data%2Dcentric+tasks%E2%80%93a+teacher%2Dstudent+approach%2E+arXiv+preprint+arXiv%3A2503%2E14167%2C+2025%2E+%E2%80%94+%5BarXiv%3A2503%2E14167%5D%28https%3A%2F%2Farx)
64. M. Post and M. Junczys-Dowmunt. Escaping the sentence-level paradigm in machine translation. arXiv preprint arXiv:2304.12959, 2023. — [arXiv:2304.12959](https://arxiv.org/abs/2304.12959) [find](https://scholar.google.com/scholar?q=M%2E+Post+and+M%2E+Junczys%2DDowmunt%2E+Escaping+the+sentence%2Dlevel+paradigm+in+machine+translation%2E+arXiv+preprint+arXiv%3A2304%2E12959%2C+2023%2E+%E2%80%94+%5BarXiv%3A2304%2E12959%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2304%2E12959%29)
65. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. [find](https://scholar.google.com/scholar?q=A%2E+Radford%2C+J%2E+Wu%2C+R%2E+Child%2C+D%2E+Luan%2C+D%2E+Amodei%2C+I%2E+Sutskever%2C+et+al%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2C+2019%2E)
66. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of machine learning research, 21(140): 1–67, 2020.
67. A. Ram, R. Prasad, C. Khatri, A. Venkatesh, R. Gabriel, Q. Liu, J. Nunn, B. Hedayatnia, M. Cheng, A. Nagar, et al. Conversational ai: The science behind the alexa prize. arXiv preprint arXiv:1801.03604, 2018. [find](https://scholar.google.com/scholar?q=A%2E+Ram%2C+R%2E+Prasad%2C+C%2E+Khatri%2C+A%2E+Venkatesh%2C+R%2E+Gabriel%2C+Q%2E+Liu%2C+J%2E+Nunn%2C+B%2E+Hedayatnia%2C+M%2E+Cheng%2C+A%2E+Nagar%2C+et+al%2E+Conversational+ai%3A+The+science+behind+the+alexa+prize%2E+arXiv+preprint+arXiv%3A1801%2E0360)
68. S. Reddy, D. Chen, and C. D. Manning. [Coqa: A conversational question answering challenge](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/). Transactions of the Association for Computational Linguistics, 7:249–266, 2019.
69. R. Sarkar, B. Sarrafzadeh, N. Chandrasekaran, N. Rangan, P. Resnik, L. Yang, and S. K. Jauhar. Conversational user-ai intervention: A study on prompt rewriting for improved llm response generation. ArXiv, abs/2503.16789, 2025. URL [https://api.semanticscholar.org/CorpusID:277244656](https://api.semanticscholar.org/CorpusID:277244656) . — [arXiv:2503.16789](https://arxiv.org/abs/2503.16789) [↗](https://api.semanticscholar.org/CorpusID:277244656)
70. Y. Scherrer, J. Tiedemann, and S. Loáiciga. Analysing concatenation approaches to document-level nmt in two different domains. In Proceedings of the Third Workshop on Discourse in Machine Translation, Hong-Kong, Nov. 2019. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Y%2E+Scherrer%2C+J%2E+Tiedemann%2C+and+S%2E+Lo%C3%A1iciga%2E+Analysing+concatenation+approaches+to+document%2Dlevel+nmt+in+two+different+domains%2E+In+Proceedings+of+the+Third+Workshop+on+Discourse+in+Machine+Translation%2C)
71. O. Shaikh, H. Mozannar, G. Bansal, A. Fourney, and E. Horvitz. Navigating rifts in human-llm grounding: Study and benchmark. arXiv preprint arXiv:2503.13975, 2025. [find](https://scholar.google.com/scholar?q=O%2E+Shaikh%2C+H%2E+Mozannar%2C+G%2E+Bansal%2C+A%2E+Fourney%2C+and+E%2E+Horvitz%2E+Navigating+rifts+in+human%2Dllm+grounding%3A+Study+and+benchmark%2E+arXiv+preprint+arXiv%3A2503%2E13975%2C+2025%2E)
72. V. Sirdeshmukh, K. Deshpande, J. Mols, L. Jin, E.-Y. Cardona, D. Lee, J. Kritz, W. Primack, S. Yue, and C. Xing. Multichallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier llms. arXiv preprint arXiv:2501.17399, 2025. [find](https://scholar.google.com/scholar?q=V%2E+Sirdeshmukh%2C+K%2E+Deshpande%2C+J%2E+Mols%2C+L%2E+Jin%2C+E%2E%2DY%2E+Cardona%2C+D%2E+Lee%2C+J%2E+Kritz%2C+W%2E+Primack%2C+S%2E+Yue%2C+and+C%2E+Xing%2E+Multichallenge%3A+A+realistic+multi%2Dturn+conversation+evaluation+benchmark+challenging+to)
73. J. Southworth, K. Migliaccio, J. Glover, J. Glover, D. Reed, C. McCarty, J. Brendemuhl, and A. Thomas. Developing a model for ai across the curriculum: Transforming the higher education landscape via innovation in ai literacy. Computers and Education: Artificial Intelligence, 4:100127, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Southworth%2C+K%2E+Migliaccio%2C+J%2E+Glover%2C+J%2E+Glover%2C+D%2E+Reed%2C+C%2E+McCarty%2C+J%2E+Brendemuhl%2C+and+A%2E+Thomas%2E+Developing+a+model+for+ai+across+the+curriculum%3A+Transforming+the+higher+education+landscape+via+)
74. Y. Sun, C. Liu, K. Zhou, J. Huang, R. Song, W. X. Zhao, F. Zhang, D. Zhang, and K. Gai. Parrot: Enhancing multi-turn instruction following for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9729–9750, 2… [find](https://scholar.google.com/scholar?q=Y%2E+Sun%2C+C%2E+Liu%2C+K%2E+Zhou%2C+J%2E+Huang%2C+R%2E+Song%2C+W%2E+X%2E+Zhao%2C+F%2E+Zhang%2C+D%2E+Zhang%2C+and+K%2E+Gai%2E+Parrot%3A+Enhancing+multi%2Dturn+instruction+following+for+large+language+models%2E+In+Proceedings+of+the+62nd+Annual+)
75. G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
76. M. Terry, C. Kulkarni, M. Wattenberg, L. Dixon, and M. R. Morris. Interactive ai alignment: specification, process, and evaluation alignment. arXiv preprint arXiv:2311.00710, 2023. — [arXiv:2311.00710](https://arxiv.org/abs/2311.00710) [find](https://scholar.google.com/scholar?q=M%2E+Terry%2C+C%2E+Kulkarni%2C+M%2E+Wattenberg%2C+L%2E+Dixon%2C+and+M%2E+R%2E+Morris%2E+Interactive+ai+alignment%3A+specification%2C+process%2C+and+evaluation+alignment%2E+arXiv+preprint+arXiv%3A2311%2E00710%2C+2023%2E+%E2%80%94+%5BarXiv%3A2311%2E00710)
77. P. N. Venkit, P. Laban, Y. Zhou, Y. Mao, and C.-S. Wu. Search engines in an ai era: The false promise of factual and verifiable source-cited responses. arXiv preprint arXiv:2410.22349, 2024. — [arXiv:2410.22349](https://arxiv.org/abs/2410.22349) [find](https://scholar.google.com/scholar?q=P%2E+N%2E+Venkit%2C+P%2E+Laban%2C+Y%2E+Zhou%2C+Y%2E+Mao%2C+and+C%2E%2DS%2E+Wu%2E+Search+engines+in+an+ai+era%3A+The+false+promise+of+factual+and+verifiable+source%2Dcited+responses%2E+arXiv+preprint+arXiv%3A2410%2E22349%2C+2024%2E+%E2%80%94+%5BarXiv%3A)
78. S. Vijayvargiya, X. Zhou, A. Yerukola, M. Sap, and G. Neubig. Interactive agents to overcome ambiguity in software engineering. arXiv preprint arXiv:2502.13069, 2025. — [arXiv:2502.13069](https://arxiv.org/abs/2502.13069) [find](https://scholar.google.com/scholar?q=S%2E+Vijayvargiya%2C+X%2E+Zhou%2C+A%2E+Yerukola%2C+M%2E+Sap%2C+and+G%2E+Neubig%2E+Interactive+agents+to+overcome+ambiguity+in+software+engineering%2E+arXiv+preprint+arXiv%3A2502%2E13069%2C+2025%2E+%E2%80%94+%5BarXiv%3A2502%2E13069%5D%28https%3A%2F%2Farxi)
79. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). arXiv preprint arXiv:1804.07461, 2018.
80. X. Wang, Z. Wang, J. Liu, Y. Chen, L. Yuan, H. Peng, and H. Ji. [Mint: Evaluating llms in multi-turn interaction with tools and language feedback](https://md.research.elamur.ai/free/en/papers/wang-2023-mint-evaluating-llms-multi-turn/). In The Twelfth International Conference on Learning Representations, 2024.
81. J. D. Weisz, J. He, M. Muller, G. Hoefer, R. Miles, and W. Geyer. Design principles for generative ai applications. Proceedings of the CHI Conference on Human Factors in Computing Systems, 2024. URL [https://api.semanticscholar.org/CorpusID:267301068](https://api.semanticscholar.org/CorpusID:267301068) . [↗](https://api.semanticscholar.org/CorpusID:267301068)
82. J. Wester, T. Schrills, H. Pohl, and N. van Berkel. “as an ai language model, i cannot”: Investigating llm denials of user requests. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pages 1–14, 2024. [find](https://scholar.google.com/scholar?q=J%2E+Wester%2C+T%2E+Schrills%2C+H%2E+Pohl%2C+and+N%2E+van+Berkel%2E+%E2%80%9Cas+an+ai+language+model%2C+i+cannot%E2%80%9D%3A+Investigating+llm+denials+of+user+requests%2E+In+Proceedings+of+the+2024+CHI+Conference+on+Human+Factors+in+Compu)
83. F. Wildenburg, M. Hanna, and S. Pezzelle. Do pre-trained language models detect and understand semantic underspecification? ask the dust! ArXiv, abs/2402.12486, 2024. URL [https://api.semanticscholar.org/](https://api.semanticscholar.org/) CorpusID:267759784 . [↗](https://api.semanticscholar.org/)
84. Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, et al. [Autogen: Enabling next-gen llm applications via multi-agent conversation](https://md.research.elamur.ai/free/en/papers/wu-2023-autogen-enabling-next-gen-llm/). arXiv preprint arXiv:2308.08155, 2023.
85. F. Yan, H. Mao, C. C.-J. Ji, T. Zhang, S. G. Patil, I. Stoica, and J. E. Gonzalez. Berkeley function calling leaderboard. [https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html](https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html) , 2024. [↗](https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html)
86. T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, et al. [Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task](https://md.research.elamur.ai/free/en/papers/yu-2018-spider-large-scale-human-labeled/). arXiv preprint arXiv:1809.08887, 2018.
87. J. D. Zamfirescu-Pereira, R. Y. Wong, B. Hartmann, and Q. Yang. Why johnny can’t prompt: how non-ai experts try (and fail) to design llm prompts. In Proceedings of the 2023 CHI conference on human factors in computing systems, pages 1–21, 2023. [find](https://scholar.google.com/scholar?q=J%2E+D%2E+Zamfirescu%2DPereira%2C+R%2E+Y%2E+Wong%2C+B%2E+Hartmann%2C+and+Q%2E+Yang%2E+Why+johnny+can%E2%80%99t+prompt%3A+how+non%2Dai+experts+try+%28and+fail%29+to+design+llm+prompts%2E+In+Proceedings+of+the+2023+CHI+conference+on+human+fac)
88. L. Zheng, W.-L. Chiang, Y. Sheng, T. Li, S. Zhuang, Z. Wu, Y. Zhuang, Z. Li, Z. Lin, E. P. Xing, et al. [Lmsys-chat-1m: A large-scale real-world llm conversation dataset](https://md.research.elamur.ai/free/en/papers/zheng-2023-lmsys-chat-1m-large-scale/). arXiv preprint arXiv:2309.11998, 2023.
89. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36: 46595–46623, 2023. [find](https://scholar.google.com/scholar?q=L%2E+Zheng%2C+W%2E%2DL%2E+Chiang%2C+Y%2E+Sheng%2C+S%2E+Zhuang%2C+Z%2E+Wu%2C+Y%2E+Zhuang%2C+Z%2E+Lin%2C+Z%2E+Li%2C+D%2E+Li%2C+E%2E+Xing%2C+et+al%2E+Judging+llm%2Das%2Da%2Djudge+with+mt%2Dbench+and+chatbot+arena%2E+Advances+in+Neural+Information+Processing+S)
90. R. Zhong, T. Yu, and D. Klein. Semantic evaluation for text-to-sql with distilled test suites. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 396–411, 2020. [find](https://scholar.google.com/scholar?q=R%2E+Zhong%2C+T%2E+Yu%2C+and+D%2E+Klein%2E+Semantic+evaluation+for+text%2Dto%2Dsql+with+distilled+test+suites%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+pages+)
91. G. K. Zipf. Human behavior and the principle of least effort: An introduction to human eoclogy. Addison-Wesley Press, 1949. [find](https://scholar.google.com/scholar?q=G%2E+K%2E+Zipf%2E+Human+behavior+and+the+principle+of+least+effort%3A+An+introduction+to+human+eoclogy%2E+Addison%2DWesley+Press%2C+1949%2E)

*Updated 24 October 2026*