---
title: "Lost in Simulation — paper by Seshadri et al. (2026)"
description: "Agentic benchmarks commonly replace real people with an LLM that plays the user, but this approach… 46 references, cited by 4 papers here."
authors: ["Preethi Seshadri", "Samuel Cahyawijaya", "Ayomide Odumakinde", "Sameer Singh", "Seraphina Goldfarb-Tarrant"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.2192/"
doi: "10.18653/v1/2026.acl-long.2192"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/seshadri-2026-lost-simulation-llm-simulated-users/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/seshadri-2026-lost-simulation-llm-simulated-users/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/3/) / Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations

# Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations

**Authors:** Preethi Seshadri, [Samuel Cahyawijaya](https://md.research.elamur.ai/free/en/authors/samuel-cahyawijaya/), [Ayomide Odumakinde](https://md.research.elamur.ai/free/en/authors/ayomide-odumakinde/), [Sameer Singh](https://md.research.elamur.ai/free/en/authors/sameer-singh/) and [Seraphina Goldfarb-Tarrant](https://md.research.elamur.ai/free/en/authors/seraphina-goldfarb-tarrant/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.2192/) · [PDF](https://aclanthology.org/2026.acl-long.2192.pdf) · [DOI 10.18653/v1/2026.acl-long.2192](https://doi.org/10.18653/v1/2026.acl-long.2192) · [OpenAlex](https://openalex.org/W7166876594)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · 4 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Agentic benchmarks commonly replace real people with an LLM that plays the user, but this approach has rarely been checked against actual human behaviour. Seshadri et al. examine this practice along three dimensions: robustness (stability across simulator models), validity (whether simulated results predict human results), and fairness (whether the proxy works equally well for different user groups). They selected 18 τ-Bench retail tasks spread evenly across difficulty levels and conducted a user study with English speakers based in four countries (Nigeria, Kenya, India, and the US), keeping GPT-4o as the agent throughout. Swapping the simulator LLM changes agent success noticeably. Simulated results also misjudge human outcomes in a systematic way, depending on task difficulty. The gaps are largest for AAVE and Indian English speakers. Simulated users also behave unnaturally, for example by being overly polite and asking many questions. The authors conclude that relying on a single simulator can misrepresent agent capability and hide demographic disparities. They recommend reporting results from several simulators and validating them against human data.

## Key points
- Task difficulty was estimated from 5 τ-Bench runs with GPT-4o as both agent and user. The authors then sampled 3 tasks for each of six success-rate levels (0–100%), giving 18 retail tasks.
- Participants were recruited via Prolific, except in Nigeria, where snowball sampling was used. Each completed 4 tasks: 2 harder and 2 easier. US participants were screened as White SAE or Black AAVE speakers and stratified by age.
- Agent success rates varied by up to 9 percentage points depending on which LLM played the user.
- Simulation underestimated agent success on the hardest tasks and overestimated it on moderately difficult ones.
- AAVE speakers had lower success and larger calibration errors than SAE speakers, and the gap grew with age. Data and prompts are released at github.com/preethiseshadri518/lost-in-simulation.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of California, Irvine](https://md.research.elamur.ai/free/en/universities/university-of-california-irvine/) — University, Irvine, United States · 37 works · 36,319 citations
- [Cohere](https://md.research.elamur.ai/free/en/companies/cohere/) — Company, San Francisco, United States · 15 works · 1,223 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Preethi Seshadri, Samuel Cahyawijaya, Ayomide Odumakinde, Sameer Singh and Seraphina Goldfarb-Tarrant. 2026. Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{seshadri2026lost,
  title = {{Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations}},
  author = {Preethi Seshadri, Samuel Cahyawijaya, Ayomide Odumakinde, Sameer Singh and Seraphina Goldfarb-Tarrant},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2192},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2192}
}
```

- [← previous LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification](https://md.research.elamur.ai/free/en/papers/yang-2026-longspec-long-context-lossless-speculative/)
- [next → Mathematical Proof as a Litmus Test: Revealing Failure Modes of Advanced Large Reasoning Models](https://md.research.elamur.ai/free/en/papers/guo-2026-mathematical-proof-as-litmus-test/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/3/)

## References

46 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dhruv Agarwal, Mor Naaman, and Aditya Vashistha. 2025. Ai suggestions homogenize writing toward western styles and diminish cultural nuances. CHI ’25, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Dhruv+Agarwal%2C+Mor+Naaman%2C+and+Aditya+Vashistha%2E+2025%2E+Ai+suggestions+homogenize+writing+toward+western+styles+and+diminish+cultural+nuances%2E+CHI+%E2%80%9925%2C+New+York%2C+NY%2C+USA%2E+Association+for+Computing+Mach)
2. Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. 2025. τ 2 -bench: Evaluating conversational agents in a dual-control environment. Preprint, arXiv:2506.07982. [↗](https://arxiv.org/abs/2506.07982)
3. Elisa Bassignana, Amanda Cercas Curry, and Dirk Hovy. 2025. [The AI gap: How socioeconomic status affects language technology interactions](https://md.research.elamur.ai/free/en/papers/bassignana-2025-ai-gap-how-socioeconomic-status/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18647–18664, Vienna, Austria. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2505.12158)
4. Samuel Cahyawijaya, Delong Chen, Yejin Bang, Leila Khalatbari, Bryan Wilie, Ziwei Ji, Etsuko Ishii, and Pascale Fung. 2025. High-dimension human value representation in large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5303–5330, Albuquerque, New Mexico. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2404.07900)
5. Serina Chang, Ashton Anderson, and Jake M. Hofman. 2025. [ChatBench: From static benchmarks to human-AI evaluation](https://md.research.elamur.ai/free/en/papers/chang-2025-chatbench-static-benchmarks-human-ai/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 26009– 26038, Vienna, Austria. Association for Computational Linguistics.
6. CNBC. 2025. Ai travel agents planning future trip far beyond assistant status. [find](https://scholar.google.com/scholar?q=CNBC%2E+2025%2E+Ai+travel+agents+planning+future+trip+far+beyond+assistant+status%2E)
7. Cohere. 2025. Command a: An enterprise-ready large language model. Preprint, arXiv:2504.00698. [↗](https://arxiv.org/abs/2504.00698)
8. Kaustubh Deshpande, Ved Sirdeshmukh, Johannes Baptist Mols, Lifeng Jin, Ed-Yeremai Hernandez-Cardona, Dean Lee, Jeremy Kritz, Willow E. Primack, Summer Yue, and Chen Xing. 2025. Multi-Challenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier LLMs. In Findings of the Association for Computational Linguistics: ACL 2025, pages 18632–18702, Vienna, Austria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kaustubh+Deshpande%2C+Ved+Sirdeshmukh%2C+Johannes+Baptist+Mols%2C+Lifeng+Jin%2C+Ed%2DYeremai+Hernandez%2DCardona%2C+Dean+Lee%2C+Jeremy+Kritz%2C+Willow+E%2E+Primack%2C+Summer+Yue%2C+and+Chen+Xing%2E+2025%2E+Multi%2DChallenge%3A+A+rea)
9. Yao Dou, Michel Galley, Baolin Peng, Chris Kedzie, Weixin Cai, Alan Ritter, Chris Quirk, Wei Xu, and Jianfeng Gao. 2025. SimulatorArena: Are user simulators reliable proxies for multi-turn evaluation of AI assistants? In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 35200–35278, Suzhou, China. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2510.05444)
10. Esin Durmus, Karina Nguyen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. 2024. [Towards measuring the representation of subjective global opinions in language models](https://md.research.elamur.ai/free/en/papers/durmus-2023-towards-measuring-representation-subjective-global/). In First Conference on Language Modeling. [↗](https://doi.org/10.48550/arxiv.2306.16388)
11. Sophie Groenwold, Lily Ou, Aesha Parekh, Samhita Honnavalli, Sharon Levy, Diba Mirza, and William Yang Wang. 2020. [Investigating African-American Vernacular English in transformer-based text generation](https://md.research.elamur.ai/free/en/papers/groenwold-2020-investigating-african-american-vernacular-english/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5877–5883, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.emnlp-main.473)
12. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 1321–1330. JMLR.org. [↗](https://doi.org/10.48550/arxiv.1706.04599)
13. Luna Luan Haoyue and Hichang Cho. 2024. Factors influencing intention to engage in human–chatbot interaction: examining user perceptions and context culture orientation. 24(1):607–620. [↗](https://doi.org/10.1007/s10209-023-01087-7)
14. Muyu He, Anand Kumar, Tsach Mackey, Meghana Rajeev, James Zou, and Nazneen Rajani. 2025. Impatient users confuse ai agents: High-fidelity simulations of human traits for testing agents. Preprint, arXiv:2510.04491. [↗](https://arxiv.org/abs/2510.04491)
15. Jonathan Ivey, Shivani Kumar, Jiayu Liu, Hua Shen, Sushrita Rakshit, Rohan Raju, Haotian Zhang, Aparna Ananthasubramaniam, Junghwan Kim, Bowen Yi, Dustin Wright, Abraham Israeli, Anders Giovanni Møller, Lechen Zhang, and David Jurgens. 2024. Real or robotic? assessing whether llms accurately simulate qualities of human responses in dialogue. Preprint, arXiv:2409.08330. [↗](https://arxiv.org/abs/2409.08330)
16. Kimi Team. 2026. [Kimi k2: Open agentic intelligence](https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-k2-open-agentic/). Preprint, arXiv:2507.20534. [↗](https://arxiv.org/abs/2507.20534)
17. Nayeon Lee, Yejin Bang, Holy Lovenia, Samuel Cahyawijaya, Wenliang Dai, and Pascale Fung. 2023. Survey of social bias in vision-language models. Preprint, arXiv:2309.14381. [↗](https://arxiv.org/abs/2309.14381)
18. Nayeon Lee, Chani Jung, Junho Myung, Jiho Jin, Jose Camacho-Collados, Juho Kim, and Alice Oh. 2024. Exploring cross-cultural differences in English hate speech annotations: From dataset construction to analysis. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4205–4224, Mexico City, Mexico. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2308.16705)
19. Zihan Liu, Han Li, Anfan Chen, Renwen Zhang, and Yi-Chieh Lee. 2024. Understanding public perceptions of ai conversational agents: A cross-cultural analysis. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI ’24, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Zihan+Liu%2C+Han+Li%2C+Anfan+Chen%2C+Renwen+Zhang%2C+and+Yi%2DChieh+Lee%2E+2024%2E+Understanding+public+perceptions+of+ai+conversational+agents%3A+A+cross%2Dcultural+analysis%2E+In+Proceedings+of+the+2024+CHI+Conference+)
20. Yuxuan Lu, Jing Huang, Yan Han, Bingsheng Yao, Sisong Bei, Jiri Gesi, Yaochen Xie, Zheshen, Wang, Qi He, and Dakuo Wang. 2025. [Can llm agents simulate multi-turn human behavior? evidence from real online customer behavior data](https://md.research.elamur.ai/free/en/papers/lu-2026-can-llm-agents-simulate-multi/). Preprint, arXiv:2503.20749. [↗](https://arxiv.org/abs/2503.20749)
21. Shuhaib Mehri, Xiaocheng Yang, Takyoung Kim, Gokhan Tur, Shikib Mehri, and Dilek Hakkani-Tür. 2025. [Goal alignment in llm-based user simulators for conversational ai](https://md.research.elamur.ai/free/en/papers/mehri-2026-goal-alignment-llm-based-user/). Preprint, arXiv:2507.20152. [↗](https://arxiv.org/abs/2507.20152)
22. OpenAI. 2025. [gpt-oss-120b gpt-oss-20b model card](https://md.research.elamur.ai/free/en/papers/agarwal-2025-gpt-oss-120b-gpt-oss/). Preprint, arXiv:2508.10925. [↗](https://arxiv.org/abs/2508.10925)
23. Siddhesh Pawar, Junyeong Park, Jiho Jin, Arnav Arora, Junho Myung, Srishti Yadav, Faiz Ghifari Haznitrama, Inhwa Song, Alice Oh, and Isabelle Augenstein. 2025. [Survey of cultural awareness in language models: Text and beyond](https://md.research.elamur.ai/free/en/papers/pawar-2025-survey-cultural-awareness-language-models/). Computational Linguistics, 51(3):907–1004. [↗](https://doi.org/10.1162/coli.a.14)
24. Pew Research Center. 2025. Ai in americans’ lives: Awareness, experiences and attitudes. [https://www.pewresearch.org/science/2025/09/17/ai-in-americans-lives-awareness-experiences-and-attitudes/](https://www.pewresearch.org/science/2025/09/17/ai-in-americans-lives-awareness-experiences-and-attitudes/). Accessed: 2025-01-04. [↗](https://www.pewresearch.org/science/2025/09/17/ai-in-americans-lives-awareness-experiences-and-attitudes/)
25. Vinodkumar Prabhakaran, Christopher Homan, Lora Aroyo, Aida Mostafazadeh Davani, Alicia Parrish, Alex Taylor, Mark Diaz, Ding Wang, and Gregory Serapio-García. 2024. GRASP: A disagreement analysis framework to assess group associations in perspectives. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3473–3492, Mexico City, Mexico. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2311.05074)
26. Haoyi Qiu, Alexander Fabbri, Divyansh Agarwal, Kung-Hsiang Huang, Sarah Tan, Nanyun Peng, and Chien-Sheng Wu. 2025. Evaluating cultural and social awareness of LLM web agents. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3978–4005, Albuquerque, New Mexico. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Haoyi+Qiu%2C+Alexander+Fabbri%2C+Divyansh+Agarwal%2C+Kung%2DHsiang+Huang%2C+Sarah+Tan%2C+Nanyun+Peng%2C+and+Chien%2DSheng+Wu%2E+2025%2E+Evaluating+cultural+and+social+awareness+of+LLM+web+agents%2E+In+Findings+of+the+Assoc)
27. Olawale Salaudeen, Anka Reuel, Ahmed Ahmed, Suhana Bedi, Zachary Robertson, Sudharsan Sundar, Ben Domingue, Angelina Wang, and Sanmi Koyejo. 2025. Measurement to meaning: A validity-centered framework for ai evaluation. Preprint, arXiv:2505.10573. [↗](https://arxiv.org/abs/2505.10573)
28. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. [Whose opinions do language models reflect?](https://md.research.elamur.ai/free/en/papers/santurkar-2023-whose-opinions-language-models-reflect/) In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org. [↗](https://doi.org/10.48550/arxiv.2303.17548)
29. Sebastin Santy, Jenny Liang, Ronan Le Bras, Katharina Reinecke, and Maarten Sap. 2023. [NLPositionality: Characterizing design biases of datasets and models](https://md.research.elamur.ai/free/en/papers/santy-2023-nlpositionality-characterizing-design-biases-datasets/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9080–9102, Toronto, Canada. Association for Computational Linguistics.
30. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. [The risk of racial bias in hate speech detection](https://md.research.elamur.ai/free/en/papers/sap-2019-risk-racial-bias-hate-speech/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668–1678, Florence, Italy. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/p19-1163)
31. Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. [Annotators with attitudes: How annotator beliefs and identities bias toxic language detection](https://md.research.elamur.ai/free/en/papers/sap-2022-annotators-attitudes-how-annotator-beliefs/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5884–5906, Seattle, United States. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2111.07997)
32. Yijia Shao, Vinay Samuel, Yucheng Jiang, John Yang, and Diyi Yang. 2025. Collaborative gym: A framework for enabling and evaluating human-agent collaboration. Preprint, arXiv:2412.15701. [↗](https://arxiv.org/abs/2412.15701)
33. Huaman Sun, Jiaxin Pei, Minje Choi, and David Jurgens. 2025. Sociodemographic prompting is not yet an effective approach for simulating subjective judgments with LLMs. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 845–854, Albuquerque, New Mexico. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Huaman+Sun%2C+Jiaxin+Pei%2C+Minje+Choi%2C+and+David+Jurgens%2E+2025%2E+Sociodemographic+prompting+is+not+yet+an+effective+approach+for+simulating+subjective+judgments+with+LLMs%2E+In+Proceedings+of+the+2025+Confe)
34. Yan Tao, Olga Viberg, Ryan S Baker, and René F Kizilcec. 2024. Cultural bias and cultural alignment of large language models. PNAS Nexus, 3(9). [↗](https://doi.org/10.1093/pnasnexus/pgae346)
35. Kimberly Truong, Riccardo Fogliato, Hoda Heidari, and Steven Wu. 2025. Persona-augmented benchmarking: Evaluating LLMs across diverse writing styles. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 22687–22720, Suzhou, China. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2507.22168)
36. Yu-Min Tseng, Yu-Chao Huang, Teng-Yun Hsiao, Wei-Lin Chen, Chao-Wei Huang, Yu Meng, and Yun-Nung Chen. 2024. [Two tales of persona in LLMs: A survey of role-playing and personalization](https://md.research.elamur.ai/free/en/papers/tseng-2024-two-tales-persona-llms-survey/). In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 16612–16631, Miami, Florida, USA. Association for Computational Linguistics.
37. Haoxin Wang, Xianhan Peng, Huang Cheng, Yizhe Huang, Ming Gong, Chenghan Yang, Yang Liu, and Jiang Lin. 2025a. ECom-bench: Can LLM agent resolve real-world E-commerce customer support issues? In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 276–284, Suzhou (China). Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2507.05639)
38. Wenxuan Wang, Wenxiang Jiao, Jingyuan Huang, Ruyi Dai, Jen-tse Huang, Zhaopeng Tu, and Michael Lyu. 2024. [Not all countries celebrate thanksgiving: On the cultural dominance in large language models](https://md.research.elamur.ai/free/en/papers/wang-2024-not-all-countries-celebrate-thanksgiving/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6349–6384, Bangkok, Thailand. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2310.12481)
39. Zhefan Wang, Ning Geng, Zhiqiang Guo, Weizhi Ma, and Min Zhang. 2025b. Human vs. agent in task-oriented conversations. In Proceedings of the 2025 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, SIGIR-AP 2025, page 133–142, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Zhefan+Wang%2C+Ning+Geng%2C+Zhiqiang+Guo%2C+Weizhi+Ma%2C+and+Min+Zhang%2E+2025b%2E+Human+vs%2E+agent+in+task%2Doriented+conversations%2E+In+Proceedings+of+the+2025+Annual+International+ACM+SIGIR+Conference+on+Research+)
40. Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. 2024. Travelplanner: a benchmark for real-world planning with language agents. ICML’24. JMLR.org. [↗](https://doi.org/10.48550/arxiv.2402.01622)
41. Jane Xing, Tianyi Niu, and Shashank Srivastava. 2025. Chameleon LLMs: User personas influence chatbot personality shifts. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 17314–17332, Suzhou, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jane+Xing%2C+Tianyi+Niu%2C+and+Shashank+Srivastava%2E+2025%2E+Chameleon+LLMs%3A+User+personas+influence+chatbot+personality+shifts%2E+In+Proceedings+of+the+2025+Conference+on+Empirical+Methods+in+Natural+Language)
42. Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Zhiruo Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Mel-roy Maben, Raj Mehta, Wayne Chi, Lawrence Ke-unho Jang, Yiqing Xie, and 2 others. 2025. Theagentcompany: Benchmarking LLM agents on consequential real world tasks. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track. [find](https://scholar.google.com/scholar?q=Frank+F%2E+Xu%2C+Yufan+Song%2C+Boxuan+Li%2C+Yuxuan+Tang%2C+Kritanjali+Jain%2C+Mengxue+Bao%2C+Zora+Zhiruo+Wang%2C+Xuhui+Zhou%2C+Zhitong+Guo%2C+Murong+Cao%2C+Mingyang+Yang%2C+Hao+Yang+Lu%2C+Amaad+Martin%2C+Zhe+Su%2C+Leander+Mel%2Droy+)
43. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2025. [τ -bench: A benchmark for Tool-Agent-User interaction in real-world domains](https://md.research.elamur.ai/free/en/papers/yao-2024-tau-bench-benchmark-tool/). In International Conference on Representation Learning, volume 2025, pages 9965–10017. [↗](https://doi.org/10.48550/arxiv.2406.12045)
44. Se-eun Yoon, Zhankui He, Jessica Echterhoff, and Julian McAuley. 2024. Evaluating large language models as generative user simulators for conversational recommendation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1490–1504, Mexico City, Mexico. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2403.09738)
45. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. [Webarena: A realistic web environment for building autonomous agents](https://md.research.elamur.ai/free/en/papers/zhou-2023-webarena-realistic-web-environment-building/). In The Twelfth International Conference on Learning Representations.
46. Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy K Zhang, Shu Liu, Sasha Cui, Sayash Kapoor, Shayne Longpre, Kevin Meng, Rebecca Weiss, Fazl Barez, Rahul Gupta, Jwala Dhamala, Jacob Merizian, Mario Giulianelli, Harry Coppock, Cozmin Ududec, Antony Kellermann, Jasjeet S Sekhon, and 7 others. 2025. Establishing best practices in building rigorous agentic benchmarks. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track. [find](https://scholar.google.com/scholar?q=Yuxuan+Zhu%2C+Tengjun+Jin%2C+Yada+Pruksachatkun%2C+Andy+K+Zhang%2C+Shu+Liu%2C+Sasha+Cui%2C+Sayash+Kapoor%2C+Shayne+Longpre%2C+Kevin+Meng%2C+Rebecca+Weiss%2C+Fazl+Barez%2C+Rahul+Gupta%2C+Jwala+Dhamala%2C+Jacob+Merizian%2C+Mario+G)

*Updated 1 November 2026*