---
title: "SAGE — paper by Shea et al. (2026) — summary and ideas"
description: "SAGE tackles the difficulty of evaluating multi-turn interactive agents, such as customer-support or sales… 42 references, cited by 1 paper here."
authors: ["Ryan Shea", "Yunan Lu", "Liang Qiu", "Zhou Yu"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: EACL 2026"
source: "https://aclanthology.org/2026.findings-eacl.147/"
doi: "10.18653/v1/2026.findings-eacl.147"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shea-2026-sage-top-down-bottom-up/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shea-2026-sage-top-down-bottom-up/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/17/) / SAGE : A Top-Down Bottom-Up Knowledge-Grounded User Simulator for Multi-turn Agent Evaluation

# SAGE : A Top-Down Bottom-Up Knowledge-Grounded User Simulator for Multi-turn Agent Evaluation

**Authors:** Ryan Shea, Yunan Lu, [Liang Qiu](https://md.research.elamur.ai/free/en/authors/liang-qiu/) and [Zhou Yu](https://md.research.elamur.ai/free/en/authors/zhou-yu/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: EACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-eacl.147/) · [PDF](https://aclanthology.org/2026.findings-eacl.147.pdf) · [DOI 10.18653/v1/2026.findings-eacl.147](https://doi.org/10.18653/v1/2026.findings-eacl.147) · [OpenAlex](https://openalex.org/W7140089769)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/shea-2026-sage-top-down-bottom-up/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

SAGE tackles the difficulty of evaluating multi-turn interactive agents, such as customer-support or sales bots, without relying on expensive human testers. Earlier LLM-based user simulators usually play generic users whose goals barely match the business the agent serves. SAGE grounds simulated users in two kinds of company knowledge. The top-down kind comes from business logic: ideal customer profiles (ICPs), Big Five personality traits and demographic attributes. The bottom-up kind is drawn from the agent's own infrastructure: product catalogs, FAQs, webpages and knowledge bases.

The pipeline has four steps:
- An LLM turns the company's listed agent capabilities into user goals.
- Each goal is combined with matching infrastructure knowledge and a consistent user profile to form a task scenario.
- An LLM simulator holds multi-turn conversations with the target agent.
- The resulting dialogues are analyzed for errors.

Automatic metrics and human judgments show that SAGE produces more realistic and more varied conversations than ablated variants. It also surfaces up to a third more agent bugs. This makes it a practical tool for iterative agent development, especially in cold-start settings with no real interaction logs.

## Key points

- SAGE finds roughly 25–33% more agent errors than baseline simulation approaches.
- In ablations, the variant without ICPs gave repetitive, generic turns. The variant without Agent-Infra asked off-domain questions, such as comparing a restaurant-serving robot for farm use.
- On a RAG-based Sales Agent, SAGE mostly uncovered "Unspecific Response" and "Inaccurate Answer" errors. Most errors appeared after the first turn, which supports the case for multi-turn testing.
- Richtech Robotics supplied human–agent interaction data used in the evaluation.
- Acknowledged limitations:
  - Each scenario uses one goal and one knowledge item.
  - The attributes are designed for customer-facing agents.
  - Diversity is measured only with reference-free lexical measures.
  - The simulator is not compared directly against real user logs.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Columbia University](https://md.research.elamur.ai/free/en/universities/columbia-university/) — University, New York, United States · 241 works · 54,053 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [💻 Software](https://md.research.elamur.ai/free/en/shelves/computer-science/software/) · 92 works

## Cite

Cite as: Ryan Shea, Yunan Lu, Liang Qiu and Zhou Yu. 2026. SAGE : A Top-Down Bottom-Up Knowledge-Grounded User Simulator for Multi-turn Agent Evaluation. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shea2026sage,
  title = {{SAGE : A Top-Down Bottom-Up Knowledge-Grounded User Simulator for Multi-turn Agent Evaluation}},
  author = {Ryan Shea, Yunan Lu, Liang Qiu and Zhou Yu},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.147},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.147}
}
```

- [← previous Sagarmatha at SemEval-2026 Task 9: Heterogeneous Ensembling and Hierarchical Task Conditioning for Multilingual Latent Distributional Divergence Modeling](https://md.research.elamur.ai/free/en/papers/maharjan-2026-sagarmatha-semeval-2026-task-9/)
- [next → Sakura at BEA 2026 Shared Task 1: What Makes Vocabulary Difficult?](https://md.research.elamur.ai/free/en/papers/nohejl-2026-sakura-bea-2026-shared-task/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/17/)

## References

42 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Adnan Ahmad, Stefan Hillmann, and Sebastian Möller. 2025. Simulating User Diversity in Task-Oriented Dialogue Systems using Large Language Models. arXiv preprint. ArXiv:2502.12813 [cs] version: 1. — [arXiv:2502.12813](https://arxiv.org/abs/2502.12813) [find](https://scholar.google.com/scholar?q=Adnan+Ahmad%2C+Stefan+Hillmann%2C+and+Sebastian+M%C3%B6ller%2E+2025%2E+Simulating+User+Diversity+in+Task%2DOriented+Dialogue+Systems+using+Large+Language+Models%2E+arXiv+preprint%2E+ArXiv%3A2502%2E12813+%5Bcs%5D+version%3A+1%2E+%E2%80%94+%5B)
2. Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. 2025. τ 2 -bench: Evaluating conversational agents in a dual-control environment. Preprint, arXiv:2506.07982. [find](https://scholar.google.com/scholar?q=Victor+Barres%2C+Honghua+Dong%2C+Soham+Ray%2C+Xujie+Si%2C+and+Karthik+Narasimhan%2E+2025%2E+%CF%84+2+%2Dbench%3A+Evaluating+conversational+agents+in+a+dual%2Dcontrol+environment%2E+Preprint%2C+arXiv%3A2506%2E07982%2E)
3. Michel Borgh, Juan Xu, and Marin Sikkenk. 2020. Identifying, analyzing, and finding solutions to the sales lead black hole: A design science approach. Industrial Marketing Management, 88. BANT-criteria (i.e., budget, authority, need, timing) are used to determine prospects. [find](https://scholar.google.com/scholar?q=Michel+Borgh%2C+Juan+Xu%2C+and+Marin+Sikkenk%2E+2020%2E+Identifying%2C+analyzing%2C+and+finding+solutions+to+the+sales+lead+black+hole%3A+A+design+science+approach%2E+Industrial+Marketing+Management%2C+88%2E+BANT%2Dcriteri)
4. Nicolas Bougie and Narimasa Watanabe. 2025. SimUSER: Simulating User Behavior with Large Language Models for Recommender System Evaluation. arXiv preprint. ArXiv:2504.12722 [cs] version: 1. [find](https://scholar.google.com/scholar?q=Nicolas+Bougie+and+Narimasa+Watanabe%2E+2025%2E+SimUSER%3A+Simulating+User+Behavior+with+Large+Language+Models+for+Recommender+System+Evaluation%2E+arXiv+preprint%2E+ArXiv%3A2504%2E12722+%5Bcs%5D+version%3A+1%2E)
5. Luyu Chen, Quanyu Dai, Zeyu Zhang, Xueyang Feng, Mingyu Zhang, Pengcheng Tang, Xu Chen, Yue Zhu, and Zhenhua Dong. 2025. RecUserSim: A Realistic and Diverse User Simulator for Evaluating Conversational Recommender Systems. In Companion Proceedings of the ACM on Web Conference 2025, pages 133–142. Ar… — [arXiv:2507.22897](https://arxiv.org/abs/2507.22897) [find](https://scholar.google.com/scholar?q=Luyu+Chen%2C+Quanyu+Dai%2C+Zeyu+Zhang%2C+Xueyang+Feng%2C+Mingyu+Zhang%2C+Pengcheng+Tang%2C+Xu+Chen%2C+Yue+Zhu%2C+and+Zhenhua+Dong%2E+2025%2E+RecUserSim%3A+A+Realistic+and+Diverse+User+Simulator+for+Evaluating+Conversationa)
6. Myra Cheng, Esin Durmus, and Dan Jurafsky. 2023. [Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models](https://md.research.elamur.ai/free/en/papers/cheng-2023-marked-personas-natural-language-prompts/). arXiv preprint. ArXiv:2305.18189 [cs].
7. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan-Jiang Jiang, and 3290 [other…](https://md.research.elamur.ai/free/en/papers/comanici-2025-gemini-2-5-pushing-frontier/)
8. Sam Davidson, Salvatore Romeo, Raphael Shu, James Gung, Arshit Gupta, Saab Mansour, and Yi Zhang. 2023. User simulation with large language models for evaluating task-oriented dialogue. Preprint, arXiv:2309.13233. — [arXiv:2309.13233](https://arxiv.org/abs/2309.13233) [find](https://scholar.google.com/scholar?q=Sam+Davidson%2C+Salvatore+Romeo%2C+Raphael+Shu%2C+James+Gung%2C+Arshit+Gupta%2C+Saab+Mansour%2C+and+Yi+Zhang%2E+2023%2E+User+simulation+with+large+language+models+for+evaluating+task%2Doriented+dialogue%2E+Preprint%2C+arXi)
9. Kaustubh Dhole. 2024. KAUCUS - knowledgeable user simulators for training large language models. In Proceedings of the 1st Workshop on Simulating Conversational Intelligence in Chat (SCI-CHAT 2024), pages 53–65, St. Julians, Malta. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kaustubh+Dhole%2E+2024%2E+KAUCUS+%2D+knowledgeable+user+simulators+for+training+large+language+models%2E+In+Proceedings+of+the+1st+Workshop+on+Simulating+Conversational+Intelligence+in+Chat+%28SCI%2DCHAT+2024%29%2C+p)
10. Esin Durmus, Karina Nguyen, Thomas I. Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. 2024. [Towards Measuring t…](https://md.research.elamur.ai/free/en/papers/durmus-2023-towards-measuring-representation-subjective-global/)
11. Kung-Hsiang Huang, Akshara Prabhakar, Onkar Thorat, Divyansh Agarwal, Prafulla Kumar Choubey, Yixin Mao, Silvio Savarese, Caiming Xiong, and Chien-Sheng Wu. 2025. CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions. arXiv preprint. ArXiv:2505.18878 [cs]… — [arXiv:2505.18878](https://arxiv.org/abs/2505.18878) [find](https://scholar.google.com/scholar?q=Kung%2DHsiang+Huang%2C+Akshara+Prabhakar%2C+Onkar+Thorat%2C+Divyansh+Agarwal%2C+Prafulla+Kumar+Choubey%2C+Yixin+Mao%2C+Silvio+Savarese%2C+Caiming+Xiong%2C+and+Chien%2DSheng+Wu%2E+2025%2E+CRMArena%2DPro%3A+Holistic+Assessment+of+)
12. Oliver P. John and Sanjay Srivastava. 1999. The big five trait taxonomy: History, measurement, and theoretical perspectives. In Handbook of Personality: Theory and Research, pages 102–138. Guilford Press. [find](https://scholar.google.com/scholar?q=Oliver+P%2E+John+and+Sanjay+Srivastava%2E+1999%2E+The+big+five+trait+taxonomy%3A+History%2C+measurement%2C+and+theoretical+perspectives%2E+In+Handbook+of+Personality%3A+Theory+and+Research%2C+pages+102%E2%80%93138%2E+Guilford+Pr)
13. Yannis Katsis, Sara Rosenthal, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee, Lucian Popa, Vraj Shah, Huaiyu Zhu, Danish Contractor, and Marina Danilevsky. 2025. Mtrag: A multi-turn conversational benchmark for evaluating retrieval-augmented generation systems. Preprint, arXiv:2501.03468. — [arXiv:2501.03468](https://arxiv.org/abs/2501.03468) [find](https://scholar.google.com/scholar?q=Yannis+Katsis%2C+Sara+Rosenthal%2C+Kshitij+Fadnis%2C+Chulaka+Gunasekara%2C+Young%2DSuk+Lee%2C+Lucian+Popa%2C+Vraj+Shah%2C+Huaiyu+Zhu%2C+Danish+Contractor%2C+and+Marina+Danilevsky%2E+2025%2E+Mtrag%3A+A+multi%2Dturn+conversational)
14. Samira Khodabandehlou, S. Alireza Hashemi Golpayegani, and Mahmoud Zivari Rahman. 2020. An effective recommender system based on personality traits, demographics and behavior of customers in time context. Data Technologies and Applications, 55(1):149–174. [find](https://scholar.google.com/scholar?q=Samira+Khodabandehlou%2C+S%2E+Alireza+Hashemi+Golpayegani%2C+and+Mahmoud+Zivari+Rahman%2E+2020%2E+An+effective+recommender+system+based+on+personality+traits%2C+demographics+and+behavior+of+customers+in+time+cont)
15. Viljami Kivistö. 2024. Customer profiling and purchase decision influencers: an empirical case study from the management consulting industry. [find](https://scholar.google.com/scholar?q=Viljami+Kivist%C3%B6%2E+2024%2E+Customer+profiling+and+purchase+decision+influencers%3A+an+empirical+case+study+from+the+management+consulting+industry%2E)
16. Chuyi Kong, Yaxin Fan, Xiang Wan, Feng Jiang, and Benyou Wang. 2024. PlatoLM: Teaching LLMs in multi-round dialogue via a user simulator. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7841–7863, Bangkok, Thailand. Associatio… [find](https://scholar.google.com/scholar?q=Chuyi+Kong%2C+Yaxin+Fan%2C+Xiang+Wan%2C+Feng+Jiang%2C+and+Benyou+Wang%2E+2024%2E+PlatoLM%3A+Teaching+LLMs+in+multi%2Dround+dialogue+via+a+user+simulator%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+Association+f)
17. Hadas Kotek, Rikker Dockum, and David Q. Sun. 2023. Gender bias and stereotypes in Large Language Models. In Proceedings of The ACM Collective Intelligence Conference, pages 12–24. ArXiv:2308.14921 [cs]. — [arXiv:2308.14921](https://arxiv.org/abs/2308.14921) [find](https://scholar.google.com/scholar?q=Hadas+Kotek%2C+Rikker+Dockum%2C+and+David+Q%2E+Sun%2E+2023%2E+Gender+bias+and+stereotypes+in+Large+Language+Models%2E+In+Proceedings+of+The+ACM+Collective+Intelligence+Conference%2C+pages+12%E2%80%9324%2E+ArXiv%3A2308%2E14921+%5Bc)
18. Kristopher Kyle. 2025. lexical_diversity. [find](https://scholar.google.com/scholar?q=Kristopher+Kyle%2E+2025%2E+lexical%5Fdiversity%2E)
19. Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. 2025. [Llms get lost in multi-turn conversation](https://md.research.elamur.ai/free/en/papers/laban-2025-llms-get-lost-multi-turn/). Preprint, arXiv:2505.06120.
20. Preethi Lahoti, Nicholas Blumm, Xiao Ma, Raghavendra Kotikalapudi, Sahitya Potluri, Qijun Tan, Hansa Srinivasan, Ben Packer, Ahmad Beirami, Alex Beutel, and Jilin Chen. 2023. Improving Diversity of Demographic Representation in Large Language Models via Collective-Critiques and Self-Voting. arXiv pr… [find](https://scholar.google.com/scholar?q=Preethi+Lahoti%2C+Nicholas+Blumm%2C+Xiao+Ma%2C+Raghavendra+Kotikalapudi%2C+Sahitya+Potluri%2C+Qijun+Tan%2C+Hansa+Srinivasan%2C+Ben+Packer%2C+Ahmad+Beirami%2C+Alex+Beutel%2C+and+Jilin+Chen%2E+2023%2E+Improving+Diversity+of+De)
21. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. [A diversity-promoting objective function for neural conversation models](https://md.research.elamur.ai/free/en/papers/li-2016-diversity-promoting-objective-function-neural/). In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Sa…
22. Xiang Luo, Zhiwen Tang, Jin Wang, and Xuejie Zhang. 2024. DuetSim: Building User Simulator with Dual Large Language Models for Task-Oriented Dialogues. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages… [find](https://scholar.google.com/scholar?q=Xiang+Luo%2C+Zhiwen+Tang%2C+Jin+Wang%2C+and+Xuejie+Zhang%2E+2024%2E+DuetSim%3A+Building+User+Simulator+with+Dual+Large+Language+Models+for+Task%2DOriented+Dialogues%2E+In+Proceedings+of+the+2024+Joint+International+C)
23. Alistair Mavin, Philip Wilkinson, Adrian Harwood, and Mark Novak. 2009. Easy approach to requirements syntax (ears). In 2009 17th IEEE International Requirements Engineering Conference, pages 317–322. trends and open problems. In Findings of the Association for Computational Linguistics: EMNLP 2023,… [find](https://scholar.google.com/scholar?q=Alistair+Mavin%2C+Philip+Wilkinson%2C+Adrian+Harwood%2C+and+Mark+Novak%2E+2009%2E+Easy+approach+to+requirements+syntax+%28ears%29%2E+In+2009+17th+IEEE+International+Requirements+Engineering+Conference%2C+pages+317%E2%80%93322%2E)
24. Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. [StereoSet: Measuring stereotypical bias in pretrained language models](https://md.research.elamur.ai/free/en/papers/nadeem-2021-stereoset-measuring-stereotypical-bias-pretrained/). arXiv preprint. ArXiv:2004.09456 [cs].
25. OpenAI. 2025a. OpenAI: Introducing GPT-4.1 in the api. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2025a%2E+OpenAI%3A+Introducing+GPT%2D4%2E1+in+the+api%2E)
26. OpenAI. 2025b. OpenAI: Introducing GPT-5. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2025b%2E+OpenAI%3A+Introducing+GPT%2D5%2E)
27. Diksha Panwar, Swati Anand, Farmaan Ali, and K. Singal. 2019. Consumer decision making process models and their applications to market strategy. International Management Review, 15:36. [find](https://scholar.google.com/scholar?q=Diksha+Panwar%2C+Swati+Anand%2C+Farmaan+Ali%2C+and+K%2E+Singal%2E+2019%2E+Consumer+decision+making+process+models+and+their+applications+to+market+strategy%2E+International+Management+Review%2C+15%3A36%2E)
28. Qwen Team. 2024. Qwen2.5: A party of foundation models. [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+2024%2E+Qwen2%2E5%3A+A+party+of+foundation+models%2E)
29. Scott I. Rick, Cynthia E. Cryder, and George Loewenstein. 2008. Tightwads and spendthrifts. Journal of consumer research, 34(6):767–782. [find](https://scholar.google.com/scholar?q=Scott+I%2E+Rick%2C+Cynthia+E%2E+Cryder%2C+and+George+Loewenstein%2E+2008%2E+Tightwads+and+spendthrifts%2E+Journal+of+consumer+research%2C+34%286%29%3A767%E2%80%93782%2E)
30. Susana Santos and Helena Martins Gonçalves. 2021. The consumer decision journey: A literature review of the foundational models and theories and a future perspective. Technological Forecasting and Social Change, 173:121117. [find](https://scholar.google.com/scholar?q=Susana+Santos+and+Helena+Martins+Gon%C3%A7alves%2E+2021%2E+The+consumer+decision+journey%3A+A+literature+review+of+the+foundational+models+and+theories+and+a+future+perspective%2E+Technological+Forecasting+and+Soc)
31. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. [Whose Opinions Do Language Models Reflect?](https://md.research.elamur.ai/free/en/papers/santurkar-2023-whose-opinions-language-models-reflect/) arXiv preprint. ArXiv:2303.17548 [cs].
32. Ivan Sekulić, Silvia Terragni, Victor Guimarães, Nghia Khau, Bruna Guedes, Modestas Filipavicius, André Ferreira Manso, and Roland Mathis. 2024. Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems. arXiv preprint. ArXiv:2402.13374 [cs]. [find](https://scholar.google.com/scholar?q=Ivan+Sekulic%CC%81%2C+Silvia+Terragni%2C+Victor+Guimar%C3%A3es%2C+Nghia+Khau%2C+Bruna+Guedes%2C+Modestas+Filipavicius%2C+Andr%C3%A9+Ferreira+Manso%2C+and+Roland+Mathis%2E+2024%2E+Reliable+LLM%2Dbased+User+Simulator+for+Task%2DOriented+Di)
33. Weiyan Shi, Kun Qian, Xuewei Wang, and Zhou Yu. 2019. How to build user simulators to train RL-based dialog systems. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pag… [find](https://scholar.google.com/scholar?q=Weiyan+Shi%2C+Kun+Qian%2C+Xuewei+Wang%2C+and+Zhou+Yu%2E+2019%2E+How+to+build+user+simulators+to+train+RL%2Dbased+dialog+systems%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Proc)
34. Lu Sun, Shihan Fu, Bingsheng Yao, Yuxuan Lu, Wenbo Li, Hansu Gu, Jiri Gesi, Jing Huang, Chen Luo, and Dakuo Wang. 2025. Llm agent meets agentic ai: Can llm agents simulate customers to evaluate agentic-ai-based shopping assistants? Preprint, arXiv:2509.21501. — [arXiv:2509.21501](https://arxiv.org/abs/2509.21501) [find](https://scholar.google.com/scholar?q=Lu+Sun%2C+Shihan+Fu%2C+Bingsheng+Yao%2C+Yuxuan+Lu%2C+Wenbo+Li%2C+Hansu+Gu%2C+Jiri+Gesi%2C+Jing+Huang%2C+Chen+Luo%2C+and+Dakuo+Wang%2E+2025%2E+Llm+agent+meets+agentic+ai%3A+Can+llm+agents+simulate+customers+to+evaluate+agenti)
35. Vinod Muthusamy, Yara Rizk, Kiran Kate, Praveen Venkateswaran, Vatche Isahagian, Ashu Gulati, and Parijat Dube. 2023. Towards large language model-based personal agents in the enterprise: Current Yuchong Sun, Che Liu, Kun Zhou, Jinwen Huang, Ruihua Song, Xin Zhao, Fuzheng Zhang, Di Zhang, and Kun Ga… [find](https://scholar.google.com/scholar?q=Vinod+Muthusamy%2C+Yara+Rizk%2C+Kiran+Kate%2C+Praveen+Venkateswaran%2C+Vatche+Isahagian%2C+Ashu+Gulati%2C+and+Parijat+Dube%2E+2023%2E+Towards+large+language+model%2Dbased+personal+agents+in+the+enterprise%3A+Current+Yuch)
36. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9729–9750, Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Proceedings+of+the+62nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Papers%29%2C+pages+9729%E2%80%939750%2C+Bangkok%2C+Thailand%2E+Association+for+Computational+Linguistics%2E)
37. Olivier Toubia, George Z. Gui, Tianyi Peng, Daniel J. Merlau, Ang Li, and Haozhe Chen. 2025. Twin-2K-500: A dataset for building digital twins of over 2,000 people based on their answers to over 500 questions. arXiv preprint. ArXiv:2505.17479 [cs]. — [arXiv:2505.17479](https://arxiv.org/abs/2505.17479) [find](https://scholar.google.com/scholar?q=Olivier+Toubia%2C+George+Z%2E+Gui%2C+Tianyi+Peng%2C+Daniel+J%2E+Merlau%2C+Ang+Li%2C+and+Haozhe+Chen%2E+2025%2E+Twin%2D2K%2D500%3A+A+dataset+for+building+digital+twins+of+over+2%2C000+people+based+on+their+answers+to+over+500+q)
38. Kuang Wang, Xianfei Li, Shenghao Yang, Li Zhou, Feng Jiang, and Haizhou Li. 2025. Know you first and be you better: Modeling human-like user simulators via implicit profiles. Preprint, arXiv:2502.18968. [find](https://scholar.google.com/scholar?q=Kuang+Wang%2C+Xianfei+Li%2C+Shenghao+Yang%2C+Li+Zhou%2C+Feng+Jiang%2C+and+Haizhou+Li%2E+2025%2E+Know+you+first+and+be+you+better%3A+Modeling+human%2Dlike+user+simulators+via+implicit+profiles%2E+Preprint%2C+arXiv%3A2502%2E1896)
39. Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. 2024. HelpSteer2: Open-source dataset for training top-performing reward models. arXiv preprint. ArXiv:2406.08673 [cs]. {Art T.} Weinstein. 1994. Market Se… [find](https://scholar.google.com/scholar?q=Zhilin+Wang%2C+Yi+Dong%2C+Olivier+Delalleau%2C+Jiaqi+Zeng%2C+Gerald+Shen%2C+Daniel+Egert%2C+Jimmy+J%2E+Zhang%2C+Makesh+Narsimhan+Sreedhar%2C+and+Oleksii+Kuchaiev%2E+2024%2E+HelpSteer2%3A+Open%2Dsource+dataset+for+training+top%2D)
40. Jeanine M. Williamson. 2018. Chapter 1 - Individual Differences. In Jeanine M. Williamson, editor, Teaching to Individual Differences in Science and Engineering Librarianship, pages 1–10. Chandos Publishing. [find](https://scholar.google.com/scholar?q=Jeanine+M%2E+Williamson%2E+2018%2E+Chapter+1+%2D+Individual+Differences%2E+In+Jeanine+M%2E+Williamson%2C+editor%2C+Teaching+to+Individual+Differences+in+Science+and+Engineering+Librarianship%2C+pages+1%E2%80%9310%2E+Chandos+Publ)
41. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2024. [$τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://md.research.elamur.ai/free/en/papers/yao-2024-tau-bench-benchmark-tool/). arXiv preprint. ArXiv:2406.12045.
42. Qiyuan Zhang, Yufei Wang, Yuxin Jiang, Liangyou Li, Chuhan Wu, Yasheng Wang, Xin Jiang, Lifeng Shang, Ruiming Tang, Fuyuan Lyu, and Chen Ma. 2025. [Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judge](https://md.research.elamur.ai/free/en/papers/zhang-2025-crowd-comparative-reasoning-unlocking-comprehensive/). In Proceedings of the 63rd Annual Meeting of the Association for Com…

*Updated 31 October 2026*