---
title: "APIGen-MT — paper by Prabhakar et al. (2025)"
description: "First, an LLM-driven generator creates task \"blueprints.\" Each blueprint pairs a user intent with… 56 references, cited by 34 papers here."
authors: ["Akshara Prabhakar", "Zuxin Liu", "Ming Zhu", "Jianguo Zhang", "Tulika Awalgaonkar", "Shiyu Wang", "Zhiwei Liu", "Haolin Chen", "Thai Hoang", "Juan Carlos Niebles", "Shelby Heinecke", "Weiran Yao", "Huan Wang", "Silvio Savarese", "Caiming Xiong"]
year: 2025
venue: "arXiv:2504.03601"
source: "https://arxiv.org/abs/2504.03601"
doi: "10.48550/arxiv.2504.03601"
arxiv: "2504.03601"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/prabhakar-2025-apigen-mt-agentic-pipeline-multi/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/prabhakar-2025-apigen-mt-agentic-pipeline-multi/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/) / APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay

# APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay

**Authors:** [Akshara Prabhakar](https://md.research.elamur.ai/free/en/authors/akshara-prabhakar/), [Zuxin Liu](https://md.research.elamur.ai/free/en/authors/zuxin-liu/), [Ming Zhu](https://md.research.elamur.ai/free/en/authors/ming-zhu/), [Jianguo Zhang](https://md.research.elamur.ai/free/en/authors/jianguo-zhang/), Tulika Awalgaonkar, [Shiyu Wang](https://md.research.elamur.ai/free/en/authors/shiyu-wang/), [Zhiwei Liu](https://md.research.elamur.ai/free/en/authors/zhiwei-liu/), Haolin Chen, [Thai Hoang](https://md.research.elamur.ai/free/en/authors/thai-hoang/), [Juan Carlos Niebles](https://md.research.elamur.ai/free/en/authors/juan-carlos-niebles/), [Shelby Heinecke](https://md.research.elamur.ai/free/en/authors/shelby-heinecke/), [Weiran Yao](https://md.research.elamur.ai/free/en/authors/weiran-yao/), [Huan Wang](https://md.research.elamur.ai/free/en/authors/huan-wang/), [Silvio Savarese](https://md.research.elamur.ai/free/en/authors/silvio-savarese/) and [Caiming Xiong](https://md.research.elamur.ai/free/en/authors/caiming-xiong/) · **Year:** 2025 · **Venue:** [arXiv:2504.03601](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2025](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2025/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2504.03601) · [PDF](https://arxiv.org/pdf/2504.03601) · [DOI 10.48550/arxiv.2504.03601](https://doi.org/10.48550/arxiv.2504.03601) · [arXiv 2504.03601](https://arxiv.org/abs/2504.03601) · [OpenAlex](https://openalex.org/W4415980674)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [34](https://md.research.elamur.ai/free/en/papers/prabhakar-2025-apigen-mt-agentic-pipeline-multi/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Building capable multi-turn AI agents requires training data that captures realistic exchanges between humans and agents. Such data is rare and costly to collect by hand. APIGen-MT, from Salesforce AI Research, addresses this with a two-stage synthetic pipeline. First, an LLM-driven generator creates task "blueprints." Each blueprint pairs a user intent with ground-truth API actions and expected outputs, built from sampled APIs, domain rules, data and user personas. Every blueprint must pass format and execution checks and a majority vote by an LLM reviewer committee, and failures are sent back with summarized feedback for revision. Second, each validated blueprint guides a simulated conversation in which an LLM acts as the human user while the agent under test works inside an environment where its actions are actually run. Only trajectories whose actions match the ground truth are kept. The authors fine-tuned Llama and Qwen models on this data to create the xLAM-2-fc-r family. These models beat open baselines and rival or surpass GPT-4o and Claude 3.5 on τ-bench and BFCL v3. Smaller models perform especially well, which shows that verified synthetic data can substitute for expensive manual annotation.

## Key points
- **Released resources:** 5K trajectories (APIGen-MT-5k) and the xLAM-2-fc-r models, published on Hugging Face (Salesforce/APIGen-MT-5k, Salesforce/xLAM-2).
- **Model family:** Models range from 1B to 70B parameters and are built on Llama 3.1/3.2 and Qwen 2.5.
- **Task modeling and complexity:** The authors frame multi-turn interaction as a POMDP. They use "reverse task recombination" to make tasks more complex.
- **Results on τ-retail:** Over 5 trials, xLAM-2-70b-fc-r reaches a 67.1% average success rate, compared with 62.8% for gpt-4o.
  - A Best-of-N user simulator raises scores to 68.8% and 67.0% respectively.
  - It also cuts variance: from 9.7 to 4.0 for xLAM and from 11.1 to 2.6 for gpt-4o.
- **Consistency on τ-airline:** xLAM-2-70b-fc-r scores higher than Claude on pass^5 despite a slightly lower pass^1, indicating more reliable behavior across trials.
- **Limitations:** The pipeline discards failed trajectories and its validation adds compute overhead.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 506 works · 330,052 citations
- [Salesforce](https://md.research.elamur.ai/free/en/companies/salesforce/) — Company, San Francisco, United States · 59 works · 18,172 citations
- [OpenAI](https://md.research.elamur.ai/free/en/companies/openai/) — Company, San Francisco, United States · 23 works · 27,874 citations
- [Westlake University](https://md.research.elamur.ai/free/en/universities/westlake-university/) — University, Hangzhou, China · 51 works · 8,572 citations
- [Emory University](https://md.research.elamur.ai/free/en/universities/emory-university/) — University, Atlanta, United States · 42 works · 39,899 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 1,105 works

## Cite

Cite as: Akshara Prabhakar, Zuxin Liu, Ming Zhu, Jianguo Zhang, Tulika Awalgaonkar, Shiyu Wang, Zhiwei Liu, Haolin Chen, Thai Hoang, Juan Carlos Niebles, Shelby Heinecke, Weiran Yao, Huan Wang, Silvio Savarese and Caiming Xiong. 2025. APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay. arXiv:2504.03601.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{prabhakar2025apigen,
  title = {{APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay}},
  author = {Akshara Prabhakar, Zuxin Liu, Ming Zhu, Jianguo Zhang, Tulika Awalgaonkar, Shiyu Wang, Zhiwei Liu, Haolin Chen, Thai Hoang, Juan Carlos Niebles, Shelby Heinecke, Weiran Yao, Huan Wang, Silvio Savarese and Caiming Xiong},
  year = {2025},
  doi = {10.48550/arxiv.2504.03601},
  eprint = {2504.03601},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2504.03601}
}
```

- [← previous AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator](https://md.research.elamur.ai/free/en/papers/fan-2025-ai-hospital-benchmarking-large-language/)
- [next → DeAL: Decoding-time Alignment for Large Language Models](https://md.research.elamur.ai/free/en/papers/huang-2025-deal-decoding-time-alignment-large/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/)

## References

56 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. S. Agashe, J. Han, S. Gan, J. Yang, A. Li, and X. E. Wang. Agent s: An open agentic framework that uses computers like a human. arXiv preprint arXiv:2410.08164, 2024. — [arXiv:2410.08164](https://arxiv.org/abs/2410.08164) [find](https://scholar.google.com/scholar?q=S%2E+Agashe%2C+J%2E+Han%2C+S%2E+Gan%2C+J%2E+Yang%2C+A%2E+Li%2C+and+X%2E+E%2E+Wang%2E+Agent+s%3A+An+open+agentic+framework+that+uses+computers+like+a+human%2E+arXiv+preprint+arXiv%3A2410%2E08164%2C+2024%2E+%E2%80%94+%5BarXiv%3A2410%2E08164%5D%28https%3A%2F%2Farxi)
2. Anthropic. Claude 3.5 sonnet, 2024. 3-5-models-and-computer-use. URL [https://www.anthropic.com/news/Claude](https://www.anthropic.com/news/Claude) 3.7 sonnet, 2025. [↗](https://www.anthropic.com/news/Claude)
3. Anthropic. claude-3-7-sonnet. URL [https://www.anthropic.com/news/](https://www.anthropic.com/news/) [↗](https://www.anthropic.com/news/)
4. Anthropic. Claude think tool, 2025. URL [https://www.anthropic.com/engineering/](https://www.anthropic.com/engineering/) claude-think-tool. [↗](https://www.anthropic.com/engineering/)
5. A. Antoniades, A. Örwall, K. Zhang, Y. Xie, A. Goyal, and W. Wang. Swe-search: Enhancing software agents with monte carlo tree search and iterative refinement. arXiv preprint arXiv:2410.20285, 2024. [find](https://scholar.google.com/scholar?q=A%2E+Antoniades%2C+A%2E+%C3%96rwall%2C+K%2E+Zhang%2C+Y%2E+Xie%2C+A%2E+Goyal%2C+and+W%2E+Wang%2E+Swe%2Dsearch%3A+Enhancing+software+agents+with+monte+carlo+tree+search+and+iterative+refinement%2E+arXiv+preprint+arXiv%3A2410%2E20285%2C+2024%2E)
6. S. Arcadinho, D. Aparício, and M. Almeida. Automated test generation to evaluate tool-augmented llms as conversational ai agents. arXiv preprint arXiv:2409.15934, 2024. [find](https://scholar.google.com/scholar?q=S%2E+Arcadinho%2C+D%2E+Apar%C3%ADcio%2C+and+M%2E+Almeida%2E+Automated+test+generation+to+evaluate+tool%2Daugmented+llms+as+conversational+ai+agents%2E+arXiv+preprint+arXiv%3A2409%2E15934%2C+2024%2E)
7. D. Bahdanau, N. Gontier, G. Huang, E. Kamalloo, R. Pardinas, A. Piché, T. Scholak, O. Shliazhko, J. P. Tremblay, K. Ghanem, et al. Tapeagents: a holistic framework for agent development and optimization. arXiv preprint arXiv:2412.08445, 2024. [find](https://scholar.google.com/scholar?q=D%2E+Bahdanau%2C+N%2E+Gontier%2C+G%2E+Huang%2C+E%2E+Kamalloo%2C+R%2E+Pardinas%2C+A%2E+Pich%C3%A9%2C+T%2E+Scholak%2C+O%2E+Shliazhko%2C+J%2E+P%2E+Tremblay%2C+K%2E+Ghanem%2C+et+al%2E+Tapeagents%3A+a+holistic+framework+for+agent+development+and+optimizati)
8. Z. Bi, K. Han, C. Liu, Y. Tang, and Y. Wang. [Forest-of-thought: Scaling test-time compute for enhancing llm reasoning](https://md.research.elamur.ai/free/en/papers/bi-2024-forest-thought-scaling-test-time/). arXiv preprint arXiv:2412.09078, 2024.
9. T. Cai, X. Wang, T. Ma, X. Chen, and D. Zhou. [Large language models as tool makers](https://md.research.elamur.ai/free/en/papers/cai-2023-large-language-models-tool-makers/). arXiv preprint arXiv:2305.17126, 2023.
10. CAMEL-AI.org. Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation. [https://github.com/camel-ai/owl](https://github.com/camel-ai/owl), 2025. Accessed: 2025-03-07. [↗](https://github.com/camel-ai/owl)
11. M. Chen, sunhaoze, T. Li, F. Yang, H. Liang, KeerLu, B. CUI, W. Zhang, Z. Zhou, and weipeng chen. Facilitating multi-turn function calling for LLMs via compositional instruction tuning. In The Thirteenth International Conference on Learning Representations, 2025. URL [https://openreview.net/forum?id=…](https://openreview.net/forum?id=…) [↗](https://openreview.net/forum?id=…)
12. Z. Chen, K. Liu, Q. Wang, W. Zhang, J. Liu, D. Lin, K. Chen, and F. Zhao. [Agent-flan: Designing data and methods of effective agent tuning for large language models](https://md.research.elamur.ai/free/en/papers/chen-2024-agent-flan-designing-data-methods/). arXiv preprint arXiv:2403.12881, 2024.
13. Z. Chen, M. Li, Y. Huang, Y. Du, M. Fang, and T. Zhou. Atlas: Agent tuning via learning critical steps. arXiv preprint arXiv:2503.02197, 2025. [find](https://scholar.google.com/scholar?q=Z%2E+Chen%2C+M%2E+Li%2C+Y%2E+Huang%2C+Y%2E+Du%2C+M%2E+Fang%2C+and+T%2E+Zhou%2E+Atlas%3A+Agent+tuning+via+learning+critical+steps%2E+arXiv+preprint+arXiv%3A2503%2E02197%2C+2025%2E)
14. S. Cognition. Apt-1 blog, 2025. URL [https://www.scaledcognition.com/blog/apt-1](https://www.scaledcognition.com/blog/apt-1). [↗](https://www.scaledcognition.com/blog/apt-1)
15. T. Dao. [Flashattention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/), 2023. URL https://arxiv.org/abs/2307.08691. [↗](https://arxiv.org/abs/2307.08691)
16. A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783, 2024.
17. T. Ge, X. Chan, X. Wang, D. Yu, H. Mi, and D. Yu. [Scaling synthetic data creation with 1,000,000,000 personas](https://md.research.elamur.ai/free/en/papers/ge-2024-scaling-synthetic-data-creation-1/), 2024. URL https://arxiv.org/abs/2406.20094. [↗](https://arxiv.org/abs/2406.20094)
18. S. A. Hayati, T. Jung, T. Bodding-Long, S. Kar, A. Sethy, J.-K. Kim, and D. Kang. Chain-of-instructions: Compositional instruction tuning on large language models. arXiv preprint arXiv:2402.11532, 2024. — [arXiv:2402.11532](https://arxiv.org/abs/2402.11532) [find](https://scholar.google.com/scholar?q=S%2E+A%2E+Hayati%2C+T%2E+Jung%2C+T%2E+Bodding%2DLong%2C+S%2E+Kar%2C+A%2E+Sethy%2C+J%2E%2DK%2E+Kim%2C+and+D%2E+Kang%2E+Chain%2Dof%2Dinstructions%3A+Compositional+instruction+tuning+on+large+language+models%2E+arXiv+preprint+arXiv%3A2402%2E11532%2C+202)
19. K.-H. Huang, A. Prabhakar, S. Dhawan, Y. Mao, H. Wang, S. Savarese, C. Xiong, P. Laban, and C.-S. Wu. [Crmarena: Understanding the capacity of llm agents to perform professional crm tasks in realistic environments](https://md.research.elamur.ai/free/en/papers/huang-2025-crmarena-understanding-capacity-llm-agents/), 2025. URL https://arxiv.org/abs/2411.02305. [↗](https://arxiv.org/abs/2411.02305)
20. E. Levi and I. Kadar. Intellagent: A multi-agent framework for evaluating conversational ai systems. arXiv preprint arXiv:2501.11067, 2025. — [arXiv:2501.11067](https://arxiv.org/abs/2501.11067) [find](https://scholar.google.com/scholar?q=E%2E+Levi+and+I%2E+Kadar%2E+Intellagent%3A+A+multi%2Dagent+framework+for+evaluating+conversational+ai+systems%2E+arXiv+preprint+arXiv%3A2501%2E11067%2C+2025%2E+%E2%80%94+%5BarXiv%3A2501%2E11067%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2501%2E11067%29)
21. G. Li, H. A. A. K. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem. [Camel: Communicative agents for "mind" exploration of large language model society](https://md.research.elamur.ai/free/en/papers/li-2023-camel-communicative-agents-mind/). In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
22. X. Li, H. Zou, and P. Liu. [Torl: Scaling tool-integrated rl](https://md.research.elamur.ai/free/en/papers/li-2025-torl-scaling-tool-integrated-rl/), 2025. URL [https://arxiv.org/](https://arxiv.org/) abs/2503.23383. [↗](https://arxiv.org/)
23. Y. Li, Y. Li, X. Wang, Y. Jiang, Z. Zhang, X. Zheng, H. Wang, H.-T. Zheng, P. Xie, P. S. Yu, et al. Benchmarking multimodal retrieval augmented generation with dynamic vqa dataset and self-adaptive planning agent. arXiv preprint arXiv:2411.02937, 2024. — [arXiv:2411.02937](https://arxiv.org/abs/2411.02937) [find](https://scholar.google.com/scholar?q=Y%2E+Li%2C+Y%2E+Li%2C+X%2E+Wang%2C+Y%2E+Jiang%2C+Z%2E+Zhang%2C+X%2E+Zheng%2C+H%2E+Wang%2C+H%2E%2DT%2E+Zheng%2C+P%2E+Xie%2C+P%2E+S%2E+Yu%2C+et+al%2E+Benchmarking+multimodal+retrieval+augmented+generation+with+dynamic+vqa+dataset+and+self%2Dadaptive+pl)
24. W. Liu, X. Huang, X. Zeng, X. Hao, S. Yu, D. Li, S. Wang, W. Gan, Z. Liu, Y. Yu, et al. [Toolace: Winning the points of llm function calling](https://md.research.elamur.ai/free/en/papers/liu-2024-toolace-winning-points-llm-function/). arXiv preprint arXiv:2409.00920, 2024.
25. Z. Liu, J. Zhang, K. Asadi, Y. Liu, D. Zhao, S. Sabach, and R. Fakoor. Tail: Task-specific adapters for imitation learning with large pretrained models. arXiv preprint arXiv:2310.05905, 2023. — [arXiv:2310.05905](https://arxiv.org/abs/2310.05905) [find](https://scholar.google.com/scholar?q=Z%2E+Liu%2C+J%2E+Zhang%2C+K%2E+Asadi%2C+Y%2E+Liu%2C+D%2E+Zhao%2C+S%2E+Sabach%2C+and+R%2E+Fakoor%2E+Tail%3A+Task%2Dspecific+adapters+for+imitation+learning+with+large+pretrained+models%2E+arXiv+preprint+arXiv%3A2310%2E05905%2C+2023%2E+%E2%80%94+%5BarXiv)
26. Z. Liu, T. Hoang, J. Zhang, M. Zhu, T. Lan, J. Tan, W. Yao, Z. Liu, Y. Feng, R. RN, et al. Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. Advances in Neural Information Processing Systems, 37:54463–54482, 2024. [find](https://scholar.google.com/scholar?q=Z%2E+Liu%2C+T%2E+Hoang%2C+J%2E+Zhang%2C+M%2E+Zhu%2C+T%2E+Lan%2C+J%2E+Tan%2C+W%2E+Yao%2C+Z%2E+Liu%2C+Y%2E+Feng%2C+R%2E+RN%2C+et+al%2E+Apigen%3A+Automated+pipeline+for+generating+verifiable+and+diverse+function%2Dcalling+datasets%2E+Advances+in+Neura)
27. I. Loshchilov and F. Hutter. arXiv:1711.05101, 2017. Decoupled weight decay regularization. arXiv preprint [find](https://scholar.google.com/scholar?q=I%2E+Loshchilov+and+F%2E+Hutter%2E+arXiv%3A1711%2E05101%2C+2017%2E+Decoupled+weight+decay+regularization%2E+arXiv+preprint)
28. J. Lu, T. Holleis, Y. Zhang, B. Aumayer, F. Nan, F. Bai, S. Ma, S. Ma, M. Li, G. Yin, et al. [Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities](https://md.research.elamur.ai/free/en/papers/lu-2025-toolsandbox-stateful-conversational-interactive-evaluation/). arXiv preprint arXiv:2408.04682, 2024.
29. A. Mitra, S. Patel, T. Chakrabarty, and C. Baral. Agentinstruct: An agentic framework for generating high-quality synthetic instruction data. arXiv preprint arXiv:2402.12360, 2024. — [arXiv:2402.12360](https://arxiv.org/abs/2402.12360) [find](https://scholar.google.com/scholar?q=A%2E+Mitra%2C+S%2E+Patel%2C+T%2E+Chakrabarty%2C+and+C%2E+Baral%2E+Agentinstruct%3A+An+agentic+framework+for+generating+high%2Dquality+synthetic+instruction+data%2E+arXiv+preprint+arXiv%3A2402%2E12360%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E12360)
30. J. Pan, X. Wang, G. Neubig, N. Jaitly, H. Ji, A. Suhr, and Y. Zhang. [Training software engineering agents and verifiers with swe-gym](https://md.research.elamur.ai/free/en/papers/pan-2024-training-software-engineering-agents/). arXiv preprint arXiv:2412.21139, 2024.
31. J. Pan, R. Shar, J. Pfau, A. Talwalkar, H. He, and V. Chen. When benchmarks talk: Re-evaluating code llms with interactive feedback, 2025. URL https://arxiv.org/abs/2502.18413. [↗](https://arxiv.org/abs/2502.18413)
32. J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22, 2023. [find](https://scholar.google.com/scholar?q=J%2E+S%2E+Park%2C+J%2E+O%E2%80%99Brien%2C+C%2E+J%2E+Cai%2C+M%2E+R%2E+Morris%2C+P%2E+Liang%2C+and+M%2E+S%2E+Bernstein%2E+Generative+agents%3A+Interactive+simulacra+of+human+behavior%2E+In+Proceedings+of+the+36th+annual+acm+symposium+on+user+inte)
33. Y. Qin, S. Hu, Y. Lin, W. Chen, N. Ding, G. Cui, Z. Zeng, X. Zhou, Y. Huang, C. Xiao, et al. [Tool learning with foundation models](https://md.research.elamur.ai/free/en/papers/qin-2023-tool-learning-foundation-models/). ACM Computing Surveys, 57(4):1–40, 2024.
34. Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan,… [find](https://scholar.google.com/scholar?q=Qwen%2C+%3A%2C+A%2E+Yang%2C+B%2E+Yang%2C+B%2E+Zhang%2C+B%2E+Hui%2C+B%2E+Zheng%2C+B%2E+Yu%2C+C%2E+Li%2C+D%2E+Liu%2C+F%2E+Huang%2C+H%2E+Wei%2C+H%2E+Lin%2C+J%2E+Yang%2C+J%2E+Tu%2C+J%2E+Zhang%2C+J%2E+Yang%2C+J%2E+Yang%2C+J%2E+Zhou%2C+J%2E+Lin%2C+K%2E+Dang%2C+K%2E+Lu%2C+K%2E+Bao%2C+K%2E+Yang%2C+L%2E+)
35. J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20, page 3505–3506, New York, NY, USA, 202… — [doi:10.1145/3394486.3406703](https://doi.org/10.1145/3394486.3406703) [find](https://scholar.google.com/scholar?q=J%2E+Rasley%2C+S%2E+Rajbhandari%2C+O%2E+Ruwase%2C+and+Y%2E+He%2E+Deepspeed%3A+System+optimizations+enable+training+deep+learning+models+with+over+100+billion+parameters%2E+In+Proceedings+of+the+26th+ACM+SIGKDD+Internatio)
36. T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom. [Toolformer: Language models can teach themselves to use tools](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/). Advances in Neural Information Processing Systems, 36:68539–68551, 2023.
37. S. Sengupta, K. Curtis, A. Mallipeddi, A. Mathur, J. Ross, and L. Gou. Mag-v: A multi-agent framework for synthetic data generation and verification. arXiv preprint arXiv:2412.04494, 2024. — [arXiv:2412.04494](https://arxiv.org/abs/2412.04494) [find](https://scholar.google.com/scholar?q=S%2E+Sengupta%2C+K%2E+Curtis%2C+A%2E+Mallipeddi%2C+A%2E+Mathur%2C+J%2E+Ross%2C+and+L%2E+Gou%2E+Mag%2Dv%3A+A+multi%2Dagent+framework+for+synthetic+data+generation+and+verification%2E+arXiv+preprint+arXiv%3A2412%2E04494%2C+2024%2E+%E2%80%94+%5BarXiv%3A24)
38. J. Shim, G. Seo, C. Lim, and Y. Jo. Tooldial: Multi-turn dialogue generation method for tool-augmented language models. In The Thirteenth International Conference on Learning Representations, 2025. URL [https://openreview.net/forum?id=J1J5eGJsKZ](https://openreview.net/forum?id=J1J5eGJsKZ). [↗](https://openreview.net/forum?id=J1J5eGJsKZ)
39. N. Shinn, F. Cassano, A. Gopinath, K. R. Narasimhan, and S. Yao. [Reflexion: language agents with verbal reinforcement learning](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/). In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL [https://openreview.net/forum?id=vAElhFcKW6](https://openreview.net/forum?id=vAElhFcKW6). [↗](https://openreview.net/forum?id=vAElhFcKW6)
40. V. Sirdeshmukh, K. Deshpande, J. Mols, L. Jin, E.-Y. Cardona, D. Lee, J. Kritz, W. Primack, S. Yue, and C. Xing. Multichallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier llms. arXiv preprint arXiv:2501.17399, 2025. [find](https://scholar.google.com/scholar?q=V%2E+Sirdeshmukh%2C+K%2E+Deshpande%2C+J%2E+Mols%2C+L%2E+Jin%2C+E%2E%2DY%2E+Cardona%2C+D%2E+Lee%2C+J%2E+Kritz%2C+W%2E+Primack%2C+S%2E+Yue%2C+and+C%2E+Xing%2E+Multichallenge%3A+A+realistic+multi%2Dturn+conversation+evaluation+benchmark+challenging+to)
41. H. Su, R. Sun, J. Yoon, P. Yin, T. Yu, and S. Ö. Arık. Learn-by-interact: A data-centric framework for self-adaptive agents in realistic environments. arXiv preprint arXiv:2501.10893, 2025. — [arXiv:2501.10893](https://arxiv.org/abs/2501.10893) [find](https://scholar.google.com/scholar?q=H%2E+Su%2C+R%2E+Sun%2C+J%2E+Yoon%2C+P%2E+Yin%2C+T%2E+Yu%2C+and+S%2E+%C3%96%2E+Ar%C4%B1k%2E+Learn%2Dby%2Dinteract%3A+A+data%2Dcentric+framework+for+self%2Dadaptive+agents+in+realistic+environments%2E+arXiv+preprint+arXiv%3A2501%2E10893%2C+2025%2E+%E2%80%94+%5BarXiv%3A2)
42. S. Tang, X. Pang, Z. Liu, B. Tang, R. Ye, X. Dong, Y. Wang, and S. Chen. [Synthesizing post-training data for llms through multi-agent simulation](https://md.research.elamur.ai/free/en/papers/tang-2025-synthesizing-post-training-data-llms/). arXiv preprint arXiv:2410.14251, 2024.
43. J. Wang, J. Zhou, M. Wen, X. Mo, H. Zhang, Q. Lin, C. Jin, X. Wang, W. Zhang, and Q. Peng. Hammerbench: Fine-grained function-calling evaluation in real mobile device scenarios. arXiv preprint arXiv:2412.16516, 2024. — [arXiv:2412.16516](https://arxiv.org/abs/2412.16516) [find](https://scholar.google.com/scholar?q=J%2E+Wang%2C+J%2E+Zhou%2C+M%2E+Wen%2C+X%2E+Mo%2C+H%2E+Zhang%2C+Q%2E+Lin%2C+C%2E+Jin%2C+X%2E+Wang%2C+W%2E+Zhang%2C+and+Q%2E+Peng%2E+Hammerbench%3A+Fine%2Dgrained+function%2Dcalling+evaluation+in+real+mobile+device+scenarios%2E+arXiv+preprint+arXiv%3A2)
44. G. Wölflein, D. Ferber, D. Truhn, O. Arandjelović, and J. N. Kather. [Llm agents making agent tools](https://md.research.elamur.ai/free/en/papers/wolflein-2025-llm-agents-making-agent-tools/). arXiv preprint arXiv:2502.11705, 2025.
45. F. Yan, H. Mao, C. C.-J. Ji, T. Zhang, S. G. Patil, I. Stoica, and J. E. Gonzalez. Berkeley function calling leaderboard. 2024. [find](https://scholar.google.com/scholar?q=F%2E+Yan%2C+H%2E+Mao%2C+C%2E+C%2E%2DJ%2E+Ji%2C+T%2E+Zhang%2C+S%2E+G%2E+Patil%2C+I%2E+Stoica%2C+and+J%2E+E%2E+Gonzalez%2E+Berkeley+function+calling+leaderboard%2E+2024%2E)
46. J. Yang, A. Prabhakar, S. Yao, K. Pei, and K. R. Narasimhan. Language agents as hackers: Evaluating cybersecurity skills with capture the flag. In Multi-Agent Security Workshop @ NeurIPS’23, 2023. URL [https://openreview.net/forum?id=KOZwk7BFc3](https://openreview.net/forum?id=KOZwk7BFc3). [↗](https://openreview.net/forum?id=KOZwk7BFc3)
47. J. Yang, A. Prabhakar, K. Narasimhan, and S. Yao. [Intercode: Standardizing and benchmarking interactive coding with execution feedback](https://md.research.elamur.ai/free/en/papers/yang-2023-intercode-standardizing-benchmarking-interactive-coding/). Advances in Neural Information Processing Systems, 36, 2024.
48. R. Yang, F. Ye, J. Li, S. Yuan, Y. Zhang, Z. Tu, X. Li, and D. Yang. The lighthouse of language: Enhancing llm agents via critique-guided improvement. arXiv preprint arXiv:2503.16024, 2025. — [arXiv:2503.16024](https://arxiv.org/abs/2503.16024) [find](https://scholar.google.com/scholar?q=R%2E+Yang%2C+F%2E+Ye%2C+J%2E+Li%2C+S%2E+Yuan%2C+Y%2E+Zhang%2C+Z%2E+Tu%2C+X%2E+Li%2C+and+D%2E+Yang%2E+The+lighthouse+of+language%3A+Enhancing+llm+agents+via+critique%2Dguided+improvement%2E+arXiv+preprint+arXiv%3A2503%2E16024%2C+2025%2E+%E2%80%94+%5BarXiv%3A2)
49. S. Yao, N. Shinn, P. Razavi, and K. Narasimhan. Tau-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.[12045, 2024](https://md.research.elamur.ai/free/en/papers/yao-2024-tau-bench-benchmark-tool/).
50. F. Yin, Z. Wang, I.-H. Hsu, J. Yan, K. Jiang, Y. Chen, J. Gu, L. T. Le, K.-W. Chang, C.-Y. Lee, H. Palangi, and T. Pfister. Magnet: Multi-turn tool-use data synthesis and distillation via graph translation, 2025. URL https://arxiv.org/abs/2503.07826. [↗](https://arxiv.org/abs/2503.07826)
51. Y. Zeng, X. Ding, Y. Wang, W. Liu, W. Ning, Y. Hou, X. Huang, B. Qin, and T. Liu. Boosting tool use of large language models via iterative reinforced fine-tuning. arXiv preprint arXiv:2501.09766, 2025. — [arXiv:2501.09766](https://arxiv.org/abs/2501.09766) [find](https://scholar.google.com/scholar?q=Y%2E+Zeng%2C+X%2E+Ding%2C+Y%2E+Wang%2C+W%2E+Liu%2C+W%2E+Ning%2C+Y%2E+Hou%2C+X%2E+Huang%2C+B%2E+Qin%2C+and+T%2E+Liu%2E+Boosting+tool+use+of+large+language+models+via+iterative+reinforced+fine%2Dtuning%2E+arXiv+preprint+arXiv%3A2501%2E09766%2C+2025)
52. J. Zhang, T. Lan, M. Zhu, Z. Liu, T. Hoang, S. Kokane, W. Yao, J. Tan, A. Prabhakar, H. Chen, et al. [xlam: A family of large action models to empower ai agent systems](https://md.research.elamur.ai/free/en/papers/zhang-2025-xlam-family-large-action-models/). arXiv preprint arXiv:2409.03215, 2024.
53. J. Zhang, T. Hoang, M. Zhu, Z. Liu, S. Wang, T. Awalgaonkar, A. Prabhakar, H. Chen, W. Yao, Z. Liu, et al. Actionstudio: A lightweight framework for data and training of action models. arXiv preprint arXiv:2503.22673, 2025. — [arXiv:2503.22673](https://arxiv.org/abs/2503.22673) [find](https://scholar.google.com/scholar?q=J%2E+Zhang%2C+T%2E+Hoang%2C+M%2E+Zhu%2C+Z%2E+Liu%2C+S%2E+Wang%2C+T%2E+Awalgaonkar%2C+A%2E+Prabhakar%2C+H%2E+Chen%2C+W%2E+Yao%2C+Z%2E+Liu%2C+et+al%2E+Actionstudio%3A+A+lightweight+framework+for+data+and+training+of+action+models%2E+arXiv+preprint+)
54. K. Zhang, W. Yao, Z. Liu, Y. Feng, Z. Liu, R. Rithesh, T. Lan, L. Li, R. Lou, J. Xu, et al. Diversity empowers intelligence: Integrating expertise of software engineering agents. In The Thirteenth International Conference on Learning Representations, 2024. [find](https://scholar.google.com/scholar?q=K%2E+Zhang%2C+W%2E+Yao%2C+Z%2E+Liu%2C+Y%2E+Feng%2C+Z%2E+Liu%2C+R%2E+Rithesh%2C+T%2E+Lan%2C+L%2E+Li%2C+R%2E+Lou%2C+J%2E+Xu%2C+et+al%2E+Diversity+empowers+intelligence%3A+Integrating+expertise+of+software+engineering+agents%2E+In+The+Thirteenth+Int)
55. Y. Zhang, J. Lu, and N. Jaitly. Probing the multi-turn planning capabilities of LLMs via 20 question games. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1495–1516, Bangkok, Th… [find](https://scholar.google.com/scholar?q=Y%2E+Zhang%2C+J%2E+Lu%2C+and+N%2E+Jaitly%2E+Probing+the+multi%2Dturn+planning+capabilities+of+LLMs+via+20+question+games%2E+In+L%2E%2DW%2E+Ku%2C+A%2E+Martins%2C+and+V%2E+Srikumar%2C+editors%2C+Proceedings+of+the+62nd+Annual+Meeting+of)
56. Y. Zheng, R. Zhang, J. Zhang, Y. Ye, Z. Luo, Z. Feng, and Y. Ma. [Llamafactory: Unified efficient fine-tuning of 100+ language models](https://md.research.elamur.ai/free/en/papers/zheng-2024-llamafactory-unified-efficient-fine-tuning/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for…

*Updated 2 November 2026*