---
title: "TPS-Bench — paper by Xu et al. (2026) — summary and ideas"
description: "Agent benchmarks usually check whether an LLM finishes a task. 44 references. Among them: 0."
authors: ["Hanwen Xu", "Xuyao Huang", "Yuzhe Liu", "Zhijie Deng"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1614/"
doi: "10.18653/v1/2026.acl-long.1614"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xu-2026-tps-bench-evaluating-ai-agents/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xu-2026-tps-bench-evaluating-ai-agents/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/54/) / TPS-Bench: Evaluating AI Agents' Tool Planning & Scheduling Abilities in Compounding Tasks

# TPS-Bench: Evaluating AI Agents' Tool Planning & Scheduling Abilities in Compounding Tasks

**Authors:** Hanwen Xu, Xuyao Huang, Yuzhe Liu and [Zhijie Deng](https://md.research.elamur.ai/free/en/authors/zhijie-deng/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1614/) · [PDF](https://aclanthology.org/2026.acl-long.1614.pdf) · [DOI 10.18653/v1/2026.acl-long.1614](https://doi.org/10.18653/v1/2026.acl-long.1614) · [OpenAlex](https://openalex.org/W7166836923)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Agent benchmarks usually check whether an LLM finishes a task. They rarely check how efficiently it coordinates tools when one request bundles several subtasks, such as a weather lookup, a flight status check and an attraction search. TPS-Bench addresses this by testing two abilities. Tool planning means choosing suitable tools from a large, heterogeneous repository. Tool scheduling means running independent subtasks in parallel while respecting dependencies. The authors collected 15 MCP servers exposing 141 tools and had LLMs generate subtasks from the tool descriptions. They combined these into 200 compounding tasks at two levels: Easy, with loosely related subtasks, and Hard, with tightly interdependent ones. An LLM-as-a-judge rates completion, and efficiency is measured in tokens, turns and wall-clock time. Across seven closed and open models, planning is mostly adequate but scheduling styles diverge. Sequential calling is more accurate but slow, while parallel calling is fast but less reliable. A small GRPO experiment on Qwen3-1.7B suggests that RL can make scheduling both faster and more accurate. This matters because real deployments care about latency and cost, not only success.

## Key points
- **Resource:** TPS-Bench has 200 compounding tasks in two levels, TPS-Bench-Easy (at most five subtasks) and TPS-Bench-Hard, built on 15 MCP servers with 141 tools. Models evaluated include GPT-4o, Kimi-K2, DeepSeek-R1, GLM-4.5, QwQ-32B, Qwen3-32B and Qwen3-1.7B.
- **Main results on TPS-Bench-Hard:**
  - GLM-4.5 achieves the best completion rate (64.72%) but relies on mostly sequential calls, costing about 217.8 s and 14k tokens per query.
  - GPT-4o favours parallel calls, taking about 76.84 s and 9k tokens, but reaches only 45.08%.
  - Qwen3-32B is the strongest open baseline at 56.72% with about 8.0k tokens.
- **Scheduling ablation:** Forcing serial execution costs more tokens and time. For DeepSeek-R1, tokens rose from 11.6k to 12.6k and time from 361.6 s to 423.6 s. However, it can raise accuracy; GLM-4.5 improved from 63.1% to 71.8%.
- **Tool-selection ablation:** Three strategies were compared: no selection, rule-based top-10 selection and model self-selection. Completion rates barely change, but token and time costs differ substantially.
- **RL study:** Qwen3-1.7B was trained with GRPO on 597 samples, using a reward from Gemini-2.5 flash that scores completion and parallelism. On the Hard level:
  - Task completion rose from 26.75% to 35.17%, and the tool selection score from 65.26% to 82.34%.
  - Time fell from 42.0 to 34.8, turns from 2.4 to 1.9, and output tokens from 2.2k to 1.6k.
  - The authors summarise this as roughly a 6% completion gain and a 14% time reduction. The table figures differ from this summary (about +8.4 points and −17%).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Shanghai Jiao Tong University](https://md.research.elamur.ai/free/en/universities/shanghai-jiao-tong-university/) — University, Shanghai, China · 212 works · 19,544 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Hanwen Xu, Xuyao Huang, Yuzhe Liu and Zhijie Deng. 2026. TPS-Bench: Evaluating AI Agents' Tool Planning & Scheduling Abilities in Compounding Tasks. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xu2026tps,
  title = {{TPS-Bench: Evaluating AI Agents' Tool Planning \& Scheduling Abilities in Compounding Tasks}},
  author = {Hanwen Xu, Xuyao Huang, Yuzhe Liu and Zhijie Deng},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1614},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1614}
}
```

- [← previous TPA: Next Token Probability Attribution for Detecting Hallucinations in RAG](https://md.research.elamur.ai/free/en/papers/lu-2026-tpa-next-token-probability-attribution/)
- [next → TRAC: Teacher-Guided Token Reward with Adaptive Calibration for Robust Policy Optimization](https://md.research.elamur.ai/free/en/papers/wu-2026-trac-teacher-guided-token-reward/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/54/)

## References

44 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Y. Bai, Kimi Team, Y. Bao, and et al. 2025. [Kimi k2: Open agentic intelligence](https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-k2-open-agentic/). Preprint, arXiv:2507.20534.
2. Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. 2025. [τ 2 -bench: Evaluating conversational agents in a dual-control environment](https://md.research.elamur.ai/free/en/papers/barres-2025-tau-2-bench-evaluating-conversational/). Preprint, arXiv:2506.07982.
3. Ma Chang, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. 2024. [Agentboard: An analytical evaluation board of multi-turn llm agents](https://md.research.elamur.ai/free/en/papers/ma-2024-agentboard-analytical-evaluation-board-multi/). Advances in neural information processing systems, 37:74325–74362.
4. Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, and 1 others. 2023. T-eval: Evaluating the tool utilization capability of large language models step by step. arXiv preprint arXiv:2312.14033. [find](https://scholar.google.com/scholar?q=Zehui+Chen%2C+Weihua+Du%2C+Wenwei+Zhang%2C+Kuikun+Liu%2C+Jiangning+Liu%2C+Miao+Zheng%2C+Jingming+Zhuo%2C+Songyang+Zhang%2C+Dahua+Lin%2C+Kai+Chen%2C+and+1+others%2E+2023%2E+T%2Deval%3A+Evaluating+the+tool+utilization+capability+o)
5. G. Comanici and 1 others. 2025. [Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities](https://md.research.elamur.ai/free/en/papers/comanici-2025-gemini-2-5-pushing-frontier/). https://arxiv.org/abs/2507.06261. ArXiv preprint. [↗](https://arxiv.org/abs/2507.06261)
6. DeepSeek-AI. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). Preprint, arXiv:2501.12948.
7. Zishan Guo, Yufei Huang, and Deyi Xiong. 2024. Ctooleval: A chinese benchmark for llm-powered agent evaluation in real-world api interactions. In Findings of the Association for Computational Linguistics ACL 2024, pages 15711–15724. [find](https://scholar.google.com/scholar?q=Zishan+Guo%2C+Yufei+Huang%2C+and+Deyi+Xiong%2E+2024%2E+Ctooleval%3A+A+chinese+benchmark+for+llm%2Dpowered+agent+evaluation+in+real%2Dworld+api+interactions%2E+In+Findings+of+the+Association+for+Computational+Linguist)
8. Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Chenxing Wei, Danyang Li, Jiaqi Chen, Jiayi Zhang, Jinlin Wang, Li Zhang, Lingyao Zhang, Min Yang, Mingchen Zhuge, Taicheng Guo, Tuo Zhou, Wei Tao, Xiangru Tang, and 8 others. 2024. [Data interpreter: An llm agent for data scien…](https://md.research.elamur.ai/free/en/papers/hong-2025-data-interpreter-llm-agent-data/)
9. Xinyi Hou, Yanjie Zhao, Shenao Wang, and Haoyu Wang. 2025. Model context protocol (mcp): Landscape, security threats, and future research directions. arXiv preprint arXiv:2503.23278. — [arXiv:2503.23278](https://arxiv.org/abs/2503.23278) [find](https://scholar.google.com/scholar?q=Xinyi+Hou%2C+Yanjie+Zhao%2C+Shenao+Wang%2C+and+Haoyu+Wang%2E+2025%2E+Model+context+protocol+%28mcp%29%3A+Landscape%2C+security+threats%2C+and+future+research+directions%2E+arXiv+preprint+arXiv%3A2503%2E23278%2E+%E2%80%94+%5BarXiv%3A2503%2E232)
10. Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and 1 others. 2023. [Meta-tool benchmark for large language models: Deciding whether to use tools and which to use](https://md.research.elamur.ai/free/en/papers/huang-2023-metatool-benchmark-large-language-models/). arXiv preprint arXiv:2310.03128.
11. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. [Swe-bench: Can language models resolve real-world github issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/) arXiv preprint arXiv:2310.06770.
12. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. [Swe-bench: Can language models resolve real-world github issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/) Preprint, arXiv:2310.06770.
13. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
14. Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Erran Li Li, Ruohan Zhang, and 1 others. 2024. Embodied agent interface: Benchmarking llms for embodied decision making. Advances in Neural Information Processing Systems, 37:100428–100534. [find](https://scholar.google.com/scholar?q=Manling+Li%2C+Shiyu+Zhao%2C+Qineng+Wang%2C+Kangrui+Wang%2C+Yu+Zhou%2C+Sanjana+Srivastava%2C+Cem+Gokmen%2C+Tony+Lee%2C+Erran+Li+Li%2C+Ruohan+Zhang%2C+and+1+others%2E+2024%2E+Embodied+agent+interface%3A+Benchmarking+llms+for+emb)
15. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, and 1 others. 2023. [Agentbench: Evaluating llms as agents](https://md.research.elamur.ai/free/en/papers/liu-2023-agentbench-evaluating-llms-agents/). arXiv preprint arXiv:2308.03688.
16. Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Hongyi Jin, Tianqi Chen, and Zhihao Jia. 2023. [Towards efficient generative large language model serving: A survey from algorithms to systems](https://md.research.elamur.ai/free/en/papers/miao-2023-towards-efficient-generative-large-language/). ACM Computing Surveys.
17. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022. [Webgpt: Browser-assis…](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/)
18. OpenAI. 2024. [GPT-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). Preprint, arXiv:2410.21276. Please cite as OpenAI (2024).
19. OpenAI. 2025. Introducing deep research. [https://openai.com/index/](https://openai.com/index/) introducing-deep-research/. Accessed 2025-09-23. [↗](https://openai.com/index/)
20. OpenAI. 2025. Introducing gpt-5. [https://openai.com/index/introducing-gpt-5/](https://openai.com/index/introducing-gpt-5/). Accessed 2025-09-24. [↗](https://openai.com/index/introducing-gpt-5/)
21. OpenAI. 2025. Openai o3 and o4-mini system card. [https://openai.com/index/](https://openai.com/index/) o3-o4-mini-system-card/. Accessed: 2025-09-24. [↗](https://openai.com/index/)
22. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. [Toolllm: Facilitating large language models to master 16000+ real…](https://md.research.elamur.ai/free/en/papers/qin-2023-toolllm-facilitating-large-language-models/)
23. Qwen. 2025. Qwq-32b: Embracing the power of reinforcement learning. [find](https://scholar.google.com/scholar?q=Qwen%2E+2025%2E+Qwq%2D32b%3A+Embracing+the+power+of+reinforcement+learning%2E)
24. Partha Pratim Ray. 2025. A survey on model context protocol: Architecture, state-of-the-art, challenges and future directions. Authorea Preprints. [find](https://scholar.google.com/scholar?q=Partha+Pratim+Ray%2E+2025%2E+A+survey+on+model+context+protocol%3A+Architecture%2C+state%2Dof%2Dthe%2Dart%2C+challenges+and+future+directions%2E+Authorea+Preprints%2E)
25. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). Preprint, arXiv:2402.03300.
26. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2024. [Hybridflow: A flexible and efficient rlhf framework](https://md.research.elamur.ai/free/en/papers/sheng-2024-hybridflow-flexible-efficient-rlhf-framework/). arXiv preprint arXiv: 2409.19256.
27. Aditi Singh, Abul Ehtesham, Saket Kumar, and Tala Talaei Khoei. 2025. A survey of the model context protocol (mcp): Standardizing context to enhance large language models (llms). [find](https://scholar.google.com/scholar?q=Aditi+Singh%2C+Abul+Ehtesham%2C+Saket+Kumar%2C+and+Tala+Talaei+Khoei%2E+2025%2E+A+survey+of+the+model+context+protocol+%28mcp%29%3A+Standardizing+context+to+enhance+large+language+models+%28llms%29%2E)
28. Aarohi Srivastava, Abhinav Rastogi, Rao, and 1 others. 2023. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). Transactions on machine learning research.
29. Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. 2024. [Appworld: A controllable world of apps and people for benchmarking interactive coding agents](https://md.research.elamur.ai/free/en/papers/trivedi-2024-appworld-controllable-world-apps-people/). arXiv preprint arXiv:2407.18901.
30. Ningning Wang, Xavier Hu, Pai Liu, He Zhu, Yue Hou, Heyuan Huang, Shengyu Zhang, Jian Yang, Jiaheng Liu, Ge Zhang, Changwang Zhang, Jun Wang, Yuchen Eleanor Jiang, and Wangchunshu Zhou. 2025a. Efficient agents: Building effective agents while reducing cost. Preprint, arXiv:2508.02694. — [arXiv:2508.02694](https://arxiv.org/abs/2508.02694) [find](https://scholar.google.com/scholar?q=Ningning+Wang%2C+Xavier+Hu%2C+Pai+Liu%2C+He+Zhu%2C+Yue+Hou%2C+Heyuan+Huang%2C+Shengyu+Zhang%2C+Jian+Yang%2C+Jiaheng+Liu%2C+Ge+Zhang%2C+Changwang+Zhang%2C+Jun+Wang%2C+Yuchen+Eleanor+Jiang%2C+and+Wangchunshu+Zhou%2E+2025a%2E+Efficie)
31. Zili Wang, Tianyu Zhang, Lei Zhu, Haoli Bai, Lu Hou, Shiming Xiang, Xianzhi Yu, and Wulong Liu. 2025b. Faster and better llms via latency-aware test-time scaling. arXiv preprint arXiv:2505.19634. [find](https://scholar.google.com/scholar?q=Zili+Wang%2C+Tianyu+Zhang%2C+Lei+Zhu%2C+Haoli+Bai%2C+Lu+Hou%2C+Shiming+Xiang%2C+Xianzhi+Yu%2C+and+Wulong+Liu%2E+2025b%2E+Faster+and+better+llms+via+latency%2Daware+test%2Dtime+scaling%2E+arXiv+preprint+arXiv%3A2505%2E19634%2E)
32. Shirley Wu, Shiyu Zhao, Qian Huang, Kexin Huang, Michihiro Yasunaga, Kaidi Cao, Vassilis N. Ioannidis, Karthik Subbian, Jure Leskovec, and James Zou. 2024. Avatar: Optimizing llm agents for tool usage via contrastive reasoning. In Advances in Neural Information Processing Systems, volume 37, pages 2… [find](https://scholar.google.com/scholar?q=Shirley+Wu%2C+Shiyu+Zhao%2C+Qian+Huang%2C+Kexin+Huang%2C+Michihiro+Yasunaga%2C+Kaidi+Cao%2C+Vassilis+N%2E+Ioannidis%2C+Karthik+Subbian%2C+Jure+Leskovec%2C+and+James+Zou%2E+2024%2E+Avatar%3A+Optimizing+llm+agents+for+tool+usage)
33. Yue Wu, Xuan Tang, Tom M Mitchell, and Yuanzhi Li. 2023. Smartplay: A benchmark for llms as intelligent agents. arXiv preprint arXiv:2310.01557. [find](https://scholar.google.com/scholar?q=Yue+Wu%2C+Xuan+Tang%2C+Tom+M+Mitchell%2C+and+Yuanzhi+Li%2E+2023%2E+Smartplay%3A+A+benchmark+for+llms+as+intelligent+agents%2E+arXiv+preprint+arXiv%3A2310%2E01557%2E)
34. Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, and 1 others. 2025. [The rise and potential of large language model based agents: A survey](https://md.research.elamur.ai/free/en/papers/xi-2023-rise-potential-large-language/). Science China Information Sciences, 68(2):121101.
35. Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. 2024. [Travelplanner: A benchmark for real-world planning with language agents](https://md.research.elamur.ai/free/en/papers/xie-2024-travelplanner-benchmark-real-world-planning/). arXiv preprint arXiv:2402.01622.
36. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
37. John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. [Swe-agent: Agent-computer interfaces enable automated software engineering](https://md.research.elamur.ai/free/en/papers/yang-2024-swe-agent-agent-computer-interfaces/). Preprint, arXiv:2405.15793.
38. Junjie Ye, Guanyu Li, Songyang Gao, Caishuang Huang, Yilong Wu, Sixian Li, Xiaoran Fan, Shihan Dou, Qi Zhang, Tao Gui, and 1 others. 2024. [Tooleyes: Fine-grained evaluation for tool learning capabilities of large language models in real-world scenarios](https://md.research.elamur.ai/free/en/papers/ye-2025-tooleyes-fine-grained-evaluation-tool/). arXiv preprint arXiv:2401.00741.
39. Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. 2024. [Assistantbench: Can web agents solve realistic and time-consuming tasks?](https://md.research.elamur.ai/free/en/papers/yoran-2024-assistantbench-can-web-agents-solve/) Preprint, arXiv:2407.15711.
40. Peijie Yu, Yifan Yang, Jinjian Li, Zelong Zhang, Haorui Wang, Xiao Feng, and Feng Zhang. 2025. c 3 -bench: The things real disturbing llm based agent in multi-tasking. arXiv preprint arXiv:2505.18746. — [arXiv:2505.18746](https://arxiv.org/abs/2505.18746) [find](https://scholar.google.com/scholar?q=Peijie+Yu%2C+Yifan+Yang%2C+Jinjian+Li%2C+Zelong+Zhang%2C+Haorui+Wang%2C+Xiao+Feng%2C+and+Feng+Zhang%2E+2025%2E+c+3+%2Dbench%3A+The+things+real+disturbing+llm+based+agent+in+multi%2Dtasking%2E+arXiv+preprint+arXiv%3A2505%2E18746%2E)
41. Aohan Zeng and 1 others. 2025. [Glm-4.5: Agentic, reasoning, and coding (arc) foundation models](https://md.research.elamur.ai/free/en/papers/team-2025-4-glm-4-5-agentic/). arXiv preprint arXiv:2508.06471.
42. Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. 2025. [Aflow: Automating agentic workflow generation](https://md.research.elamur.ai/free/en/papers/zhang-2024-aflow-automating-agentic-workflow/). Preprint, arXiv:2410.10762.
43. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Preprint, arXiv:2306.05685.
44. Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, and 1 others. 2024. [A survey on efficient inference for large language models](https://md.research.elamur.ai/free/en/papers/zhou-2024-survey-efficient-inference-large-language/). arXiv preprint arXiv:2404.14294.

*Updated 2 November 2026*