---
title: "OctoTools — paper by Lu et al. (2026) — summary and ideas"
description: "LLMs still struggle with hard problems that combine image understanding, specialist knowledge, calculation and… 10.6 points. 61 references."
authors: ["Pan Lu", "Bowen Chen", "Sheng Liu", "Rahul Thapa", "Joseph Boen", "James Zou"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1/"
doi: "10.18653/v1/2026.acl-long.1"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lu-2026-octotools-multi-agent-framework-extensible/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lu-2026-octotools-multi-agent-framework-extensible/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/4/) / OctoTools: A Multi-Agent Framework with Extensible Tools for Complex Reasoning

# OctoTools: A Multi-Agent Framework with Extensible Tools for Complex Reasoning

**Authors:** [Pan Lu](https://md.research.elamur.ai/free/en/authors/pan-lu/), Bowen Chen, [Sheng Liu](https://md.research.elamur.ai/free/en/authors/sheng-liu/), [Rahul Thapa](https://md.research.elamur.ai/free/en/authors/rahul-thapa/), [Joseph Boen](https://md.research.elamur.ai/free/en/authors/joseph-boen/) and [James Zou](https://md.research.elamur.ai/free/en/authors/james-zou/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1/) · [PDF](https://aclanthology.org/2026.acl-long.1.pdf) · [DOI 10.18653/v1/2026.acl-long.1](https://doi.org/10.18653/v1/2026.acl-long.1) · [OpenAlex](https://openalex.org/W7166862435)

**Citations:** 5 citations (OpenAlex, 2026-10-14)

## Summary
LLMs still struggle with hard problems that combine image understanding, specialist knowledge, calculation and several reasoning steps. Earlier tool-augmented systems tend to need fine-tuning data, cover only one domain or a narrow set of tools, or cannot refine a solution over multiple steps. OctoTools is an agentic framework that works without any weight updates. Each tool (e.g., a Python calculator, web search, an image captioner, an object detector) is wrapped in a standardized "tool card" that describes its inputs, outputs, demos and known limitations, so tools can be added or swapped without re-engineering. A planner first drafts a global plan and then picks the next action step by step. An executor turns each action into a runnable command and records the results in a shared context. A verifier then checks this context before a summarizer writes the answer. A greedy toolset-optimization procedure picks a useful subset of tools for each task. Across 16 benchmarks, OctoTools clearly beats plain GPT-4o and popular agent frameworks given the same tools. This suggests that a modular, extensible tool design can generalize across domains.

## Key points
- Evaluation spans 16 tasks, including MathVista, MMLU-Pro, MedQA and GAIA-Text. Average accuracy reaches about 58.5%, compared with 49.2% for GPT-4o, a gain of 9.3 points.
- Given the same toolsets, OctoTools surpasses AutoGen, GPT-Functions and LangChain with margins reaching as high as 10.6 points, including a 7.3-point lead over LangChain.
- The architecture has three parts:
  - **Tool cards:** metadata plus usage constraints for each tool.
  - **Planner:** Query Analyzer, Action Predictor, Context Verifier and Solution Summarizer.
  - **Executor:** Command Generator and Command Executor.
- Task-specific toolset selection uses a greedy search on validation data. It is efficient but not globally optimal, and it is fixed per task rather than per query.
- Reported limitations:
  - Dependence on the accuracy and availability of external tools.
  - Latency, with some queries taking over 90 seconds.
  - Security and privacy risks from executing generated code or calling third-party APIs.
  - Reproducibility drift as external services change.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 434 works · 314,419 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Pan Lu, Bowen Chen, Sheng Liu, Rahul Thapa, Joseph Boen and James Zou. 2026. OctoTools: A Multi-Agent Framework with Extensible Tools for Complex Reasoning. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{lu2026octotools,
  title = {{OctoTools: A Multi-Agent Framework with Extensible Tools for Complex Reasoning}},
  author = {Pan Lu, Bowen Chen, Sheng Liu, Rahul Thapa, Joseph Boen and James Zou},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1}
}
```

- [← previous NAKBA NLP 2026: Shared Task on Arabic Handwritten Manuscript Understanding (Palestine Memory–Omar Al-Saleh Memoir)](https://md.research.elamur.ai/free/en/papers/hamoud-2026-nakba-nlp-2026-shared-task/)
- [next → PaCoRe: Learning to Scale Test-Time Compute with Parallel Coordinated Reasoning](https://md.research.elamur.ai/free/en/papers/hu-2026-pacore-learning-scale-test-time/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/4/)

## References

61 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anthropic. 2024. Claude 3.5 haiku. [https://www.an](https://www.an) thropic.com/claude/haiku. Model announcement. [↗](https://www.an)
2. Michael Fore, Simranjit Singh, and Dimitrios Stamoulis. 2024. Geckopt: Llm system efficiency via intent-based tool selection. In Proceedings of the Great Lakes Symposium on VLSI 2024, pages 353–354. [find](https://scholar.google.com/scholar?q=Michael+Fore%2C+Simranjit+Singh%2C+and+Dimitrios+Stamoulis%2E+2024%2E+Geckopt%3A+Llm+system+efficiency+via+intent%2Dbased+tool+selection%2E+In+Proceedings+of+the+Great+Lakes+Symposium+on+VLSI+2024%2C+pages+353%E2%80%93354%2E)
3. AutoGen. 2024. Autogen. [https://github.com/mic](https://github.com/mic) rosoft/autogen. [↗](https://github.com/mic)
4. Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, and 1 others. 2024. [Omni-MATH: A universal olympiad level mathematic benchmark for large language models](https://md.research.elamur.ai/free/en/papers/gao-2024-omni-math-universal-olympiad-level/). arXiv preprint arXiv:2410.07985.
5. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. — [Qwen2.5-VL Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2025-qwen2-5-vl-technical/)
6. Zhenni Bi, Kai Han, Chuanjian Liu, Yehui Tang, and Yunhe Wang. 2024. [Forest-of-thought: Scaling test-time compute for enhancing llm reasoning](https://md.research.elamur.ai/free/en/papers/bi-2024-forest-thought-scaling-test-time/). arXiv preprint arXiv:2412.09078.
7. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764–10799. PMLR. [find](https://scholar.google.com/scholar?q=Luyu+Gao%2C+Aman+Madaan%2C+Shuyan+Zhou%2C+Uri+Alon%2C+Pengfei+Liu%2C+Yiming+Yang%2C+Jamie+Callan%2C+and+Graham+Neubig%2E+2023%2E+Pal%3A+Program%2Daided+language+models%2E+In+International+Conference+on+Machine+Learning%2C+page)
8. Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. 2023. Chemcrow: Augmenting large-language models with chemistry tools. arXiv preprint arXiv:2304.05376. [find](https://scholar.google.com/scholar?q=Andres+M+Bran%2C+Sam+Cox%2C+Oliver+Schilter%2C+Carlo+Baldassari%2C+Andrew+D+White%2C+and+Philippe+Schwaller%2E+2023%2E+Chemcrow%3A+Augmenting+large%2Dlanguage+models+with+chemistry+tools%2E+arXiv+preprint+arXiv%3A2304%2E0537)
9. Deepanway Ghosal, Vernon Toh Yan Han, Chia Yew Ken, and Soujanya Poria. 2024. Are language models puzzle prodigies? algorithmic puzzles unveil serious challenges in multimodal reasoning. arXiv preprint arXiv:2403.03864. — [arXiv:2403.03864](https://arxiv.org/abs/2403.03864) [find](https://scholar.google.com/scholar?q=Deepanway+Ghosal%2C+Vernon+Toh+Yan+Han%2C+Chia+Yew+Ken%2C+and+Soujanya+Poria%2E+2024%2E+Are+language+models+puzzle+prodigies%3F+algorithmic+puzzles+unveil+serious+challenges+in+multimodal+reasoning%2E+arXiv+preprin)
10. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in Neural Information Processing Systems (NeurIPS), 33:1877–1901.
11. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. [Making the v in vqa matter: Elevating the role of image understanding in visual question answering](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/). In Proceedings of the IEEE Conference on Computer Vision and pattern Recognition, pages 6904–6913.
12. Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. 2021. [Emerging properties in self-supervised vision transformers](https://md.research.elamur.ai/free/en/papers/caron-2021-emerging-properties-self-supervised-vision/). In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660.
13. Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, and 1 others. 2024. [HallusionBench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models](https://md.research.elamur.ai/free/en/papers/guan-2023-hallusionbench-advanced-diagnostic-suite-entangled/). In Proceedings of…
14. Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiec, and Nikolai Grigorev. 2022. [Internet-augmented language models through few-shot prompting for open-domain question answering](https://md.research.elamur.ai/free/en/papers/lazaridou-2022-internet-augmented-language-models-through/). arXiv preprint arXiv:2203.05115.
15. Binxu Li, Tiankai Yan, Yuanting Pan, Jie Luo, Ruiyang Ji, Jiayuan Ding, Zhe Xu, Shilong Liu, Haoyu Dong, Zihao Lin, and 1 others. 2024a. Mmedagent: Learning to use medical tools with multi-modal agent. arXiv preprint arXiv:2407.02483. [find](https://scholar.google.com/scholar?q=Binxu+Li%2C+Tiankai+Yan%2C+Yuanting+Pan%2C+Jie+Luo%2C+Ruiyang+Ji%2C+Jiayuan+Ding%2C+Zhe+Xu%2C+Shilong+Liu%2C+Haoyu+Dong%2C+Zihao+Lin%2C+and+1+others%2E+2024a%2E+Mmedagent%3A+Learning+to+use+medical+tools+with+multi%2Dmodal+agent)
16. Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. 2025. [rstar-math: Small llms can master math reasoning with self-evolved deep thinking](https://md.research.elamur.ai/free/en/papers/guan-2025-rstar-math-small-llms-can/). arXiv preprint arXiv:2501.04519.
17. Zelong Li, Wenyue Hua, Hao Wang, He Zhu, and Yongfeng Zhang. 2024b. Formal-LLM: Integrating formal language and natural language for controllable llm-based agents. arXiv preprint arXiv:2402.00798. — [arXiv:2402.00798](https://arxiv.org/abs/2402.00798) [find](https://scholar.google.com/scholar?q=Zelong+Li%2C+Wenyue+Hua%2C+Hao+Wang%2C+He+Zhu%2C+and+Yongfeng+Zhang%2E+2024b%2E+Formal%2DLLM%3A+Integrating+formal+language+and+natural+language+for+controllable+llm%2Dbased+agents%2E+arXiv+preprint+arXiv%3A2402%2E00798%2E+%E2%80%94+%5B)
18. Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. 2020. PathVQA: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286. — [arXiv:2003.10286](https://arxiv.org/abs/2003.10286) [find](https://scholar.google.com/scholar?q=Xuehai+He%2C+Yichen+Zhang%2C+Luntian+Mou%2C+Eric+Xing%2C+and+Pengtao+Xie%2E+2020%2E+PathVQA%3A+30000%2B+questions+for+medical+visual+question+answering%2E+arXiv+preprint+arXiv%3A2003%2E10286%2E+%E2%80%94+%5BarXiv%3A2003%2E10286%5D%28https%3A%2F%2Fa)
19. Junwei Liao, Muning Wen, Jun Wang, and Weinan Zhang. 2025. Marft: Multi-agent reinforcement fine-tuning. arXiv preprint arXiv:2504.16129. — [arXiv:2504.16129](https://arxiv.org/abs/2504.16129) [find](https://scholar.google.com/scholar?q=Junwei+Liao%2C+Muning+Wen%2C+Jun+Wang%2C+and+Weinan+Zhang%2E+2025%2E+Marft%3A+Multi%2Dagent+reinforcement+fine%2Dtuning%2E+arXiv+preprint+arXiv%3A2504%2E16129%2E+%E2%80%94+%5BarXiv%3A2504%2E16129%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2504%2E16129%29)
20. Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, and 1 others. 2024. [MetaGPT: Meta programming for a multi-agent collaborative framework](https://md.research.elamur.ai/free/en/papers/hong-2023-metagpt-meta-programming-multi/). In International Conference on Learning Representations (ICLR).
21. Nathan Lile. 2024. Math twenty four (24s game) dataset. [https://huggingface.co/datasets/nlile/24](https://huggingface.co/datasets/nlile/24) -game. [↗](https://huggingface.co/datasets/nlile/24)
22. Adam Dahlgren Lindström and Savitha Sam Abraham. 2022. CLEVR-Math: A dataset for compositional language, visual and mathematical reasoning. arXiv preprint arXiv:2208.05358. — [arXiv:2208.05358](https://arxiv.org/abs/2208.05358) [find](https://scholar.google.com/scholar?q=Adam+Dahlgren+Lindstr%C3%B6m+and+Savitha+Sam+Abraham%2E+2022%2E+CLEVR%2DMath%3A+A+dataset+for+compositional+language%2C+visual+and+mathematical+reasoning%2E+arXiv+preprint+arXiv%3A2208%2E05358%2E+%E2%80%94+%5BarXiv%3A2208%2E05358%5D%28https%3A)
23. Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. 2024. Visual sketchpad: Sketching as a visual chain of thought for multimodal language models. arXiv preprint arXiv:2406.09403. [find](https://scholar.google.com/scholar?q=Yushi+Hu%2C+Weijia+Shi%2C+Xingyu+Fu%2C+Dan+Roth%2C+Mari+Ostendorf%2C+Luke+Zettlemoyer%2C+Noah+A+Smith%2C+and+Ranjay+Krishna%2E+2024%2E+Visual+sketchpad%3A+Sketching+as+a+visual+chain+of+thought+for+multimodal+language+mo)
24. Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. 2021. SLAKE: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1650–1654. IEEE. [find](https://scholar.google.com/scholar?q=Bo+Liu%2C+Li%2DMing+Zhan%2C+Li+Xu%2C+Lin+Ma%2C+Yan+Yang%2C+and+Xiao%2DMing+Wu%2E+2021%2E+SLAKE%3A+A+semantically%2Dlabeled+knowledge%2Denhanced+dataset+for+medical+visual+question+answering%2E+In+2021+IEEE+18th+International+S)
25. Zhenlan Ji, Daoyuan Wu, Pingchuan Ma, Zongjie Li, and Shuai Wang. 2024. Testing and understanding erroneous planning in llm agents through synthesized user inputs. arXiv preprint arXiv:2404.17833. — [arXiv:2404.17833](https://arxiv.org/abs/2404.17833) [find](https://scholar.google.com/scholar?q=Zhenlan+Ji%2C+Daoyuan+Wu%2C+Pingchuan+Ma%2C+Zongjie+Li%2C+and+Shuai+Wang%2E+2024%2E+Testing+and+understanding+erroneous+planning+in+llm+agents+through+synthesized+user+inputs%2E+arXiv+preprint+arXiv%3A2404%2E17833%2E+%E2%80%94+%5B)
26. Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, and 1 others. 2023. Llava-plus: Learning to use tools for creating multimodal agents. arXiv preprint arXiv:2311.05437. — [arXiv:2311.05437](https://arxiv.org/abs/2311.05437) [find](https://scholar.google.com/scholar?q=Shilong+Liu%2C+Hao+Cheng%2C+Haotian+Liu%2C+Hao+Zhang%2C+Feng+Li%2C+Tianhe+Ren%2C+Xueyan+Zou%2C+Jianwei+Yang%2C+Hang+Su%2C+Jun+Zhu%2C+and+1+others%2E+2023%2E+Llava%2Dplus%3A+Learning+to+use+tools+for+creating+multimodal+agents%2E+a)
27. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. [What disease does this patient have? a large-scale open domain question answering dataset from medical exams](https://md.research.elamur.ai/free/en/papers/jin-2020-disease-this-patient-have-large/). Applied Sciences, 11(14):6421.
28. Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Long Phi Le, Georg Gerber, and 1 others. 2024a. A visual-language foundation model for computational pathology. Nature Medicine, 30:863–874. [find](https://scholar.google.com/scholar?q=Ming+Y+Lu%2C+Bowen+Chen%2C+Drew+FK+Williamson%2C+Richard+J+Chen%2C+Ivy+Liang%2C+Tong+Ding%2C+Guillaume+Jaume%2C+Igor+Odintsov%2C+Long+Phi+Le%2C+Georg+Gerber%2C+and+1+others%2E+2024a%2E+A+visual%2Dlanguage+foundation+model+for+)
29. Yeonghun Kang and Jihan Kim. 2024. Chatmof: an artificial intelligence system for predicting and generating metal-organic frameworks using large language models. Nature Communications, 15(1):4705. [find](https://scholar.google.com/scholar?q=Yeonghun+Kang+and+Jihan+Kim%2E+2024%2E+Chatmof%3A+an+artificial+intelligence+system+for+predicting+and+generating+metal%2Dorganic+frameworks+using+large+language+models%2E+Nature+Communications%2C+15%281%29%3A4705%2E)
30. Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2022. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406. [find](https://scholar.google.com/scholar?q=Tushar+Khot%2C+Harsh+Trivedi%2C+Matthew+Finlayson%2C+Yao+Fu%2C+Kyle+Richardson%2C+Peter+Clark%2C+and+Ashish+Sabharwal%2E+2022%2E+Decomposed+prompting%3A+A+modular+approach+for+solving+complex+tasks%2E+arXiv+preprint+arXi)
31. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024b. [MathVista: Evaluating mathematical reasoning of foundation models in visual contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/). In International Conference on Learning Representations (ICLR).
32. Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2022. [Internet-augmented dialogue generation](https://md.research.elamur.ai/free/en/papers/komeili-2022-internet-augmented-dialogue-generation/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8460–8478, Dublin, Ireland. Association for Computational Linguistics.
33. Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. [Chameleon: Plug-and-play compositional reasoning with large language models](https://md.research.elamur.ai/free/en/papers/lu-2023-chameleon-plug-play-compositional-reasoning/). Advances in Neural Information Processing Systems, 35.
34. Inc LangChain. 2024. LangChain. [https://github.com/langchain-ai/langchain](https://github.com/langchain-ai/langchain). [↗](https://github.com/langchain-ai/langchain)
35. Elias Lumer. 2024. Toolshed: Scale tool-equipped agents with advanced rag-tool fusion and tool knowledge bases. arXiv preprint arXiv:2410.14594. — [arXiv:2410.14594](https://arxiv.org/abs/2410.14594) [find](https://scholar.google.com/scholar?q=Elias+Lumer%2E+2024%2E+Toolshed%3A+Scale+tool%2Dequipped+agents+with+advanced+rag%2Dtool+fusion+and+tool+knowledge+bases%2E+arXiv+preprint+arXiv%3A2410%2E14594%2E+%E2%80%94+%5BarXiv%3A2410%2E14594%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2410%2E14594%29)
36. Samuel Schmidgall, Rojin Ziaei, Carl Harris, Eduardo Reis, Jeffrey Jopling, and Michael Moor. 2024. Agentclinic: a multimodal agent benchmark to evaluate ai in simulated clinical environments. arXiv preprint arXiv:2405.07960. — [arXiv:2405.07960](https://arxiv.org/abs/2405.07960) [find](https://scholar.google.com/scholar?q=Samuel+Schmidgall%2C+Rojin+Ziaei%2C+Carl+Harris%2C+Eduardo+Reis%2C+Jeffrey+Jopling%2C+and+Michael+Moor%2E+2024%2E+Agentclinic%3A+a+multimodal+agent+benchmark+to+evaluate+ai+in+simulated+clinical+environments%2E+arXiv+p)
37. Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2023. [GAIA: a benchmark for general ai assistants](https://md.research.elamur.ai/free/en/papers/mialon-2023-gaia-benchmark-general-ai-assistants/). arXiv preprint arXiv:2311.12983.
38. Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen, Kushal Arora, Joshua Lane, and 1 others. 2022. [Blenderbot 3: a deployed conversational agent that continually learns to responsibly engage](https://md.research.elamur.ai/free/en/papers/shuster-2022-blenderbot-3-deployed-conversational-agent/). arXiv preprint arXiv:2208.03188.
39. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, and 1 others. 2021. [Webgpt: Browser-assisted question-answering with human feedback](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/). arXiv preprint arXiv:2112.09332.
40. Simeng Sun, Yang Liu, Shuohang Wang, Chenguang Zhu, and Mohit Iyyer. 2023. Pearl: Prompting large language models to plan and execute actions over long documents. arXiv preprint arXiv:2305.14564. [find](https://scholar.google.com/scholar?q=Simeng+Sun%2C+Yang+Liu%2C+Shuohang+Wang%2C+Chenguang+Zhu%2C+and+Mohit+Iyyer%2E+2023%2E+Pearl%3A+Prompting+large+language+models+to+plan+and+execute+actions+over+long+documents%2E+arXiv+preprint+arXiv%3A2305%2E14564%2E)
41. OpenAI. 2023a. Function calling - openai. [https://platform.openai.com/docs/guides/functi](https://platform.openai.com/docs/guides/functi) on-calling. [↗](https://platform.openai.com/docs/guides/functi)
42. Yuxuan Sun, Hao Wu, Chenglu Zhu, Sunyi Zheng, Qizi Chen, Kai Zhang, Yunlong Zhang, Dan Wan, Xiaoxiao Lan, Mengyue Zheng, and 1 others. 2025. Path-MMU: A massive multimodal expert-level benchmark for understanding and reasoning in pathology. In European Conference on Computer Vision, pages 56–73. Spr… [find](https://scholar.google.com/scholar?q=Yuxuan+Sun%2C+Hao+Wu%2C+Chenglu+Zhu%2C+Sunyi+Zheng%2C+Qizi+Chen%2C+Kai+Zhang%2C+Yunlong+Zhang%2C+Dan+Wan%2C+Xiaoxiao+Lan%2C+Mengyue+Zheng%2C+and+1+others%2E+2025%2E+Path%2DMMU%3A+A+massive+multimodal+expert%2Dlevel+benchmark+for+u)
43. OpenAI. 2023b. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
44. Varatheepan Paramanayakam, Andreas Karatzas, Iraklis Anagnostopoulos, and Dimitrios Stamoulis. 2024. Less is more: Optimizing function calling for llm execution on edge devices. arXiv preprint arXiv:2411.15399. — [arXiv:2411.15399](https://arxiv.org/abs/2411.15399) [find](https://scholar.google.com/scholar?q=Varatheepan+Paramanayakam%2C+Andreas+Karatzas%2C+Iraklis+Anagnostopoulos%2C+and+Dimitrios+Stamoulis%2E+2024%2E+Less+is+more%3A+Optimizing+function+calling+for+llm+execution+on+edge+devices%2E+arXiv+preprint+arXiv%3A2)
45. Heyi Tao, Sethuraman TV, Michal Shlapentokh-Rothman, Derek Hoiem, and Heng Ji. 2023. Web-wise: Web interface control and sequential exploration with large language models. arXiv preprint arXiv:2310.16042. — [arXiv:2310.16042](https://arxiv.org/abs/2310.16042) [find](https://scholar.google.com/scholar?q=Heyi+Tao%2C+Sethuraman+TV%2C+Michal+Shlapentokh%2DRothman%2C+Derek+Hoiem%2C+and+Heng+Ji%2E+2023%2E+Web%2Dwise%3A+Web+interface+control+and+sequential+exploration+with+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E16)
46. Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. 2020. [Unsupervised question decomposition for question answering](https://md.research.elamur.ai/free/en/papers/perez-2020-unsupervised-question-decomposition-question-answering/). arXiv preprint arXiv:2002.09758.
47. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, and 41 others. 2022… — [LaMDA: Language Models for Dialog Applications](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/)
48. Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. 2025. ToolRL: Reward is all tool learning needs. arXiv preprint arXiv:2504.13958. [find](https://scholar.google.com/scholar?q=Cheng+Qian%2C+Emre+Can+Acikgoz%2C+Qi+He%2C+Hongru+Wang%2C+Xiusi+Chen%2C+Dilek+Hakkani%2DT%C3%BCr%2C+Gokhan+Tur%2C+and+Heng+Ji%2E+2025%2E+ToolRL%3A+Reward+is+all+tool+learning+needs%2E+arXiv+preprint+arXiv%3A2504%2E13958%2E)
49. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2023. [GPQA: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). arXiv preprint arXiv:2311.12022.
50. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, and 1 others. 2024. [MMLU-Pro: A more robust and challenging multi-task language understanding benchmark](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/). arXiv preprint arXiv:2406.01574.
51. Tianhe Ren, Yihao Chen, Qing Jiang, Zhaoyang Zeng, Yuda Xiong, Wenlong Liu, Zhengyu Ma, Junyi Shen, Yuan Gao, Xiaoke Jiang, Xingyu Chen, Zhuheng Song, Yuhong Zhang, Hongjie Huang, Han Gao, Shilong Liu, Hao Zhang, Feng Li, Kent Yu, and Lei Zhang. 2024. Dino-x: A unified vision model for open-world ob… — [arXiv:2411.14347](https://arxiv.org/abs/2411.14347) [find](https://scholar.google.com/scholar?q=Tianhe+Ren%2C+Yihao+Chen%2C+Qing+Jiang%2C+Zhaoyang+Zeng%2C+Yuda+Xiong%2C+Wenlong+Liu%2C+Zhengyu+Ma%2C+Junyi+Shen%2C+Yuan+Gao%2C+Xiaoke+Jiang%2C+Xingyu+Chen%2C+Zhuheng+Song%2C+Yuhong+Zhang%2C+Hongjie+Huang%2C+Han+Gao%2C+Shilong+Liu)
52. Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, and 1 others. 2025a. RAGEN: Understanding self-evolution in llm agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073. [find](https://scholar.google.com/scholar?q=Zihan+Wang%2C+Kangrui+Wang%2C+Qineng+Wang%2C+Pingyue+Zhang%2C+Linjie+Li%2C+Zhengyuan+Yang%2C+Xing+Jin%2C+Kefan+Yu%2C+Minh+Nhat+Nguyen%2C+Licheng+Liu%2C+and+1+others%2E+2025a%2E+RAGEN%3A+Understanding+self%2Devolution+in+llm+agen)
53. Jonathan Roberts, Kai Han, Neil Houlsby, and Samuel Albanie. 2024. SciFIBench: Benchmarking large multimodal models for scientific figure interpretation. arXiv preprint arXiv:2405.08807. — [arXiv:2405.08807](https://arxiv.org/abs/2405.08807) [find](https://scholar.google.com/scholar?q=Jonathan+Roberts%2C+Kai+Han%2C+Neil+Houlsby%2C+and+Samuel+Albanie%2E+2024%2E+SciFIBench%3A+Benchmarking+large+multimodal+models+for+scientific+figure+interpretation%2E+arXiv+preprint+arXiv%3A2405%2E08807%2E+%E2%80%94+%5BarXiv%3A2405)
54. Ziliang Wang, Xuhui Zheng, Kang An, Cijun Ouyang, Jialu Cai, Yuhang Wang, and Yichao Wu. 2025b. Stepsearch: Igniting llms search ability via stepwise proximal policy optimization. arXiv preprint arXiv:2505.15107. — [arXiv:2505.15107](https://arxiv.org/abs/2505.15107) [find](https://scholar.google.com/scholar?q=Ziliang+Wang%2C+Xuhui+Zheng%2C+Kang+An%2C+Cijun+Ouyang%2C+Jialu+Cai%2C+Yuhang+Wang%2C+and+Yichao+Wu%2E+2025b%2E+Stepsearch%3A+Igniting+llms+search+ability+via+stepwise+proximal+policy+optimization%2E+arXiv+preprint+arXiv)
55. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. [Toolformer: Language models can teach themselves to use tools](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/). Advances in Neural Information Processing Systems, 36:68539–68551.
56. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824– 24837. xAI. 2024. Grok-2 vision. [https://x.ai/news/g…](https://x.ai/news/g…) [↗](https://x.ai/news/g…)
57. An Yang and 1 others. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. [find](https://scholar.google.com/scholar?q=An+Yang+and+1+others%2E+2024%2E+Qwen2%2E5+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E15115%2E)
58. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. [React: Synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/). arXiv preprint arXiv:2210.03629.
59. Siliang Zeng, Quan Wei, William Brown, Oana Frunza, Yuriy Nevmyvaka, and Mingyi Hong. 2025. Reinforcing multi-turn reasoning in llm agents via turn-level credit assignment. arXiv preprint arXiv:2505.11821. — [arXiv:2505.11821](https://arxiv.org/abs/2505.11821) [find](https://scholar.google.com/scholar?q=Siliang+Zeng%2C+Quan+Wei%2C+William+Brown%2C+Oana+Frunza%2C+Yuriy+Nevmyvaka%2C+and+Mingyi+Hong%2E+2025%2E+Reinforcing+multi%2Dturn+reasoning+in+llm+agents+via+turn%2Dlevel+credit+assignment%2E+arXiv+preprint+arXiv%3A2505%2E1)
60. Shaokun Zhang, Yi Dong, Jieyu Zhang, Jan Kautz, Bryan Catanzaro, Andrew Tao, Qingyun Wu, Zhiding Yu, and Guilin Liu. 2025. Nemotron-research-tool-n1: Tool-using language models with reinforced reasoning. arXiv preprint arXiv:2505.00024. — [arXiv:2505.00024](https://arxiv.org/abs/2505.00024) [find](https://scholar.google.com/scholar?q=Shaokun+Zhang%2C+Yi+Dong%2C+Jieyu+Zhang%2C+Jan+Kautz%2C+Bryan+Catanzaro%2C+Andrew+Tao%2C+Qingyun+Wu%2C+Zhiding+Yu%2C+and+Guilin+Liu%2E+2025%2E+Nemotron%2Dresearch%2Dtool%2Dn1%3A+Tool%2Dusing+language+models+with+reinforced+reasoni)
61. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and 1 others. 2022. [Least-to-most prompting enables complex reasoning in large language models](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/). arXiv preprint arXiv:2205.10625.

*Updated 31 October 2026*