---
title: "ToolLLM: Facilitating Large Language Models to — Qin 2023"
description: "Open-source LLMs like LLaMA are much weaker than closed systems such as ChatGPT at fulfilling user requests through external API calls, mostly because…"
authors: ["Yujia Qin", "Shihao Liang", "Yining Ye", "Kunlun Zhu", "Lan Yan", "Yaxi Lu", "Yankai Lin", "Xin Cong", "Xiangru Tang", "Bill Qian", "Sihan Zhao", "Lauren Hong", "Runchu Tian", "Ruobing Xie", "Jie Zhou", "Mark Gerstein", "Dahai Li", "Zhiyuan Liu", "Maosong Sun"]
year: 2023
venue: "arXiv:2307.16789"
source: "https://arxiv.org/abs/2307.16789"
doi: "10.48550/arxiv.2307.16789"
arxiv: "2307.16789"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/qin-2023-toolllm-facilitating-large-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/qin-2023-toolllm-facilitating-large-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/) / ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs

# ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs

**Authors:** [Yujia Qin](https://md.research.elamur.ai/free/en/authors/yujia-qin/), [Shihao Liang](https://md.research.elamur.ai/free/en/authors/shihao-liang/), [Yining Ye](https://md.research.elamur.ai/free/en/authors/yining-ye/), Kunlun Zhu, Lan Yan, [Yaxi Lu](https://md.research.elamur.ai/free/en/authors/yaxi-lu/), [Yankai Lin](https://md.research.elamur.ai/free/en/authors/yankai-lin/), [Xin Cong](https://md.research.elamur.ai/free/en/authors/xin-cong/), [Xiangru Tang](https://md.research.elamur.ai/free/en/authors/xiangru-tang/), Bill Qian, Sihan Zhao, Lauren Hong et al. (19 authors) · **Institutions:** [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) · [Renmin University of China](https://md.research.elamur.ai/free/en/universities/renmin-university-of-china/) · [Yale University](https://md.research.elamur.ai/free/en/universities/yale-university/) · [Tencent](https://md.research.elamur.ai/free/en/companies/tencent/) · **Year:** 2023 · **Venue:** arXiv:2307.16789 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2307.16789) · [PDF](https://arxiv.org/pdf/2307.16789) · [DOI 10.48550/arxiv.2307.16789](https://doi.org/10.48550/arxiv.2307.16789) · [arXiv 2307.16789](https://arxiv.org/abs/2307.16789) · [OpenAlex](https://openalex.org/W4385474529)

**Citations:** 71 citations (OpenAlex, 2026-10-14) · [493](https://md.research.elamur.ai/free/en/papers/qin-2023-toolllm-facilitating-large-language-models/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
Open-source LLMs like LLaMA are much weaker than closed systems such as ChatGPT at fulfilling user requests through external API calls, mostly because current instruction-tuning data barely covers tool use. The authors propose ToolLLM, a framework spanning data creation, training and evaluation. Its central resource, ToolBench, is produced almost entirely by ChatGPT. Real REST APIs and their documentation are harvested from RapidAPI. ChatGPT then writes requests needing one or several tools and searches for working sequences of calls that resolve each one. Hard multi-tool cases are made tractable by DFSDT, a depth-first decision-tree procedure that explores alternative branches and backtracks, unlike a single ReACT chain. Assessment is automated by ToolEval, a ChatGPT-based judge whose verdicts correlate well with human ratings. Fine-tuning LLaMA gives ToolLLaMA, which works with a neural API retriever. It outperforms Text-Davinci-003 and Claude-2, comes close to ChatGPT, and handles unseen APIs and the external APIBench benchmark. The result is an open, reproducible path to capable tool-using models.

## Key points
- ToolBench covers 16,464 RESTful APIs from 3,451 tools across 49 RapidAPI Hub categories. It includes 126,486 instances and 469,585 real API calls, all annotated with gpt-3.5-turbo-16k.
- Instructions fall into three types: single-tool, intra-category multi-tool, and intra-collection multi-tool. Their solution paths average about 4 reasoning traces.
- DFSDT lets the model compare multiple reasoning paths and abandon dead ends. It solves instructions that ReACT fails on and makes annotation more efficient.
- ToolEval scores models with two metrics: pass rate (task completed within a budget) and win rate (pairwise comparison against ChatGPT-ReACT).
- ToolLLaMA performs slightly below GPT-4 and roughly matches Gorilla on APIBench zero-shot. Code, models and a demo are released at github.com/OpenBMB/ToolBench.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{qin2023toolllm,
  title = {{ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs}},
  author = {Qin, Yujia and Liang, Shihao and Ye, Yining and Zhu, Kunlun and Yan, Lan and Lu, Yaxi and Lin, Yankai and Cong, Xin and Tang, Xiangru and Qian, Bill and Zhao, Sihan and Hong, Lauren and Tian, Runchu and Xie, Ruobing and Zhou, Jie and Gerstein, Mark and Li, Dahai and Liu, Zhiyuan and Sun, Maosong},
  year = {2023},
  doi = {10.48550/arxiv.2307.16789},
  eprint = {2307.16789},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.16789}
}
```

- [← previous Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://md.research.elamur.ai/free/en/papers/du-2023-improving-factuality-reasoning-language-models/)
- [next → Large Language Models Can Self-Improve](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/)

## References

47 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances. ArXiv preprint, abs/2204.01691, 2022. [find](https://scholar.google.com/scholar?q=Michael+Ahn%2C+Anthony+Brohan%2C+Noah+Brown%2C+Yevgen+Chebotar%2C+Omar+Cortes%2C+Byron+David%2C+Chelsea+Finn%2C+Keerthana+Gopalakrishnan%2C+Karol+Hausman%2C+Alex+Herzog%2C+et+al%2E+Do+as+i+can%2C+not+as+i+say%3A+Grounding+lang) [our card](https://md.research.elamur.ai/free/en/papers/ahn-2022-i-i-say-grounding-language/)
2. Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Févry, et al. Promptsource: An integrated development environment and repository for natural language prompts. In Proceedings of the 60th Annual Meeting of th… [find](https://scholar.google.com/scholar?q=Stephen+Bach%2C+Victor+Sanh%2C+Zheng+Xin+Yong%2C+Albert+Webson%2C+Colin+Raffel%2C+Nihal+V+Nayak%2C+Abheesht+Sharma%2C+Taewoon+Kim%2C+M+Saiful+Bari%2C+Thibault+Fe%CC%81vry%2C+et+al%2E+Promptsource%3A+An+integrated+development+envi) [our card](https://md.research.elamur.ai/free/en/papers/bach-2022-promptsource-integrated-development-environment-repository/)
3. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. [find](https://scholar.google.com/scholar?q=Se%CC%81bastien+Bubeck%2C+Varun+Chandrasekaran%2C+Ronen+Eldan%2C+Johannes+Gehrke%2C+Eric+Horvitz%2C+Ece+Kamar%2C+Peter+Lee%2C+Yin+Tat+Lee%2C+Yuanzhi+Li%2C+Scott+Lundberg%2C+et+al%2E+Sparks+of+artificial+general+intelligence%3A+Ea) [our card](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/)
4. Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595, 2023. [find](https://scholar.google.com/scholar?q=Shouyuan+Chen%2C+Sherman+Wong%2C+Liangjian+Chen%2C+and+Yuandong+Tian%2E+Extending+context+window+of+large+language+models+via+positional+interpolation%2E+arXiv+preprint+arXiv%3A2306%2E15595%2C+2023%2E)
5. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL [https://lmsys.org/blog/2023-03-30-vicuna…](https://lmsys.org/blog/2023-03-30-vicuna…) [↗](https://lmsys.org/blog/2023-03-30-vicuna…)
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+North+A) [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
7. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233, 2023. [find](https://scholar.google.com/scholar?q=Ning+Ding%2C+Yulin+Chen%2C+Bokai+Xu%2C+Yujia+Qin%2C+Zhi+Zheng%2C+Shengding+Hu%2C+Zhiyuan+Liu%2C+Maosong+Sun%2C+and+Bowen+Zhou%2E+Enhancing+chat+language+models+by+scaling+high%2Dquality+instructional+conversations%2E+arXiv) [our card](https://md.research.elamur.ai/free/en/papers/ding-2023-enhancing-chat-language-models-scaling/)
8. Difei Gao, Lei Ji, Luowei Zhou, Kevin Qinghong Lin, Joya Chen, Zihan Fan, and Mike Zheng Shou. Assistgpt: A general multi-modal assistant that can plan, execute, inspect, and learn. arXiv preprint arXiv:2306.08640, 2023. — [arXiv:2306.08640](https://arxiv.org/abs/2306.08640) [find](https://scholar.google.com/scholar?q=Difei+Gao%2C+Lei+Ji%2C+Luowei+Zhou%2C+Kevin+Qinghong+Lin%2C+Joya+Chen%2C+Zihan+Fan%2C+and+Mike+Zheng+Shou%2E+Assistgpt%3A+A+general+multi%2Dmodal+assistant+that+can+plan%2C+execute%2C+inspect%2C+and+learn%2E+arXiv+preprint+arX)
9. Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14953–14962, 2023. [find](https://scholar.google.com/scholar?q=Tanmay+Gupta+and+Aniruddha+Kembhavi%2E+Visual+programming%3A+Compositional+visual+reasoning+without+training%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+pp%2E+1495)
10. Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings. arXiv preprint arXiv:2305.11554, 2023. [find](https://scholar.google.com/scholar?q=Shibo+Hao%2C+Tianyang+Liu%2C+Zhen+Wang%2C+and+Zhiting+Hu%2E+Toolkengpt%3A+Augmenting+frozen+language+models+with+massive+tools+via+tool+embeddings%2E+arXiv+preprint+arXiv%3A2305%2E11554%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/hao-2023-toolkengpt-augmenting-frozen-language-models/)
11. Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Lear… [find](https://scholar.google.com/scholar?q=Wenlong+Huang%2C+Pieter+Abbeel%2C+Deepak+Pathak%2C+and+Igor+Mordatch%2E+Language+models+as+zero%2Dshot+planners%3A+Extracting+actionable+knowledge+for+embodied+agents%2E+In+Kamalika+Chaudhuri%2C+Stefanie+Jegelka%2C+Le+) [our card](https://md.research.elamur.ai/free/en/papers/huang-2022-language-models-zero-shot-planners/)
12. Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. ArXiv preprint, abs/2207.05608, 2022b. [find](https://scholar.google.com/scholar?q=Wenlong+Huang%2C+Fei+Xia%2C+Ted+Xiao%2C+Harris+Chan%2C+Jacky+Liang%2C+Pete+Florence%2C+Andy+Zeng%2C+Jonathan+Tompson%2C+Igor+Mordatch%2C+Yevgen+Chebotar%2C+et+al%2E+Inner+monologue%3A+Embodied+reasoning+through+planning+with)
13. Kalervo Järvelin and Jaana Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 20(4):422–446, 2002. [find](https://scholar.google.com/scholar?q=Kalervo+Ja%CC%88rvelin+and+Jaana+Keka%CC%88la%CC%88inen%2E+Cumulated+gain%2Dbased+evaluation+of+ir+techniques%2E+ACM+Transactions+on+Information+Systems+%28TOIS%29%2C+20%284%29%3A422%E2%80%93446%2C+2002%2E)
14. Qiao Jin, Yifan Yang, Qingyu Chen, and Zhiyong Lu. Genegpt: Augmenting large language models with domain tools for improved access to biomedical information. ArXiv, 2023. [find](https://scholar.google.com/scholar?q=Qiao+Jin%2C+Yifan+Yang%2C+Qingyu+Chen%2C+and+Zhiyong+Lu%2E+Genegpt%3A+Augmenting+large+language+models+with+domain+tools+for+improved+access+to+biomedical+information%2E+ArXiv%2C+2023%2E)
15. Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A benchmark for tool-augmented llms. arXiv preprint arXiv:2304.08244, 2023a. — [arXiv:2304.08244](https://arxiv.org/abs/2304.08244) [find](https://scholar.google.com/scholar?q=Minghao+Li%2C+Feifan+Song%2C+Bowen+Yu%2C+Haiyang+Yu%2C+Zhoujun+Li%2C+Fei+Huang%2C+and+Yongbin+Li%2E+Api%2Dbank%3A+A+benchmark+for+tool%2Daugmented+llms%2E+arXiv+preprint+arXiv%3A2304%2E08244%2C+2023a%2E+%E2%80%94+%5BarXiv%3A2304%2E08244%5D%28https%3A)
16. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval), 2023b. [↗](https://github.com/tatsu-lab/alpaca_eval)
17. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3470–3487, 2022. [find](https://scholar.google.com/scholar?q=Swaroop+Mishra%2C+Daniel+Khashabi%2C+Chitta+Baral%2C+and+Hannaneh+Hajishirzi%2E+Cross%2Dtask+generalization+via+natural+language+crowdsourcing+instructions%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Asso)
18. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. ArXiv preprint, abs/2112.09332, 2021. [find](https://scholar.google.com/scholar?q=Reiichiro+Nakano%2C+Jacob+Hilton%2C+Suchir+Balaji%2C+Jeff+Wu%2C+Long+Ouyang%2C+Christina+Kim%2C+Christopher+Hesse%2C+Shantanu+Jain%2C+Vineet+Kosaraju%2C+William+Saunders%2C+et+al%2E+Webgpt%3A+Browser%2Dassisted+question%2Danswer) [our card](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/)
19. OpenAI. OpenAI: Introducing ChatGPT, 2022. URL [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt). [↗](https://openai.com/blog/chatgpt)
20. OpenAI. Gpt-4 technical report, 2023. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Gpt%2D4+technical+report%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
21. Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334, 2023. [find](https://scholar.google.com/scholar?q=Shishir+G+Patil%2C+Tianjun+Zhang%2C+Xin+Wang%2C+and+Joseph+E+Gonzalez%2E+Gorilla%3A+Large+language+model+connected+with+massive+apis%2E+arXiv+preprint+arXiv%3A2305%2E15334%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/patil-2023-gorilla-large-language-model-connected/)
22. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116… [find](https://scholar.google.com/scholar?q=Guilherme+Penedo%2C+Quentin+Malartic%2C+Daniel+Hesslow%2C+Ruxandra+Cojocaru%2C+Alessandro+Cappelli%2C+Hamza+Alobeidli%2C+Baptiste+Pannier%2C+Ebtesam+Almazrouei%2C+and+Julien+Launay%2E+The+refinedweb+dataset+for+falcon+)
23. Cheng Qian, Chi Han, Yi R Fung, Yujia Qin, Zhiyuan Liu, and Heng Ji. Creator: Disentangling abstract and concrete reasonings of large language models through tool creation. arXiv preprint arXiv:2305.14318, 2023. — [arXiv:2305.14318](https://arxiv.org/abs/2305.14318) [find](https://scholar.google.com/scholar?q=Cheng+Qian%2C+Chi+Han%2C+Yi+R+Fung%2C+Yujia+Qin%2C+Zhiyuan+Liu%2C+and+Heng+Ji%2E+Creator%3A+Disentangling+abstract+and+concrete+reasonings+of+large+language+models+through+tool+creation%2E+arXiv+preprint+arXiv%3A2305%2E1)
24. Yujia Qin, Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlun Zhu, Yankai Lin, Xu Han, Ning Ding, Huadong Wang, et al. Webcpm: Interactive web search for chinese long-form question answering. arXiv preprint arXiv:2305.06849, 2023a. [find](https://scholar.google.com/scholar?q=Yujia+Qin%2C+Zihan+Cai%2C+Dian+Jin%2C+Lan+Yan%2C+Shihao+Liang%2C+Kunlun+Zhu%2C+Yankai+Lin%2C+Xu+Han%2C+Ning+Ding%2C+Huadong+Wang%2C+et+al%2E+Webcpm%3A+Interactive+web+search+for+chinese+long%2Dform+question+answering%2E+arXiv+pr)
25. Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, et al. Tool learning with foundation models. arXiv preprint arXiv:2304.08354, 2023b. [find](https://scholar.google.com/scholar?q=Yujia+Qin%2C+Shengding+Hu%2C+Yankai+Lin%2C+Weize+Chen%2C+Ning+Ding%2C+Ganqu+Cui%2C+Zheni+Zeng%2C+Yufei+Huang%2C+Chaojun+Xiao%2C+Chi+Han%2C+et+al%2E+Tool+learning+with+foundation+models%2E+arXiv+preprint+arXiv%3A2304%2E08354%2C+202) [our card](https://md.research.elamur.ai/free/en/papers/qin-2023-tool-learning-foundation-models/)
26. Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019. [find](https://scholar.google.com/scholar?q=Nils+Reimers+and+Iryna+Gurevych%2E+Sentence%2Dbert%3A+Sentence+embeddings+using+siamese+bert%2Dnetworks%2E+arXiv+preprint+arXiv%3A1908%2E10084%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/)
27. Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389, 2009. [find](https://scholar.google.com/scholar?q=Stephen+Robertson%2C+Hugo+Zaragoza%2C+et+al%2E+The+probabilistic+relevance+framework%3A+Bm25+and+beyond%2E+Foundations+and+Trends%C2%AE+in+Information+Retrieval%2C+3%284%29%3A333%E2%80%93389%2C+2009%2E)
28. Timo Schick, Jane Dwivedi-Yu, Roberto Dessı̀, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. ArXiv preprint, abs/2302.04761, 2023. [find](https://scholar.google.com/scholar?q=Timo+Schick%2C+Jane+Dwivedi%2DYu%2C+Roberto+Dess%C4%B1%CC%80%2C+Roberta+Raileanu%2C+Maria+Lomeli%2C+Luke+Zettlemoyer%2C+Nicola+Cancedda%2C+and+Thomas+Scialom%2E+Toolformer%3A+Language+models+can+teach+themselves+to+use+tools%2E+ArXi) [our card](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/)
29. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface, 2023. [find](https://scholar.google.com/scholar?q=Yongliang+Shen%2C+Kaitao+Song%2C+Xu+Tan%2C+Dongsheng+Li%2C+Weiming+Lu%2C+and+Yueting+Zhuang%2E+Hugginggpt%3A+Solving+ai+tasks+with+chatgpt+and+its+friends+in+huggingface%2C+2023%2E)
30. Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023. [find](https://scholar.google.com/scholar?q=Noah+Shinn%2C+Federico+Cassano%2C+Beck+Labash%2C+Ashwin+Gopinath%2C+Karthik+Narasimhan%2C+and+Shunyu+Yao%2E+Reflexion%3A+Language+agents+with+verbal+reinforcement+learning%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/)
31. Yifan Song, Weimin Xiong, Dawei Zhu, Cheng Li, Ke Wang, Ye Tian, and Sujian Li. Restgpt: Connecting large language models with real-world applications via restful apis. arXiv preprint arXiv:2306.06624, 2023. [find](https://scholar.google.com/scholar?q=Yifan+Song%2C+Weimin+Xiong%2C+Dawei+Zhu%2C+Cheng+Li%2C+Ke+Wang%2C+Ye+Tian%2C+and+Sujian+Li%2E+Restgpt%3A+Connecting+large+language+models+with+real%2Dworld+applications+via+restful+apis%2E+arXiv+preprint+arXiv%3A2306%2E06624) [our card](https://md.research.elamur.ai/free/en/papers/song-2023-restgpt-connecting-large-language-models/)
32. Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05301, 2023. [find](https://scholar.google.com/scholar?q=Qiaoyu+Tang%2C+Ziliang+Deng%2C+Hongyu+Lin%2C+Xianpei+Han%2C+Qiao+Liang%2C+and+Le+Sun%2E+Toolalpaca%3A+Generalized+tool+learning+for+language+models+with+3000+simulated+cases%2E+arXiv+preprint+arXiv%3A2306%2E05301%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/tang-2023-toolalpaca-generalized-tool-learning-language/)
33. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023. [↗](https://github.com/tatsu-lab/stanford_alpaca)
34. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv prepri… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Thibaut+Lavril%2C+Gautier+Izacard%2C+Xavier+Martinet%2C+Marie%2DAnne+Lachaux%2C+Timothe%CC%81e+Lacroix%2C+Baptiste+Rozie%CC%80re%2C+Naman+Goyal%2C+Eric+Hambro%2C+Faisal+Azhar%2C+Aurelien+Rodriguez%2C+Armand+Joulin%2C+Edo) [our card](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/)
35. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+Llama+2%3A+Open+foundation+and+fine%2Dtun)
36. Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. Chatgpt for robotics: Design principles and model abilities. Technical Report MSR-TR-2023-8, Microsoft, February 2023. [find](https://scholar.google.com/scholar?q=Sai+Vemprala%2C+Rogerio+Bonatti%2C+Arthur+Bucker%2C+and+Ashish+Kapoor%2E+Chatgpt+for+robotics%3A+Design+principles+and+model+abilities%2E+Technical+Report+MSR%2DTR%2D2023%2D8%2C+Microsoft%2C+February+2023%2E)
37. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+Self%2Dinstruct%3A+Aligning+language+model+with+self+generated+instructions%2E+arXiv+preprint+)
38. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Maarten+Bosma%2C+Vincent+Y+Zhao%2C+Kelvin+Guu%2C+Adams+Wei+Yu%2C+Brian+Lester%2C+Nan+Du%2C+Andrew+M+Dai%2C+and+Quoc+V+Le%2E+Finetuned+language+models+are+zero%2Dshot+learners%2E+arXiv+preprint+arXiv%3A2109%2E01652)
39. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Xuezhi+Wang%2C+Dale+Schuurmans%2C+Maarten+Bosma%2C+Brian+Ichter%2C+Fei+Xia%2C+Ed+Chi%2C+Quoc+Le%2C+and+Denny+Zhou%2E+Chain%2Dof%2Dthought+prompting+elicits+reasoning+in+large+language+models%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
40. Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. ArXiv preprint, abs/2303.04671, 2023. [find](https://scholar.google.com/scholar?q=Chenfei+Wu%2C+Shengming+Yin%2C+Weizhen+Qi%2C+Xiaodong+Wang%2C+Zecheng+Tang%2C+and+Nan+Duan%2E+Visual+chatgpt%3A+Talking%2C+drawing+and+editing+with+visual+foundation+models%2E+ArXiv+preprint%2C+abs%2F2303%2E04671%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/wu-2023-visual-chatgpt-talking-drawing-editing/)
41. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions, 2023a. [find](https://scholar.google.com/scholar?q=Can+Xu%2C+Qingfeng+Sun%2C+Kai+Zheng%2C+Xiubo+Geng%2C+Pu+Zhao%2C+Jiazhan+Feng%2C+Chongyang+Tao%2C+and+Daxin+Jiang%2E+Wizardlm%3A+Empowering+large+language+models+to+follow+complex+instructions%2C+2023a%2E)
42. Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. On the tool manipulation capability of open-source large language models. arXiv preprint arXiv:2305.16504, 2023b. [find](https://scholar.google.com/scholar?q=Qiantong+Xu%2C+Fenglu+Hong%2C+Bo+Li%2C+Changran+Hu%2C+Zhengyu+Chen%2C+and+Jian+Zhang%2E+On+the+tool+manipulation+capability+of+open%2Dsource+large+language+models%2E+arXiv+preprint+arXiv%3A2305%2E16504%2C+2023b%2E)
43. Linyao Yang, Hongyang Chen, Zhao Li, Xiao Ding, and Xindong Wu. Chatgpt is not enough: Enhancing large language models with knowledge graphs for fact-aware language modeling. arXiv preprint arXiv:2306.11489, 2023. — [arXiv:2306.11489](https://arxiv.org/abs/2306.11489) [find](https://scholar.google.com/scholar?q=Linyao+Yang%2C+Hongyang+Chen%2C+Zhao+Li%2C+Xiao+Ding%2C+and+Xindong+Wu%2E+Chatgpt+is+not+enough%3A+Enhancing+large+language+models+with+knowledge+graphs+for+fact%2Daware+language+modeling%2E+arXiv+preprint+arXiv%3A2306)
44. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. ArXiv preprint, abs/2210.03629, 2022. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Jeffrey+Zhao%2C+Dian+Yu%2C+Nan+Du%2C+Izhak+Shafran%2C+Karthik+Narasimhan%2C+and+Yuan+Cao%2E+React%3A+Synergizing+reasoning+and+acting+in+language+models%2E+ArXiv+preprint%2C+abs%2F2210%2E03629%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/)
45. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601, 2023. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Thomas+L+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+arXiv+preprint+arXiv%3A23)
46. Yining Ye, Xin Cong, Yujia Qin, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Large language model as autonomous decision maker. arXiv preprint arXiv:2308.12519, 2023. — [arXiv:2308.12519](https://arxiv.org/abs/2308.12519) [find](https://scholar.google.com/scholar?q=Yining+Ye%2C+Xin+Cong%2C+Yujia+Qin%2C+Yankai+Lin%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+Large+language+model+as+autonomous+decision+maker%2E+arXiv+preprint+arXiv%3A2308%2E12519%2C+2023%2E+%E2%80%94+%5BarXiv%3A2308%2E12519%5D%28https%3A%2F%2Farxiv%2Eo)
47. Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. Toolqa: A dataset for llm question answering with external tools. arXiv preprint arXiv:2306.13304, 2023. [find](https://scholar.google.com/scholar?q=Yuchen+Zhuang%2C+Yue+Yu%2C+Kuan+Wang%2C+Haotian+Sun%2C+and+Chao+Zhang%2E+Toolqa%3A+A+dataset+for+llm+question+answering+with+external+tools%2E+arXiv+preprint+arXiv%3A2306%2E13304%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhuang-2023-toolqa-dataset-llm-question-answering/)