---
title: "Gorilla: Large Language Model Connected with — Patil 2023"
description: "Large language models often fail when asked to write calls to external APIs."
authors: ["Shishir G. Patil", "Tianjun Zhang", "Xin Wang", "Joseph E. Gonzalez"]
year: 2023
venue: "arXiv:2305.15334"
source: "https://arxiv.org/abs/2305.15334"
doi: "10.48550/arxiv.2305.15334"
arxiv: "2305.15334"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/patil-2023-gorilla-large-language-model-connected/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/patil-2023-gorilla-large-language-model-connected/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/) / Gorilla: Large Language Model Connected with Massive APIs

# Gorilla: Large Language Model Connected with Massive APIs

**Authors:** Shishir G. Patil, [Tianjun Zhang](https://md.research.elamur.ai/free/en/authors/tianjun-zhang/), [Xin Wang](https://md.research.elamur.ai/free/en/authors/xin-wang/), [Joseph E. Gonzalez](https://md.research.elamur.ai/free/en/authors/joseph-e-gonzalez/) · **Year:** 2023 · **Venue:** arXiv:2305.15334 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2305.15334) · [PDF](https://arxiv.org/pdf/2305.15334) · [DOI 10.48550/arxiv.2305.15334](https://doi.org/10.48550/arxiv.2305.15334) · [arXiv 2305.15334](https://arxiv.org/abs/2305.15334) · [OpenAlex](https://openalex.org/W4415795271)

**Citations:** 196 citations (openalex-snapshot · 2026-10-14) · [331](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Large language models often fail when asked to write calls to external APIs. They invent models that do not exist or pass wrong arguments, and a prompt cannot list millions of APIs that keep changing. The authors present two contributions to address this. Gorilla is a LLaMA-7B model fine-tuned specifically to produce API calls. APIBench is a benchmark built from machine-learning model cards on HuggingFace, TorchHub and TensorHub. Each API's documentation was converted into a uniform JSON record, and GPT-4 generated ten natural-language requests per API in a self-instruct fashion, guided by a few hand-written examples. Gorilla is trained with retrieved documentation included in its input ("retriever-aware" training). It can work zero-shot or alongside a retriever such as BM25 or GPT-Index. Outputs are scored by matching AST sub-trees against the reference API, which also exposes hallucinated calls. Gorilla outperforms prompted GPT-4 in API accuracy, hallucinates less, adapts to documentation changes at inference time and handles accuracy constraints competitively. This matters because reliable API invocation is a prerequisite for LLMs to serve as interfaces to software tools.

## Key points
- APIBench covers 1,645 API calls: 94 from Torch Hub, 626 from TensorFlow Hub v2 (filtered from 801) and 925 from HuggingFace (top 20 models in each of 36 domains). These yield 16,450 instruction–API pairs, and only 18 seed examples were written by hand. The extracted text also contains conflicting figures (95, 696, "over 11,000").
- Evaluation parses the generated code and locates the sub-tree rooted at the API call, for example `torch.hub.load`. This allows correct calls to be separated from wrong or hallucinated ones.
- Retriever-aware training lets Gorilla follow updated documentation at test time. Examples include switching the FCN backbone from ResNet-50 to ResNet-101 and moving the model registry from pytorch/vision to NVIDIA/DeepLearningExamples:torchhub.
- On a constraint-aware subset (Torch Hub APIs with reported accuracy, 65.26% of that hub), zero-shot Gorilla scored 47.88% versus 43.66% for both GPT-3.5 and GPT-4. With an oracle retriever, Gorilla reached 67.60% versus 69.01% for GPT-3.5.
- Code, model, data and a demo are released at gorilla.cs.berkeley.edu.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

331 papers of our library cite it (of the 140673 whose bibliographies we read); 18 of them have a page here, 313 not yet.

- [Mind2Web: Towards a Generalist Agent for the Web](https://md.research.elamur.ai/free/en/papers/deng-2023-mind2web-towards-generalist-agent-web/)
- [CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution](https://md.research.elamur.ai/free/en/papers/gu-2024-cruxeval-benchmark-code-reasoning-understanding/)
- [Evaluating Large Language Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/guo-2023-evaluating-large-language-models-comprehensive/)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/)
- [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://md.research.elamur.ai/free/en/papers/li-2023-api-bank-comprehensive-benchmark-tool/)
- [GAIA: a benchmark for General AI Assistants](https://md.research.elamur.ai/free/en/papers/mialon-2023-gaia-benchmark-general-ai-assistants/)
- [A Comprehensive Overview of Large Language Models](https://md.research.elamur.ai/free/en/papers/naveed-2023-comprehensive-overview-large-language-models/)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://md.research.elamur.ai/free/en/papers/qin-2023-toolllm-facilitating-large-language-models/)
- [ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases](https://md.research.elamur.ai/free/en/papers/tang-2023-toolalpaca-generalized-tool-learning-language/)
- [MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback](https://md.research.elamur.ai/free/en/papers/wang-2023-mint-evaluating-llms-multi-turn/)
- [Executable Code Actions Elicit Better LLM Agents](https://md.research.elamur.ai/free/en/papers/wang-2024-executable-code-actions-elicit-better/)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://md.research.elamur.ai/free/en/papers/wang-2024-openhands-open-platform-ai/)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://md.research.elamur.ai/free/en/papers/wu-2023-autogen-enabling-next-gen-llm/)
- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://md.research.elamur.ai/free/en/papers/zheng-2023-sglang-efficient-execution-structured-language/)
- [大语言模型的工具使用综述](https://md.research.elamur.ai/free/en/papers/zheng-2025-tool-use-large/)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://md.research.elamur.ai/free/en/papers/zhuang-2023-toolqa-dataset-llm-question-answering/)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://md.research.elamur.ai/free/en/papers/zhuo-2024-bigcodebench-benchmarking-code-generation-diverse/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{patil2023gorilla,
  title = {{Gorilla: Large Language Model Connected with Massive APIs}},
  author = {Patil, Shishir G. and Zhang, Tianjun and Wang, Xin and Gonzalez, Joseph E.},
  year = {2023},
  doi = {10.48550/arxiv.2305.15334},
  eprint = {2305.15334},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.15334}
}
```

- [← previous GlotLID: Language Identification for Low-Resource Languages](https://md.research.elamur.ai/free/en/papers/kargaran-2023-glotlid-language-identification-low-resource/)
- [next → GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/)

## References

48 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., et al. (2022). Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691. [find](https://scholar.google.com/scholar?q=Ahn%2C+M%2E%2C+Brohan%2C+A%2E%2C+Brown%2C+N%2E%2C+Chebotar%2C+Y%2E%2C+Cortes%2C+O%2E%2C+David%2C+B%2E%2C+Finn%2C+C%2E%2C+Gopalakrishnan%2C+K%2E%2C+Hausman%2C+K%2E%2C+Herzog%2C+A%2E%2C+et+al%2E+%282022%29%2E+Do+as+i+can%2C+not+as+i+say%3A+Grounding+language+in+robotic+affo) [our card](https://md.research.elamur.ai/free/en/papers/ahn-2022-i-i-say-grounding-language/)
2. Andor, D., He, L., Lee, K., and Pitler, E. (2019). Giving bert a calculator: Finding operations and arguments with reading comprehension. arXiv preprint arXiv:1909.00109. [find](https://scholar.google.com/scholar?q=Andor%2C+D%2E%2C+He%2C+L%2E%2C+Lee%2C+K%2E%2C+and+Pitler%2C+E%2E+%282019%29%2E+Giving+bert+a+calculator%3A+Finding+operations+and+arguments+with+reading+comprehension%2E+arXiv+preprint+arXiv%3A1909%2E00109%2E)
3. Anthropic, h.-c. (2022). Claude. [find](https://scholar.google.com/scholar?q=Anthropic%2C+h%2E%2Dc%2E+%282022%29%2E+Claude%2E)
4. Bavishi, R., Lemieux, C., Fox, R., Sen, K., and Stoica, I. (2019). Autopandas: neural-backed generators for program synthesis. Proceedings of the ACM on Programming Languages, 3(OOPSLA):1–27. [find](https://scholar.google.com/scholar?q=Bavishi%2C+R%2E%2C+Lemieux%2C+C%2E%2C+Fox%2C+R%2E%2C+Sen%2C+K%2E%2C+and+Stoica%2C+I%2E+%282019%29%2E+Autopandas%3A+neural%2Dbacked+generators+for+program+synthesis%2E+Proceedings+of+the+ACM+on+Programming+Languages%2C+3%28OOPSLA%29%3A1%E2%80%9327%2E)
5. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. [find](https://scholar.google.com/scholar?q=Brown%2C+T%2E%2C+Mann%2C+B%2E%2C+Ryder%2C+N%2E%2C+Subbiah%2C+M%2E%2C+Kaplan%2C+J%2E+D%2E%2C+Dhariwal%2C+P%2E%2C+Neelakantan%2C+A%2E%2C+Shyam%2C+P%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+et+al%2E+%282020%29%2E+Language+models+are+few%2Dshot+learners%2E+Advances+in+neural+i)
6. Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023). Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712. [find](https://scholar.google.com/scholar?q=Bubeck%2C+S%2E%2C+Chandrasekaran%2C+V%2E%2C+Eldan%2C+R%2E%2C+Gehrke%2C+J%2E%2C+Horvitz%2C+E%2E%2C+Kamar%2C+E%2E%2C+Lee%2C+P%2E%2C+Lee%2C+Y%2E+T%2E%2C+Li%2C+Y%2E%2C+Lundberg%2C+S%2E%2C+et+al%2E+%282023%29%2E+Sparks+of+artificial+general+intelligence%3A+Early+experiments+wi) [our card](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/)
7. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. [find](https://scholar.google.com/scholar?q=Chen%2C+M%2E%2C+Tworek%2C+J%2E%2C+Jun%2C+H%2E%2C+Yuan%2C+Q%2E%2C+Pinto%2C+H%2E+P%2E+d%2E+O%2E%2C+Kaplan%2C+J%2E%2C+Edwards%2C+H%2E%2C+Burda%2C+Y%2E%2C+Joseph%2C+N%2E%2C+Brockman%2C+G%2E%2C+et+al%2E+%282021%29%2E+Evaluating+large+language+models+trained+on+code%2E+arXiv+prepri) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
8. Chen, X., Lin, M., Schärli, N., and Zhou, D. (2023). Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128. [find](https://scholar.google.com/scholar?q=Chen%2C+X%2E%2C+Lin%2C+M%2E%2C+Sch%C3%A4rli%2C+N%2E%2C+and+Zhou%2C+D%2E+%282023%29%2E+Teaching+large+language+models+to+self%2Ddebug%2E+arXiv+preprint+arXiv%3A2304%2E05128%2E)
9. Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P. (2023). Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. [find](https://scholar.google.com/scholar?q=Chiang%2C+W%2E%2DL%2E%2C+Li%2C+Z%2E%2C+Lin%2C+Z%2E%2C+Sheng%2C+Y%2E%2C+Wu%2C+Z%2E%2C+Zhang%2C+H%2E%2C+Zheng%2C+L%2E%2C+Zhuang%2C+S%2E%2C+Zhuang%2C+Y%2E%2C+Gonzalez%2C+J%2E+E%2E%2C+Stoica%2C+I%2E%2C+and+Xing%2C+E%2E+P%2E+%282023%29%2E+Vicuna%3A+An+open%2Dsource+chatbot+impressing+gpt%2D4+wi)
10. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022). Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. [find](https://scholar.google.com/scholar?q=Chowdhery%2C+A%2E%2C+Narang%2C+S%2E%2C+Devlin%2C+J%2E%2C+Bosma%2C+M%2E%2C+Mishra%2C+G%2E%2C+Roberts%2C+A%2E%2C+Barham%2C+P%2E%2C+Chung%2C+H%2E+W%2E%2C+Sutton%2C+C%2E%2C+Gehrmann%2C+S%2E%2C+et+al%2E+%282022%29%2E+Palm%3A+Scaling+language+modeling+with+pathways%2E+arXiv+prepr)
11. Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., et al. (2022). Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. [find](https://scholar.google.com/scholar?q=Chung%2C+H%2E+W%2E%2C+Hou%2C+L%2E%2C+Longpre%2C+S%2E%2C+Zoph%2C+B%2E%2C+Tay%2C+Y%2E%2C+Fedus%2C+W%2E%2C+Li%2C+E%2E%2C+Wang%2C+X%2E%2C+Dehghani%2C+M%2E%2C+Brahma%2C+S%2E%2C+et+al%2E+%282022%29%2E+Scaling+instruction%2Dfinetuned+language+models%2E+arXiv+preprint+arXiv%3A2210%2E11)
12. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021). Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. [find](https://scholar.google.com/scholar?q=Cobbe%2C+K%2E%2C+Kosaraju%2C+V%2E%2C+Bavarian%2C+M%2E%2C+Chen%2C+M%2E%2C+Jun%2C+H%2E%2C+Kaiser%2C+L%2E%2C+Plappert%2C+M%2E%2C+Tworek%2C+J%2E%2C+Hilton%2C+J%2E%2C+Nakano%2C+R%2E%2C+et+al%2E+%282021%29%2E+Training+verifiers+to+solve+math+word+problems%2E+arXiv+preprint+ar) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
13. Devlin, J., Uesato, J., Bhupatiraju, S., Singh, R., Mohamed, A.-r., and Kohli, P. (2017). Robustfill: Neural program learning under noisy i/o. In International conference on machine learning, pages 990–998. PMLR. [find](https://scholar.google.com/scholar?q=Devlin%2C+J%2E%2C+Uesato%2C+J%2E%2C+Bhupatiraju%2C+S%2E%2C+Singh%2C+R%2E%2C+Mohamed%2C+A%2E%2Dr%2E%2C+and+Kohli%2C+P%2E+%282017%29%2E+Robustfill%3A+Neural+program+learning+under+noisy+i%2Fo%2E+In+International+conference+on+machine+learning%2C+pages+99)
14. Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. (2022). Pal: Program-aided language models. arXiv preprint arXiv:2211.10435. [find](https://scholar.google.com/scholar?q=Gao%2C+L%2E%2C+Madaan%2C+A%2E%2C+Zhou%2C+S%2E%2C+Alon%2C+U%2E%2C+Liu%2C+P%2E%2C+Yang%2C+Y%2E%2C+Callan%2C+J%2E%2C+and+Neubig%2C+G%2E+%282022%29%2E+Pal%3A+Program%2Daided+language+models%2E+arXiv+preprint+arXiv%3A2211%2E10435%2E)
15. Iyer, S., Lin, X. V., Pasunuru, R., Mihaylov, T., Simig, D., Yu, P., Shuster, K., Wang, T., Liu, Q., Koura, P. S., et al. (2022). Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212.12017. [find](https://scholar.google.com/scholar?q=Iyer%2C+S%2E%2C+Lin%2C+X%2E+V%2E%2C+Pasunuru%2C+R%2E%2C+Mihaylov%2C+T%2E%2C+Simig%2C+D%2E%2C+Yu%2C+P%2E%2C+Shuster%2C+K%2E%2C+Wang%2C+T%2E%2C+Liu%2C+Q%2E%2C+Koura%2C+P%2E+S%2E%2C+et+al%2E+%282022%29%2E+Opt%2Diml%3A+Scaling+language+model+instruction+meta+learning+through+the+) [our card](https://md.research.elamur.ai/free/en/papers/iyer-2022-opt-iml-scaling-language-model/)
16. Jain, N., Vaidyanath, S., Iyer, A., Natarajan, N., Parthasarathy, S., Rajamani, S., and Sharma, R. (2022). Jigsaw: Large language models meet program synthesis. In Proceedings of the 44th International Conference on Software Engineering, pages 1219–1231. [find](https://scholar.google.com/scholar?q=Jain%2C+N%2E%2C+Vaidyanath%2C+S%2E%2C+Iyer%2C+A%2E%2C+Natarajan%2C+N%2E%2C+Parthasarathy%2C+S%2E%2C+Rajamani%2C+S%2E%2C+and+Sharma%2C+R%2E+%282022%29%2E+Jigsaw%3A+Large+language+models+meet+program+synthesis%2E+In+Proceedings+of+the+44th+Internationa)
17. Kim, G., Baldi, P., and McAleer, S. (2023). Language models can solve computer tasks. arXiv preprint arXiv:2303.17491. [find](https://scholar.google.com/scholar?q=Kim%2C+G%2E%2C+Baldi%2C+P%2E%2C+and+McAleer%2C+S%2E+%282023%29%2E+Language+models+can+solve+computer+tasks%2E+arXiv+preprint+arXiv%3A2303%2E17491%2E) [our card](https://md.research.elamur.ai/free/en/papers/kim-2023-language-models-solve-computer-tasks/)
18. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. (2022). Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916. [find](https://scholar.google.com/scholar?q=Kojima%2C+T%2E%2C+Gu%2C+S%2E+S%2E%2C+Reid%2C+M%2E%2C+Matsuo%2C+Y%2E%2C+and+Iwasawa%2C+Y%2E+%282022%29%2E+Large+language+models+are+zero%2Dshot+reasoners%2E+arXiv+preprint+arXiv%3A2205%2E11916%2E) [our card](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/)
19. Komeili, M., Shuster, K., and Weston, J. (2021). Internet-augmented dialogue generation. arXiv preprint arXiv:2107.07566. [find](https://scholar.google.com/scholar?q=Komeili%2C+M%2E%2C+Shuster%2C+K%2E%2C+and+Weston%2C+J%2E+%282021%29%2E+Internet%2Daugmented+dialogue+generation%2E+arXiv+preprint+arXiv%3A2107%2E07566%2E) [our card](https://md.research.elamur.ai/free/en/papers/komeili-2022-internet-augmented-dialogue-generation/)
20. Lachaux, M.-A., Roziere, B., Chanussot, L., and Lample, G. (2020). Unsupervised translation of programming languages. arXiv preprint arXiv:2006.03511. [find](https://scholar.google.com/scholar?q=Lachaux%2C+M%2E%2DA%2E%2C+Roziere%2C+B%2E%2C+Chanussot%2C+L%2E%2C+and+Lample%2C+G%2E+%282020%29%2E+Unsupervised+translation+of+programming+languages%2E+arXiv+preprint+arXiv%3A2006%2E03511%2E) [our card](https://md.research.elamur.ai/free/en/papers/lachaux-2020-unsupervised-translation-programming-languages/)
21. Lazaridou, A., Gribovskaya, E., Stokowiec, W., and Grigorev, N. (2022). Internet-augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05115. [find](https://scholar.google.com/scholar?q=Lazaridou%2C+A%2E%2C+Gribovskaya%2C+E%2E%2C+Stokowiec%2C+W%2E%2C+and+Grigorev%2C+N%2E+%282022%29%2E+Internet%2Daugmented+language+models+through+few%2Dshot+prompting+for+open%2Ddomain+question+answering%2E+arXiv+preprint+arXiv%3A2203%2E0511) [our card](https://md.research.elamur.ai/free/en/papers/lazaridou-2022-internet-augmented-language-models-through/)
22. Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. (2023). Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161. [find](https://scholar.google.com/scholar?q=Li%2C+R%2E%2C+Allal%2C+L%2E+B%2E%2C+Zi%2C+Y%2E%2C+Muennighoff%2C+N%2E%2C+Kocetkov%2C+D%2E%2C+Mou%2C+C%2E%2C+Marone%2C+M%2E%2C+Akiki%2C+C%2E%2C+Li%2C+J%2E%2C+Chim%2C+J%2E%2C+et+al%2E+%282023%29%2E+Starcoder%3A+may+the+source+be+with+you%21+arXiv+preprint+arXiv%3A2305%2E06161%2E)
23. Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022). Competition-level code generation with alphacode. Science, 378(6624):1092–1097. [find](https://scholar.google.com/scholar?q=Li%2C+Y%2E%2C+Choi%2C+D%2E%2C+Chung%2C+J%2E%2C+Kushman%2C+N%2E%2C+Schrittwieser%2C+J%2E%2C+Leblond%2C+R%2E%2C+Eccles%2C+T%2E%2C+Keeling%2C+J%2E%2C+Gimeno%2C+F%2E%2C+Dal+Lago%2C+A%2E%2C+et+al%2E+%282022%29%2E+Competition%2Dlevel+code+generation+with+alphacode%2E+Science%2C+3) [our card](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/)
24. Liang, Y., Wu, C., Song, T., Wu, W., Xia, Y., Liu, Y., Ou, Y., Lu, S., Ji, L., Mao, S., et al. (2023). Taskmatrix. ai: Completing tasks by connecting foundation models with millions of apis. arXiv preprint arXiv:2303.16434. — [arXiv:2303.16434](https://arxiv.org/abs/2303.16434) [find](https://scholar.google.com/scholar?q=Liang%2C+Y%2E%2C+Wu%2C+C%2E%2C+Song%2C+T%2E%2C+Wu%2C+W%2E%2C+Xia%2C+Y%2E%2C+Liu%2C+Y%2E%2C+Ou%2C+Y%2E%2C+Lu%2C+S%2E%2C+Ji%2C+L%2E%2C+Mao%2C+S%2E%2C+et+al%2E+%282023%29%2E+Taskmatrix%2E+ai%3A+Completing+tasks+by+connecting+foundation+models+with+millions+of+apis%2E+arXiv+pre)
25. Menon, A., Tamuz, O., Gulwani, S., Lampson, B., and Kalai, A. (2013). A machine learning framework for programming by example. In International Conference on Machine Learning, pages 187–195. PMLR. [find](https://scholar.google.com/scholar?q=Menon%2C+A%2E%2C+Tamuz%2C+O%2E%2C+Gulwani%2C+S%2E%2C+Lampson%2C+B%2E%2C+and+Kalai%2C+A%2E+%282013%29%2E+A+machine+learning+framework+for+programming+by+example%2E+In+International+Conference+on+Machine+Learning%2C+pages+187%E2%80%93195%2E+PMLR%2E)
26. Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. (2021). Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332. [find](https://scholar.google.com/scholar?q=Nakano%2C+R%2E%2C+Hilton%2C+J%2E%2C+Balaji%2C+S%2E%2C+Wu%2C+J%2E%2C+Ouyang%2C+L%2E%2C+Kim%2C+C%2E%2C+Hesse%2C+C%2E%2C+Jain%2C+S%2E%2C+Kosaraju%2C+V%2E%2C+Saunders%2C+W%2E%2C+et+al%2E+%282021%29%2E+Webgpt%3A+Browser%2Dassisted+question%2Danswering+with+human+feedback%2E+arXiv+) [our card](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/)
27. Nijkamp, E., Hayashi, H., Xiong, C., Savarese, S., and Zhou, Y. (2023). Codegen2: Lessons for training llms on programming and natural languages. arXiv preprint arXiv:2305.02309. [find](https://scholar.google.com/scholar?q=Nijkamp%2C+E%2E%2C+Hayashi%2C+H%2E%2C+Xiong%2C+C%2E%2C+Savarese%2C+S%2E%2C+and+Zhou%2C+Y%2E+%282023%29%2E+Codegen2%3A+Lessons+for+training+llms+on+programming+and+natural+languages%2E+arXiv+preprint+arXiv%3A2305%2E02309%2E)
28. Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., and Xiong, C. (2022). Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474. [find](https://scholar.google.com/scholar?q=Nijkamp%2C+E%2E%2C+Pang%2C+B%2E%2C+Hayashi%2C+H%2E%2C+Tu%2C+L%2E%2C+Wang%2C+H%2E%2C+Zhou%2C+Y%2E%2C+Savarese%2C+S%2E%2C+and+Xiong%2C+C%2E+%282022%29%2E+Codegen%3A+An+open+large+language+model+for+code+with+multi%2Dturn+program+synthesis%2E+arXiv+preprint+arX)
29. OpenAI (2023). Gpt-4 technical report. [find](https://scholar.google.com/scholar?q=OpenAI+%282023%29%2E+Gpt%2D4+technical+report%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
30. OpenAI and [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt) (2022). Chatgpt. [↗](https://openai.com/blog/chatgpt)
31. Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021). Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207. [find](https://scholar.google.com/scholar?q=Sanh%2C+V%2E%2C+Webson%2C+A%2E%2C+Raffel%2C+C%2E%2C+Bach%2C+S%2E+H%2E%2C+Sutawika%2C+L%2E%2C+Alyafeai%2C+Z%2E%2C+Chaffin%2C+A%2E%2C+Stiegler%2C+A%2E%2C+Scao%2C+T%2E+L%2E%2C+Raja%2C+A%2E%2C+et+al%2E+%282021%29%2E+Multitask+prompted+training+enables+zero%2Dshot+task+generaliz) [our card](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
32. Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., et al. (2022). Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. [find](https://scholar.google.com/scholar?q=Scao%2C+T%2E+L%2E%2C+Fan%2C+A%2E%2C+Akiki%2C+C%2E%2C+Pavlick%2C+E%2E%2C+Ilic%CC%81%2C+S%2E%2C+Hesslow%2C+D%2E%2C+Castagn%C3%A9%2C+R%2E%2C+Luccioni%2C+A%2E+S%2E%2C+Yvon%2C+F%2E%2C+Gall%C3%A9%2C+M%2E%2C+et+al%2E+%282022%29%2E+Bloom%3A+A+176b%2Dparameter+open%2Daccess+multilingual+language+model)
33. Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., and Scialom, T. (2023). Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761. [find](https://scholar.google.com/scholar?q=Schick%2C+T%2E%2C+Dwivedi%2DYu%2C+J%2E%2C+Dess%C3%AC%2C+R%2E%2C+Raileanu%2C+R%2E%2C+Lomeli%2C+M%2E%2C+Zettlemoyer%2C+L%2E%2C+Cancedda%2C+N%2E%2C+and+Scialom%2C+T%2E+%282023%29%2E+Toolformer%3A+Language+models+can+teach+themselves+to+use+tools%2E+arXiv+preprint+ar) [our card](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/)
34. Schick, T. and Schütze, H. (2020). Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001.07676. [find](https://scholar.google.com/scholar?q=Schick%2C+T%2E+and+Sch%C3%BCtze%2C+H%2E+%282020%29%2E+Exploiting+cloze+questions+for+few+shot+text+classification+and+natural+language+inference%2E+arXiv+preprint+arXiv%3A2001%2E07676%2E) [our card](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/)
35. Shen, Y., Song, K., Tan, X., Li, D., Lu, W., and Zhuang, Y. (2023). Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580. [find](https://scholar.google.com/scholar?q=Shen%2C+Y%2E%2C+Song%2C+K%2E%2C+Tan%2C+X%2E%2C+Li%2C+D%2E%2C+Lu%2C+W%2E%2C+and+Zhuang%2C+Y%2E+%282023%29%2E+Hugginggpt%3A+Solving+ai+tasks+with+chatgpt+and+its+friends+in+huggingface%2E+arXiv+preprint+arXiv%3A2303%2E17580%2E) [our card](https://md.research.elamur.ai/free/en/papers/shen-2023-hugginggpt-solving-ai-tasks-chatgpt/)
36. Shinn, N., Labash, B., and Gopinath, A. (2023). Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366. [find](https://scholar.google.com/scholar?q=Shinn%2C+N%2E%2C+Labash%2C+B%2E%2C+and+Gopinath%2C+A%2E+%282023%29%2E+Reflexion%3A+an+autonomous+agent+with+dynamic+memory+and+self%2Dreflection%2E+arXiv+preprint+arXiv%3A2303%2E11366%2E) [our card](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/)
37. Shuster, K., Xu, J., Komeili, M., Ju, D., Smith, E. M., Roller, S., Ung, M., Chen, M., Arora, K., Lane, J., et al. (2022). Blenderbot 3: a deployed conversational agent that continually learns to responsibly engage. arXiv preprint arXiv:2208.03188. [find](https://scholar.google.com/scholar?q=Shuster%2C+K%2E%2C+Xu%2C+J%2E%2C+Komeili%2C+M%2E%2C+Ju%2C+D%2E%2C+Smith%2C+E%2E+M%2E%2C+Roller%2C+S%2E%2C+Ung%2C+M%2E%2C+Chen%2C+M%2E%2C+Arora%2C+K%2E%2C+Lane%2C+J%2E%2C+et+al%2E+%282022%29%2E+Blenderbot+3%3A+a+deployed+conversational+agent+that+continually+learns+to+resp) [our card](https://md.research.elamur.ai/free/en/papers/shuster-2022-blenderbot-3-deployed-conversational-agent/)
38. Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. (2023). Stanford alpaca: An instruction-following llama model. [https://github.com/](https://github.com/) tatsu-lab/stanford_alpaca. [↗](https://github.com/)
39. Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022). Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239. [find](https://scholar.google.com/scholar?q=Thoppilan%2C+R%2E%2C+De+Freitas%2C+D%2E%2C+Hall%2C+J%2E%2C+Shazeer%2C+N%2E%2C+Kulshreshtha%2C+A%2E%2C+Cheng%2C+H%2E%2DT%2E%2C+Jin%2C+A%2E%2C+Bos%2C+T%2E%2C+Baker%2C+L%2E%2C+Du%2C+Y%2E%2C+et+al%2E+%282022%29%2E+Lamda%3A+Language+models+for+dialog+applications%2E+arXiv+preprint) [our card](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/)
40. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. [find](https://scholar.google.com/scholar?q=Touvron%2C+H%2E%2C+Lavril%2C+T%2E%2C+Izacard%2C+G%2E%2C+Martinet%2C+X%2E%2C+Lachaux%2C+M%2E%2DA%2E%2C+Lacroix%2C+T%2E%2C+Rozi%C3%A8re%2C+B%2E%2C+Goyal%2C+N%2E%2C+Hambro%2C+E%2E%2C+Azhar%2C+F%2E%2C+et+al%2E+%282023%29%2E+Llama%3A+Open+and+efficient+foundation+language+models%2E+arX)
41. Vemprala, S., Bonatti, R., Bucker, A., and Kapoor, A. (2023). Chatgpt for robotics: Design principles and model abilities. 2023. [find](https://scholar.google.com/scholar?q=Vemprala%2C+S%2E%2C+Bonatti%2C+R%2E%2C+Bucker%2C+A%2E%2C+and+Kapoor%2C+A%2E+%282023%29%2E+Chatgpt+for+robotics%3A+Design+principles+and+model+abilities%2E+2023%2E)
42. Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. (2022a). Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560. [find](https://scholar.google.com/scholar?q=Wang%2C+Y%2E%2C+Kordi%2C+Y%2E%2C+Mishra%2C+S%2E%2C+Liu%2C+A%2E%2C+Smith%2C+N%2E+A%2E%2C+Khashabi%2C+D%2E%2C+and+Hajishirzi%2C+H%2E+%282022a%29%2E+Self%2Dinstruct%3A+Aligning+language+model+with+self+generated+instructions%2E+arXiv+preprint+arXiv%3A2212%2E105)
43. Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S., Arunkumar, A., Stap, D., et al. (2022b). Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. In Proceedings of the 2022 Conference on Empirical Methods… [find](https://scholar.google.com/scholar?q=Wang%2C+Y%2E%2C+Mishra%2C+S%2E%2C+Alipoormolabashi%2C+P%2E%2C+Kordi%2C+Y%2E%2C+Mirzaei%2C+A%2E%2C+Naik%2C+A%2E%2C+Ashok%2C+A%2E%2C+Dhanasekaran%2C+A%2E+S%2E%2C+Arunkumar%2C+A%2E%2C+Stap%2C+D%2E%2C+et+al%2E+%282022b%29%2E+Super%2Dnaturalinstructions%3A+Generalization+via+dec)
44. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. (2022). Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903. [find](https://scholar.google.com/scholar?q=Wei%2C+J%2E%2C+Wang%2C+X%2E%2C+Schuurmans%2C+D%2E%2C+Bosma%2C+M%2E%2C+Chi%2C+E%2E%2C+Le%2C+Q%2E%2C+and+Zhou%2C+D%2E+%282022%29%2E+Chain+of+thought+prompting+elicits+reasoning+in+large+language+models%2E+arXiv+preprint+arXiv%3A2201%2E11903%2E) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
45. Xu, F. F., Alon, U., Neubig, G., and Hellendoorn, V. J. (2022). A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pages 1–10. [find](https://scholar.google.com/scholar?q=Xu%2C+F%2E+F%2E%2C+Alon%2C+U%2E%2C+Neubig%2C+G%2E%2C+and+Hellendoorn%2C+V%2E+J%2E+%282022%29%2E+A+systematic+evaluation+of+large+language+models+of+code%2E+In+Proceedings+of+the+6th+ACM+SIGPLAN+International+Symposium+on+Machine+Progr)
46. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2022). React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629. [find](https://scholar.google.com/scholar?q=Yao%2C+S%2E%2C+Zhao%2C+J%2E%2C+Yu%2C+D%2E%2C+Du%2C+N%2E%2C+Shafran%2C+I%2E%2C+Narasimhan%2C+K%2E%2C+and+Cao%2C+Y%2E+%282022%29%2E+React%3A+Synergizing+reasoning+and+acting+in+language+models%2E+arXiv+preprint+arXiv%3A2210%2E03629%2E) [our card](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/)
47. Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., Xia, X., et al. (2022). Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414. [find](https://scholar.google.com/scholar?q=Zeng%2C+A%2E%2C+Liu%2C+X%2E%2C+Du%2C+Z%2E%2C+Wang%2C+Z%2E%2C+Lai%2C+H%2E%2C+Ding%2C+M%2E%2C+Yang%2C+Z%2E%2C+Xu%2C+Y%2E%2C+Zheng%2C+W%2E%2C+Xia%2C+X%2E%2C+et+al%2E+%282022%29%2E+Glm%2D130b%3A+An+open+bilingual+pre%2Dtrained+model%2E+arXiv+preprint+arXiv%3A2210%2E02414%2E) [our card](https://md.research.elamur.ai/free/en/papers/zeng-2022-glm-130b-open-bilingual-pre/)
48. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022). Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. [find](https://scholar.google.com/scholar?q=Zhang%2C+S%2E%2C+Roller%2C+S%2E%2C+Goyal%2C+N%2E%2C+Artetxe%2C+M%2E%2C+Chen%2C+M%2E%2C+Chen%2C+S%2E%2C+Dewan%2C+C%2E%2C+Diab%2C+M%2E%2C+Li%2C+X%2E%2C+Lin%2C+X%2E+V%2E%2C+et+al%2E+%282022%29%2E+Opt%3A+Open+pre%2Dtrained+transformer+language+models%2E+arXiv+preprint+arXiv%3A2205%2E)