---
title: "GeneGPT — paper by Jin et al. (2023) — summary and ideas"
description: "The authors also release GeneHop, a multi-hop dataset, and show that the system can chain several… 29 references, cited by 48 papers here."
authors: ["Qiao Jin", "Yifan Yang", "Qingyu Chen", "Zhiyong Lu"]
year: 2023
venue: "arXiv:2304.09667 · Bioinformatics, 2024"
source: "https://arxiv.org/abs/2304.09667"
doi: "10.1093/bioinformatics/btae075"
arxiv: "2304.09667"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/jin-2023-genegpt-augmenting-large-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/jin-2023-genegpt-augmenting-large-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/) / GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Information

# GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Information

**Authors:** [Qiao Jin](https://md.research.elamur.ai/free/en/authors/qiao-jin/), Yifan Yang, Qingyu Chen and [Zhiyong Lu](https://md.research.elamur.ai/free/en/authors/zhiyong-lu/) · **Year:** 2023 · **Venue:** [arXiv:2304.09667 · Bioinformatics, 2024](https://md.research.elamur.ai/free/en/venues/bioinformatics/) ([Bioinformatics 2023](https://md.research.elamur.ai/free/en/venues/bioinformatics/2023/)) · **Publisher:** [Oxford University Press](https://md.research.elamur.ai/free/en/publishers/oxford-university-press/) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://doi.org/10.1093/bioinformatics/btae075) · [PDF](https://arxiv.org/pdf/2304.09667) · [DOI 10.1093/bioinformatics/btae075](https://doi.org/10.1093/bioinformatics/btae075) · [arXiv 2304.09667](https://arxiv.org/abs/2304.09667) · [OpenAlex](https://openalex.org/W4366563389)

**Citations:** 174 citations (OpenAlex, 2026-10-14) · [48](https://md.research.elamur.ai/free/en/papers/jin-2023-genegpt-augmenting-large-language-models/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Large language models often invent facts when asked about precise database content, such as gene aliases, SNP positions or sequence alignments. Commercial search engines rarely index this kind of knowledge. GeneGPT addresses the problem by letting Codex (code-davinci-002) query the NCBI Web APIs directly.

The prompt combines four parts:
- a brief instruction,
- documentation for E-utils and the BLAST URL API,
- hand-written examples of API use,
- the test question.

During inference, a modified decoding loop detects API URLs that the model generates, executes them and feeds the returned data back into the context before generation continues.

On eight GeneTuring genomics QA tasks, the method scores 0.83 on average. This is nearly double the best baseline, the retrieval-backed New Bing (0.44), and far above domain and general models. The authors also release GeneHop, a multi-hop dataset, and show that the system can chain several sub-questions and API calls. The study suggests that specialized database tools may support biomedical information needs better than generic web retrieval.

## Key points
- **Tools used:** E-utils functions (esearch, efetch, esummary) over the gene, snp and omim databases, plus BLAST (blastn on nt). A "->" marker in the prompt signals where an API call happens.
- **GeneTuring results:** GeneGPT averages 0.83, an 88% gain over New Bing (0.44). The other baselines score far lower: GPT-3 0.16, ChatGPT 0.12, BioMedLM 0.08 and BioGPT 0.04.
- **Ablation:** worked examples help more than API documentation and transfer across tasks. A reduced variant, GeneGPT-slim, uses only two demonstrations and still performs well on all tasks.
- **GeneHop:** the dataset comprises three tasks that each require several reasoning steps: sequence gene alias, disease gene location and SNP gene function. GeneGPT averages 0.50 versus 0.24 for New Bing, and it solves chains longer than any shown in the prompt.
- **Error analysis:** errors fall into wrong API, wrong arguments, failure to extract the answer, questions unanswerable via NCBI, and other. Each task tends to concentrate on particular error types.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [National Institutes of Health](https://md.research.elamur.ai/free/en/government/national-institutes-of-health/) — Government body, Bethesda, United States · 9 works · 3,833 citations
- [National Center for Biotechnology Information](https://md.research.elamur.ai/free/en/labs/national-center-for-biotechnology-information/) — Research institute, Bethesda, United States · 3 works · 1,457 citations

📚 Shelves (8): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🧬 Biochemistry, Genetics and Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/) · 116 works · [🧬 Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/molecular-biology/) · 114 works · [🩺 Medicine](https://md.research.elamur.ai/free/en/shelves/medicine/) · 80 works · [🩺 Health Informatics](https://md.research.elamur.ai/free/en/shelves/medicine/health-informatics/) · 53 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works

## Cite

Cite as: Qiao Jin, Yifan Yang, Qingyu Chen and Zhiyong Lu. 2023. GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Information. arXiv:2304.09667 · Bioinformatics, 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{jin2023genegpt,
  title = {{GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Information}},
  author = {Qiao Jin, Yifan Yang, Qingyu Chen and Zhiyong Lu},
  year = {2023},
  doi = {10.1093/bioinformatics/btae075},
  eprint = {2304.09667},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1093/bioinformatics/btae075}
}
```

- [← previous A Survey of Hallucination in Large Foundation Models](https://md.research.elamur.ai/free/en/papers/rawte-2023-survey-hallucination-large-foundation-models/)
- [next → How Language Model Hallucinations Can Snowball](https://md.research.elamur.ai/free/en/papers/zhang-2023-language-model-hallucinations-snowball/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/)

## References

29 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Stephen F Altschul, Warren Gish, Webb Miller, Eugene W Myers, and David J Lipman. 1990. Basic local alignment search tool. Journal of molecular biology, 215(3):403–410. [find](https://scholar.google.com/scholar?q=Stephen+F+Altschul%2C+Warren+Gish%2C+Webb+Miller%2C+Eugene+W+Myers%2C+and+David+J+Lipman%2E+1990%2E+Basic+local+alignment+search+tool%2E+Journal+of+molecular+biology%2C+215%283%29%3A403%E2%80%93410%2E)
2. Grzegorz M Boratyn, Christiam Camacho, Peter S Cooper, George Coulouris, Amelia Fong, Ning Ma, Thomas L Madden, Wayne T Matten, Scott D McGinnis, Yuri Merezhuk, et al. 2013. Blast: a more efficient report with usability improvements. Nucleic acids research, 41(W1):W29–W33. [find](https://scholar.google.com/scholar?q=Grzegorz+M+Boratyn%2C+Christiam+Camacho%2C+Peter+S+Cooper%2C+George+Coulouris%2C+Amelia+Fong%2C+Ning+Ma%2C+Thomas+L+Madden%2C+Wayne+T+Matten%2C+Scott+D+McGinnis%2C+Yuri+Merezhuk%2C+et+al%2E+2013%2E+Blast%3A+a+more+efficient+re)
3. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from trillions of tokens. In International conference on machine learn… [find](https://scholar.google.com/scholar?q=Sebastian+Borgeaud%2C+Arthur+Mensch%2C+Jordan+Hoffmann%2C+Trevor+Cai%2C+Eliza+Rutherford%2C+Katie+Millican%2C+George+Bm+Van+Den+Driessche%2C+Jean%2DBaptiste+Lespiau%2C+Bogdan+Damoc%2C+Aidan+Clark%2C+et+al%2E+2022%2E+Improving+)
4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374.
6. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311.
7. John W Ely, Jerome A Osheroff, M Lee Chambliss, Mark H Ebell, and Marcy E Rosenbaum. 2005. Answering physicians’ clinical questions: obstacles and potential solutions. Journal of the American Medical Informatics Association, 12(2):217–224. [find](https://scholar.google.com/scholar?q=John+W+Ely%2C+Jerome+A+Osheroff%2C+M+Lee+Chambliss%2C+Mark+H+Ebell%2C+and+Marcy+E+Rosenbaum%2E+2005%2E+Answering+physicians%E2%80%99+clinical+questions%3A+obstacles+and+potential+solutions%2E+Journal+of+the+American+Medical+)
8. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2022. Pal: Program-aided language models. arXiv preprint arXiv:2211.10435. [find](https://scholar.google.com/scholar?q=Luyu+Gao%2C+Aman+Madaan%2C+Shuyan+Zhou%2C+Uri+Alon%2C+Pengfei+Liu%2C+Yiming+Yang%2C+Jamie+Callan%2C+and+Graham+Neubig%2E+2022%2E+Pal%3A+Program%2Daided+language+models%2E+arXiv+preprint+arXiv%3A2211%2E10435%2E)
9. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929– 3938. PMLR. [find](https://scholar.google.com/scholar?q=Kelvin+Guu%2C+Kenton+Lee%2C+Zora+Tung%2C+Panupong+Pasupat%2C+and+Mingwei+Chang%2E+2020%2E+Retrieval+augmented+language+model+pre%2Dtraining%2E+In+International+conference+on+machine+learning%2C+pages+3929%E2%80%93+3938%2E+PMLR%2E)
10. Wenpin Hou and Zhicheng Ji. 2023. Geneturing tests gpt models in genomics. bioRxiv, pages 2023–03. [find](https://scholar.google.com/scholar?q=Wenpin+Hou+and+Zhicheng+Ji%2E+2023%2E+Geneturing+tests+gpt+models+in+genomics%2E+bioRxiv%2C+pages+2023%E2%80%9303%2E)
11. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. [Survey of hallucination in natural language generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/). ACM Computing Surveys, 55(12):1–38.
12. Qiao Jin, Zheng Yuan, Guangzhi Xiong, Qianlan Yu, Huaiyuan Ying, Chuanqi Tan, Mosha Chen, Songfang Huang, Xiaozhong Liu, and Sheng Yu. 2022. Biomedical question answering: a survey of approaches and challenges. ACM Computing Surveys (CSUR), 55(2):1–36. [find](https://scholar.google.com/scholar?q=Qiao+Jin%2C+Zheng+Yuan%2C+Guangzhi+Xiong%2C+Qianlan+Yu%2C+Huaiyuan+Ying%2C+Chuanqi+Tan%2C+Mosha+Chen%2C+Songfang+Huang%2C+Xiaozhong+Liu%2C+and+Sheng+Yu%2E+2022%2E+Biomedical+question+answering%3A+a+survey+of+approaches+and+c)
13. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361.
14. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474. [find](https://scholar.google.com/scholar?q=Patrick+Lewis%2C+Ethan+Perez%2C+Aleksandra+Piktus%2C+Fabio+Petroni%2C+Vladimir+Karpukhin%2C+Naman+Goyal%2C+Heinrich+K%C3%BCttler%2C+Mike+Lewis%2C+Wen%2Dtau+Yih%2C+Tim+Rockt%C3%A4schel%2C+et+al%2E+2020%2E+Retrieval%2Daugmented+generation+f)
15. Valentin Liévin, Christoffer Egeberg Hother, and Ole Winther. 2022. Can large language models reason about medical questions? arXiv preprint arXiv:2207.08143. [find](https://scholar.google.com/scholar?q=Valentin+Li%C3%A9vin%2C+Christoffer+Egeberg+Hother%2C+and+Ole+Winther%2E+2022%2E+Can+large+language+models+reason+about+medical+questions%3F+arXiv+preprint+arXiv%3A2207%2E08143%2E)
16. Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. 2022. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 23(6). [find](https://scholar.google.com/scholar?q=Renqian+Luo%2C+Liai+Sun%2C+Yingce+Xia%2C+Tao+Qin%2C+Sheng+Zhang%2C+Hoifung+Poon%2C+and+Tie%2DYan+Liu%2E+2022%2E+Biogpt%3A+generative+pre%2Dtrained+transformer+for+biomedical+text+generation+and+mining%2E+Briefings+in+Bioinfo)
17. Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. 2023. [Augmented language models: a survey](https://md.research.elamur.ai/free/en/papers/mialon-2023-augmented-language-models-survey/). arXiv preprint arXiv:2302.07842.
18. Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023. [Capabilities of gpt-4 on medical challenge problems](https://md.research.elamur.ai/free/en/papers/nori-2023-capabilities-gpt-4-medical-challenge/). arXiv preprint arXiv:2303.13375.
19. OpenAI. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). abs/2303.08774.
20. CoRR, Aaron Parisi, Yao Zhao, and Noah Fiedel. 2022. [Talm: Tool augmented language models](https://md.research.elamur.ai/free/en/papers/parisi-2022-talm-tool-augmented-language-models/). arXiv preprint arXiv:2205.12255.
21. Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, et al. 2023. [Tool learning with foundation models](https://md.research.elamur.ai/free/en/papers/qin-2023-tool-learning-foundation-models/). arXiv preprint arXiv:2304.08354.
22. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+Ilya+Sutskever%2C+et+al%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
23. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
24. Eric W Sayers, Richa Agarwala, Evan E Bolton, J Rodney Brister, Kathi Canese, Karen Clark, Ryan Connor, Nicolas Fiorini, Kathryn Funk, Timothy Hef-feron, et al. 2019. Database resources of the national center for biotechnology information. Nucleic acids research, 47(Database issue):D23. [find](https://scholar.google.com/scholar?q=Eric+W+Sayers%2C+Richa+Agarwala%2C+Evan+E+Bolton%2C+J+Rodney+Brister%2C+Kathi+Canese%2C+Karen+Clark%2C+Ryan+Connor%2C+Nicolas+Fiorini%2C+Kathryn+Funk%2C+Timothy+Hef%2Dferon%2C+et+al%2E+2019%2E+Database+resources+of+the+nationa)
25. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. [Toolformer: Language models can teach themselves to use tools](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/). arXiv preprint arXiv:2302.04761.
26. GD Schuler, JA Epstein, H Ohkawa, and JA Kans. 1996. Entrez: molecular biology database and retrieval system. Methods in enzymology, 266:141– 162. [find](https://scholar.google.com/scholar?q=GD+Schuler%2C+JA+Epstein%2C+H+Ohkawa%2C+and+JA+Kans%2E+1996%2E+Entrez%3A+molecular+biology+database+and+retrieval+system%2E+Methods+in+enzymology%2C+266%3A141%E2%80%93+162%2E)
27. Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2022. [Large language models encode clinical knowledge](https://md.research.elamur.ai/free/en/papers/singhal-2022-large-language-models-encode-clinical/). arXiv preprint arXiv:2212.13138.
28. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022a. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). arXiv preprint arXiv:2206.07682.
29. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022b. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). arXiv preprint arXiv:2201.11903.

*Updated 24 October 2026*