---
title: "LAB-Bench — paper by Laurent et al. (2024)"
description: "One useful technique: the authors had models propose questions, then manually fixed the… 57 references, cited by 43 papers here."
authors: ["Jon M. Laurent", "Joseph D. Janizek", "Michael Ruzo", "Michaela M. Hinks", "Michael J. Hammerling", "Siddharth Narayanan", "Manvitha Ponnapati", "Andrew D. White", "Samuel G. Rodriques"]
year: 2024
venue: "arXiv:2407.10362"
source: "https://arxiv.org/abs/2407.10362"
doi: "10.48550/arxiv.2407.10362"
arxiv: "2407.10362"
licence: "CC BY-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/laurent-2024-lab-bench-measuring-capabilities/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/laurent-2024-lab-bench-measuring-capabilities/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) / LAB-Bench: Measuring Capabilities of Language Models for Biology Research

# LAB-Bench: Measuring Capabilities of Language Models for Biology Research

**Authors:** Jon M. Laurent, Joseph D. Janizek, Michael Ruzo, Michaela M. Hinks, Michael J. Hammerling, Siddharth Narayanan, Manvitha Ponnapati, [Andrew D. White](https://md.research.elamur.ai/free/en/authors/andrew-d-white/) and [Samuel G. Rodriques](https://md.research.elamur.ai/free/en/authors/samuel-g-rodriques/) · **Year:** 2024 · **Venue:** arXiv:2407.10362 · **Licence:** [CC BY-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2407.10362) · [PDF](https://arxiv.org/pdf/2407.10362) · [DOI 10.48550/arxiv.2407.10362](https://doi.org/10.48550/arxiv.2407.10362) · [arXiv 2407.10362](https://arxiv.org/abs/2407.10362) · [OpenAlex](https://openalex.org/W4402345509)

**Citations:** 19 citations (OpenAlex, 2026-10-14) · [43](https://md.research.elamur.ai/free/en/papers/laurent-2024-lab-bench-measuring-capabilities/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most science benchmarks for LLMs test textbook-style recall, not the practical work researchers do every day. LAB-Bench, from FutureHouse, fills this gap for biology. It contains more than 2,400 multiple-choice questions covering:
- literature retrieval and reasoning
- reading figures and tables
- querying biological databases
- troubleshooting lab protocols
- working with DNA and protein sequences

Experts wrote the harder categories by hand. Sequence and database items were mostly generated programmatically. One useful technique: the authors had models propose questions, then manually fixed the nonsensical ones, which often produced items the models still failed. Frontier commercial and open-source models were tested through APIs without tools and compared against PhD-level biologists. The section with detailed scores is missing from the provided excerpt. The authors stress that plausible distractors matter, because stronger models can score well by eliminating weak options and guessing rather than reasoning. The benchmark matters because consistently high scores, especially on the hardest cloning tasks, would indicate a model that could genuinely assist researchers.

## Key points
- **Eight task families:** LitQA2, SuppQA, FigQA, TableQA, DbQA, ProtocolQA, SeqQA and CloningScenarios. LitQA2, SuppQA, DbQA and SeqQA are tool-dependent, partly as RAG tests. FigQA, TableQA and ProtocolQA are tool-free.
- **CloningScenarios:** 41 "human-hard" multi-step problems that take an expert roughly 10–60 minutes each. This made human baselines hard to collect, so the authors suggest future "human proofs of possibility" instead.
- **Public/private split:** about 80% of each subtask is public on Hugging Face (`futurehouse/lab-bench`). The other 20% is held back to detect training-data contamination, and the dataset includes a canary string.
- **Uneven comparison:** models ran without tools such as web search, while human evaluators were allowed to use tools.
- **Reproducibility:** the authors release their exact prompts and code, and they validated multiple-choice results with open-answer grading.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Future House, Inc.](https://md.research.elamur.ai/free/en/ngos/future-house-inc/) — Nonprofit, San Francisco, United States · 4 works · 548 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🧬 Biochemistry, Genetics and Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/) · 263 works · [🧬 Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/molecular-biology/) · 260 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Jon M. Laurent, Joseph D. Janizek, Michael Ruzo, Michaela M. Hinks, Michael J. Hammerling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D. White and Samuel G. Rodriques. 2024. LAB-Bench: Measuring Capabilities of Language Models for Biology Research. arXiv:2407.10362.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{laurent2024lab,
  title = {{LAB-Bench: Measuring Capabilities of Language Models for Biology Research}},
  author = {Jon M. Laurent, Joseph D. Janizek, Michael Ruzo, Michaela M. Hinks, Michael J. Hammerling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D. White and Samuel G. Rodriques},
  year = {2024},
  doi = {10.48550/arxiv.2407.10362},
  eprint = {2407.10362},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.10362}
}
```

- [← previous Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens](https://md.research.elamur.ai/free/en/papers/liu-2024-infini-gram-scaling-unbounded-n/)
- [next → Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding](https://md.research.elamur.ai/free/en/papers/suzgun-2024-meta-prompting-enhancing-language-models/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/)

## References

57 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Joanna S Amberger, Carol A Bocchini, François Schiettecatte, Alan F Scott, and Ada Hamosh. Omim. org: Online mendelian inheritance in man (omim®), an online catalog of human genes and genetic disorders. Nucleic acids research, 43(D1):D789–D798, 2015. [find](https://scholar.google.com/scholar?q=Joanna+S+Amberger%2C+Carol+A+Bocchini%2C+Fran%C3%A7ois+Schiettecatte%2C+Alan+F+Scott%2C+and+Ada+Hamosh%2E+Omim%2E+org%3A+Online+mendelian+inheritance+in+man+%28omim%C2%AE%29%2C+an+online+catalog+of+human+genes+and+genetic+disorder)
2. Anthropic. Introducing the next generation of claude, March 2024. URL [https://www.anthropic.com/news/claude-3-family](https://www.anthropic.com/news/claude-3-family). Accessed: 2024-06-11. [↗](https://www.anthropic.com/news/claude-3-family)
3. Anthropic. Introducing the next generation of claude, March 2024. URL [https://www.anthropic.com/news/claude-3-5-sonnet](https://www.anthropic.com/news/claude-3-5-sonnet). Accessed: 2024-07-11. [↗](https://www.anthropic.com/news/claude-3-5-sonnet)
4. Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, et al. Lessons from the trenches on reproducible evaluation of language models. arXiv preprint arXiv:2405.14782, 2024. [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Lintang+Sutawika%2C+Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Alham+Fikri+Aji%2C+Pawan+Sasanka+Ammanamanchi%2C+Sidney+Black%2C+Jordan+Clive%2C+et+al%2E+Lessons+from+the+trenches+on+)
5. Daniil A. Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. [Autonomous chemical research with large language models](https://md.research.elamur.ai/free/en/papers/boiko-2023-autonomous-chemical-research-large-language/). Nature, 624(7992):570–578, Dec 2023. ISSN 1476-4687. doi: 10.1038/s41586-023-06792-0. URL https://doi.org/10.1038/ s41586-023-06792-0.
6. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora De… — [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
7. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/), 2023.
8. Yuhao Chen and Xiaowei Wang. mirdb: an online database for prediction of functional microrna targets. Nucleic acids research, 48(D1):D127–D131, 2020. [find](https://scholar.google.com/scholar?q=Yuhao+Chen+and+Xiaowei+Wang%2E+mirdb%3A+an+online+database+for+prediction+of+functional+microrna+targets%2E+Nucleic+acids+research%2C+48%28D1%29%3AD127%E2%80%93D131%2C+2020%2E)
9. Peter J. A. Cock, Tiago Antao, Jeffrey T. Chang, Brad A. Chapman, Cymon J. Cox, Andrew Dalke, Iddo Friedberg, Thomas Hamelryck, Frank Kauff, Bartek Wilczynski, and Michiel J. L. de Hoon. Biopython: freely available Python tools for computational molecular biology and bioinformatics. Bioinformatics,… — [doi:10.1093/bioinformatics/btp163](https://doi.org/10.1093/bioinformatics/btp163) [find](https://scholar.google.com/scholar?q=Peter+J%2E+A%2E+Cock%2C+Tiago+Antao%2C+Jeffrey+T%2E+Chang%2C+Brad+A%2E+Chapman%2C+Cymon+J%2E+Cox%2C+Andrew+Dalke%2C+Iddo+Friedberg%2C+Thomas+Hamelryck%2C+Frank+Kauff%2C+Bartek+Wilczynski%2C+and+Michiel+J%2E+L%2E+de+Hoon%2E+Biopython%3A+fr)
10. M. Wayne Davis and Erik M. Jorgensen. Ape, a plasmid editor: A freely available dna manipulation and visualization program. Frontiers in Bioinformatics, 2, 2022. ISSN 2673-7647. doi: 10.3389/fbinf.2022.818619. URL [https://www.frontiersin.org/articles/](https://www.frontiersin.org/articles/) 10.3389/fbinf.2022.818619. — [doi:10.3389/fbinf.2022.818619](https://doi.org/10.3389/fbinf.2022.818619) [↗](https://www.frontiersin.org/articles/)
11. Clémentine Fourrier, Najoung Habib, Julien Launay, and Thomas Wolf. What’s going on with the open llm leaderboard? Hugging Face, June 2023. URL [https://huggingface.co/](https://huggingface.co/) blog/open-llm-leaderboard-mmlu. Introducing gemini 1.5, google’s next-generation ai model, [↗](https://huggingface.co/)
12. Google. February 2024. URL [https://blog.google/technology/ai/](https://blog.google/technology/ai/) google-gemini-next-generation-model-february-2024/. Accessed: 2024-06-11. [↗](https://blog.google/technology/ai/)
13. Peter W Harrison, M Ridwan Amode, Olanrewaju Austine-Orimoloye, Andrey G Azov, Matthieu Barba, If Barnes, Arne Becker, Ruth Bennett, Andrew Berry, Jyothish Bhai, et al. Ensembl 2024. Nucleic Acids Research, 52(D1):D891–D899, 2024. [find](https://scholar.google.com/scholar?q=Peter+W+Harrison%2C+M+Ridwan+Amode%2C+Olanrewaju+Austine%2DOrimoloye%2C+Andrey+G+Azov%2C+Matthieu+Barba%2C+If+Barnes%2C+Arne+Becker%2C+Ruth+Bennett%2C+Andrew+Berry%2C+Jyothish+Bhai%2C+et+al%2E+Ensembl+2024%2E+Nucleic+Acids+Res)
14. Kilian Hendrickx, Lorenzo Perini, Dries Van der Plas, Wannes Meert, and Jesse Davis. Machine learning with a reject option: A survey. Machine Learning, 113(5):3073–3110, 2024. [find](https://scholar.google.com/scholar?q=Kilian+Hendrickx%2C+Lorenzo+Perini%2C+Dries+Van+der+Plas%2C+Wannes+Meert%2C+and+Jesse+Davis%2E+Machine+learning+with+a+reject+option%3A+A+survey%2E+Machine+Learning%2C+113%285%29%3A3073%E2%80%933110%2C+2024%2E)
15. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/), 2021.
16. Wenpin Hou and Zhicheng Ji. Geneturing tests gpt models in genomics. BioRxiv, 2023. [find](https://scholar.google.com/scholar?q=Wenpin+Hou+and+Zhicheng+Ji%2E+Geneturing+tests+gpt+models+in+genomics%2E+BioRxiv%2C+2023%2E)
17. Kevin Maik Jablonka, Philippe Schwaller, Andres Ortega-Guerrero, and Berend Smit. Leveraging large language models for predictive chemistry. Nat. Mach. Intell., 6(2):161–169, February 2024. [find](https://scholar.google.com/scholar?q=Kevin+Maik+Jablonka%2C+Philippe+Schwaller%2C+Andres+Ortega%2DGuerrero%2C+and+Berend+Smit%2E+Leveraging+large+language+models+for+predictive+chemistry%2E+Nat%2E+Mach%2E+Intell%2E%2C+6%282%29%3A161%E2%80%93169%2C+February+2024%2E)
18. Samy Jelassi, David Brandfonbrener, Sham M. Kakade, and Eran Malach. Repeat after me: Transformers are better than state space models at copying, 2024. [find](https://scholar.google.com/scholar?q=Samy+Jelassi%2C+David+Brandfonbrener%2C+Sham+M%2E+Kakade%2C+and+Eran+Malach%2E+Repeat+after+me%3A+Transformers+are+better+than+state+space+models+at+copying%2C+2024%2E)
19. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. [Swe-bench: Can language models resolve real-world github issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/), 2024.
20. Qiao Jin, Yifan Yang, Qingyu Chen, and Zhiyong Lu. [GeneGPT: augmenting large language models with domain tools for improved access to biomedical information](https://md.research.elamur.ai/free/en/papers/jin-2023-genegpt-augmenting-large-language-models/). Bioinformatics, 40 (2):btae075, 02 2024. ISSN 1367-4811. doi: 10.1093/bioinformatics/btae075. URL https://doi.org/10.1093/bioinformatics/btae… [↗](https://doi.org/10.1093/bioinformatics/btae…)
21. John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A A Kohl, Andrew J Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain,… — [Highly accurate protein structure prediction with AlphaFold](https://md.research.elamur.ai/free/en/papers/jumper-2021-highly-accurate-protein-structure-prediction/)
22. Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. GPT-4 passes the bar exam. SSRN Electron. J., 2023. [find](https://scholar.google.com/scholar?q=Daniel+Martin+Katz%2C+Michael+James+Bommarito%2C+Shang+Gao%2C+and+Pablo+Arredondo%2E+GPT%2D4+passes+the+bar+exam%2E+SSRN+Electron%2E+J%2E%2C+2023%2E)
23. Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliouras. [BioASQ-QA: A manually curated corpus for biomedical question answering](https://md.research.elamur.ai/free/en/papers/krithara-2023-bioasq-qa-manually-curated-corpus/). Sci. Data, 10(1): 170, March 2023.
24. Maxim V Kuleshov, Matthew R Jones, Andrew D Rouillard, Nicolas F Fernandez, Qiaonan Duan, Zichen Wang, Simon Koplev, Sherry L Jenkins, Kathleen M Jagodnik, Alexander Lachmann, et al. Enrichr: a comprehensive gene set enrichment analysis web server 2016 update. Nucleic acids research, 44(W1):W90–W97,… [find](https://scholar.google.com/scholar?q=Maxim+V+Kuleshov%2C+Matthew+R+Jones%2C+Andrew+D+Rouillard%2C+Nicolas+F+Fernandez%2C+Qiaonan+Duan%2C+Zichen+Wang%2C+Simon+Koplev%2C+Sherry+L+Jenkins%2C+Kathleen+M+Jagodnik%2C+Alexander+Lachmann%2C+et+al%2E+Enrichr%3A+a+compre)
25. Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, and Victor Tseng. [Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models](https://md.research.elamur.ai/free/en/papers/kung-2023-performance-chatgpt-usmle-potential-ai/). PLOS Digi…
26. Melissa J Landrum, Jennifer M Lee, Mark Benson, Garth Brown, Chen Chao, Shanmuga Chitipiralla, Baoshan Gu, Jennifer Hart, Douglas Hoffman, Jeffrey Hoover, et al. Clinvar: public archive of interpretations of clinically relevant variants. Nucleic acids research, 44(D1): D862–D868, 2016. [find](https://scholar.google.com/scholar?q=Melissa+J+Landrum%2C+Jennifer+M+Lee%2C+Mark+Benson%2C+Garth+Brown%2C+Chen+Chao%2C+Shanmuga+Chitipiralla%2C+Baoshan+Gu%2C+Jennifer+Hart%2C+Douglas+Hoffman%2C+Jeffrey+Hoover%2C+et+al%2E+Clinvar%3A+public+archive+of+interpretat)
27. Gorka Lasso, Sandra V Mayer, Evandro R Winkelmann, Tim Chu, Oliver Elliot, Juan Angel Patino-Galindo, Kernyu Park, Raul Rabadan, Barry Honig, and Sagi D Shapira. A structure-informed atlas of human-virus interactions. Cell, 178(6):1526–1541, 2019. [find](https://scholar.google.com/scholar?q=Gorka+Lasso%2C+Sandra+V+Mayer%2C+Evandro+R+Winkelmann%2C+Tim+Chu%2C+Oliver+Elliot%2C+Juan+Angel+Patino%2DGalindo%2C+Kernyu+Park%2C+Raul+Rabadan%2C+Barry+Honig%2C+and+Sagi+D+Shapira%2E+A+structure%2Dinformed+atlas+of+human%2Dvi)
28. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D Manning, Christopher Ré, Diana Acosta-Navas, Drew A Hudson, Eric Ze… [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+Benjamin+Newman%2C+Binhang+Yuan%2C+Bobby+Yan%2C+Ce+Zhang%2C+C)
29. Arthur Liberzon, Aravind Subramanian, Reid Pinchback, Helga Thorvaldsdóttir, Pablo Tamayo, and Jill P Mesirov. Molecular signatures database (msigdb) 3.0. Bioinformatics, 27(12): 1739–1740, 2011. [find](https://scholar.google.com/scholar?q=Arthur+Liberzon%2C+Aravind+Subramanian%2C+Reid+Pinchback%2C+Helga+Thorvaldsd%C3%B3ttir%2C+Pablo+Tamayo%2C+and+Jill+P+Mesirov%2E+Molecular+signatures+database+%28msigdb%29+3%2E0%2E+Bioinformatics%2C+27%2812%29%3A+1739%E2%80%931740%2C+2011%2E)
30. Jakub Lála, Odhran O’Donoghue, Aleksandar Shtedritski, Sam Cox, Samuel G. Rodriques, and Andrew D. White. [Paperqa: Retrieval-augmented generative agent for scientific research](https://md.research.elamur.ai/free/en/papers/lala-2023-paperqa-retrieval-augmented-generative/), 2023.
31. Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D. White, and Philippe Schwaller. Augmenting large language models with chemistry tools. Nature Machine Intelligence, 6(5):525–535, May 2024. ISSN 2522-5839. doi: 10.1038/s42256-024-00832-8. URL https://doi.org/10.1038/s42256-024-008… — [doi:10.1038/s42256-024-00832-8](https://doi.org/10.1038/s42256-024-00832-8) [↗](https://doi.org/10.1038/s42256-024-008…)
32. Amil Merchant, Simon Batzner, Samuel S Schoenholz, Muratahan Aykol, Gowoon Cheon, and Ekin Dogus Cubuk. Scaling deep learning for materials discovery. Nature, 624(7990):80–85, December 2023. [find](https://scholar.google.com/scholar?q=Amil+Merchant%2C+Simon+Batzner%2C+Samuel+S+Schoenholz%2C+Muratahan+Aykol%2C+Gowoon+Cheon%2C+and+Ekin+Dogus+Cubuk%2E+Scaling+deep+learning+for+materials+discovery%2E+Nature%2C+624%287990%29%3A80%E2%80%9385%2C+December+2023%2E)
33. Meta. Chat: meta-llama/meta-llama-3-70b-instruct, April 2024. URL [https://docs.anyscale.com/endpoints/text-generation/supported-models/](https://docs.anyscale.com/endpoints/text-generation/supported-models/) meta-llama-Meta-Llama-3-70B-Instruct. Accessed: 2024-06-11. [↗](https://docs.anyscale.com/endpoints/text-generation/supported-models/)
34. Meta. Introducing meta llama 3: The most capable openly available llm to date, April 2024. URL [https://ai.meta.com/blog/meta-llama-3/](https://ai.meta.com/blog/meta-llama-3/). Accessed: 2024-06-11. [↗](https://ai.meta.com/blog/meta-llama-3/)
35. Adrian Mirza, Nawaf Alampara, Sreekanth Kunchapu, Benedict Emoekabu, Aswanth Krishnan, Mara Wilhelmi, Macjonathan Okereke, Juliane Eberhardt, Amir Mohammad Elahi, Maximilian Greiner, Caroline T. Holick, Tanya Gupta, Mehrdad Asgari, Christina Glaubitz, Lea C. Klepsch, Yannik Köster, Jakob Meyer, Sant… [find](https://scholar.google.com/scholar?q=Adrian+Mirza%2C+Nawaf+Alampara%2C+Sreekanth+Kunchapu%2C+Benedict+Emoekabu%2C+Aswanth+Krishnan%2C+Mara+Wilhelmi%2C+Macjonathan+Okereke%2C+Juliane+Eberhardt%2C+Amir+Mohammad+Elahi%2C+Maximilian+Greiner%2C+Caroline+T%2E+Holic)
36. Eric Nguyen, Michael Poli, Matthew G. Durrant, Armin W. Thomas, Brian Kang, Jeremy Sullivan, Madelena Y. Ng, Ashley Lewis, Aman Patel, Aaron Lou, Stefano Ermon, Stephen A. Baccus, Tina Hernandez-Boussard, Christopher Ré, Patrick D. Hsu, and Brian L. Hie. Sequence modeling and design from molecular t… [find](https://scholar.google.com/scholar?q=Eric+Nguyen%2C+Michael+Poli%2C+Matthew+G%2E+Durrant%2C+Armin+W%2E+Thomas%2C+Brian+Kang%2C+Jeremy+Sullivan%2C+Madelena+Y%2E+Ng%2C+Ashley+Lewis%2C+Aman+Patel%2C+Aaron+Lou%2C+Stefano+Ermon%2C+Stephen+A%2E+Baccus%2C+Tina+Hernandez%2DBouss)
37. Josh M. Nicholson, Milo Mordaunt, Patrice Lopez, Ashish Uppala, Domenic Rosati, Neves P. Rodrigues, Peter Grabitz, and Sean C. Rife. scite: A smart citation index that displays the context of citations and classifies their intent using deep learning. Quantitative Science Studies, 2(3):882–898, 11 20… — [doi:10.1162/qss_a_00146](https://doi.org/10.1162/qss_a_00146) [find](https://scholar.google.com/scholar?q=Josh+M%2E+Nicholson%2C+Milo+Mordaunt%2C+Patrice+Lopez%2C+Ashish+Uppala%2C+Domenic+Rosati%2C+Neves+P%2E+Rodrigues%2C+Peter+Grabitz%2C+and+Sean+C%2E+Rife%2E+scite%3A+A+smart+citation+index+that+displays+the+context+of+citation)
38. Pascal Notin, Aaron Kollasch, Daniel Ritter, Lood van Niekerk, Steffanie Paul, Han Spinner, Nathan Rollins, Ada Shaw, Rose Orenbuch, Ruben Weitzman, Jonathan Frazer, Mafalda Dias, Dinko Franceschi, Yarin Gal, and Debora Marks. Proteingym: Large-scale benchmarks for protein fitness prediction and des… [find](https://scholar.google.com/scholar?q=Pascal+Notin%2C+Aaron+Kollasch%2C+Daniel+Ritter%2C+Lood+van+Niekerk%2C+Steffanie+Paul%2C+Han+Spinner%2C+Nathan+Rollins%2C+Ada+Shaw%2C+Rose+Orenbuch%2C+Ruben+Weitzman%2C+Jonathan+Frazer%2C+Mafalda+Dias%2C+Dinko+Franceschi%2C+Ya)
39. OpenAI. Hello gpt-4o, May 2024. URL [https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/). Accessed: 2024-06-11. [↗](https://openai.com/index/hello-gpt-4o/)
40. OpenAI. Gpt-4 and gpt-4 turbo, 2024. URL [https://platform.openai.com/docs/](https://platform.openai.com/docs/) models/gpt-4-and-gpt-4-turbo. Accessed: 2024-06-11. [↗](https://platform.openai.com/docs/)
41. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… — [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
42. Janet Piñero, Àlex Bravo, Núria Queralt-Rosinach, Alba Gutiérrez-Sacristán, Jordi Deu-Pons, Emilio Centeno, Javier García-García, Ferran Sanz, and Laura I Furlong. Disgenet: a comprehensive platform integrating information on human disease-associated genes and variants. Nucleic acids research, page… [find](https://scholar.google.com/scholar?q=Janet+Pi%C3%B1ero%2C+%C3%80lex+Bravo%2C+N%C3%BAria+Queralt%2DRosinach%2C+Alba+Guti%C3%A9rrez%2DSacrist%C3%A1n%2C+Jordi+Deu%2DPons%2C+Emilio+Centeno%2C+Javier+Garc%C3%ADa%2DGarc%C3%ADa%2C+Ferran+Sanz%2C+and+Laura+I+Furlong%2E+Disgenet%3A+a+comprehensive+platform+i)
43. Russell A Poldrack, Thomas Lu, and Gašper Beguš. Ai-assisted coding: Experiments with gpt-4, 2023. [find](https://scholar.google.com/scholar?q=Russell+A+Poldrack%2C+Thomas+Lu%2C+and+Ga%C5%A1per+Begu%C5%A1%2E+Ai%2Dassisted+coding%3A+Experiments+with+gpt%2D4%2C+2023%2E)
44. Sue Povey, Ruth Lovering, Elspeth Bruford, Mathew Wright, Michael Lush, and Hester Wain. The hugo gene nomenclature committee (hgnc). Human genetics, 109:678–680, 2001. [find](https://scholar.google.com/scholar?q=Sue+Povey%2C+Ruth+Lovering%2C+Elspeth+Bruford%2C+Mathew+Wright%2C+Michael+Lush%2C+and+Hester+Wain%2E+The+hugo+gene+nomenclature+committee+%28hgnc%29%2E+Human+genetics%2C+109%3A678%E2%80%93680%2C+2001%2E)
45. David Rein. Can good benchmarks contain mistakes? [https://wp.nyu.edu/arg/](https://wp.nyu.edu/arg/) can-good-benchmarks-contain-mistakes/, 2024. Accessed: 2024-05-20. [↗](https://wp.nyu.edu/arg/)
46. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. [Gpqa: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/), 2023. URL https://arxiv.org/abs/2311.12022. [↗](https://arxiv.org/abs/2311.12022)
47. Varuni Sarwal, Viorel Munteanu, Timur Suhodolschi, Dumitru Ciorba, Eleazar Eskin, Wei Wang, and Serghei Mangul. Biollmbench: A comprehensive benchmarking of large language models in bioinformatics. bioRxiv, 2023. doi: 10.1101/2023.12.19.572483. URL [https://www.biorxiv.org/content/early/2023/12/20/2…](https://www.biorxiv.org/content/early/2023/12/20/2…) — [doi:10.1101/2023.12.19.572483](https://doi.org/10.1101/2023.12.19.572483) [↗](https://www.biorxiv.org/content/early/2023/12/20/2…)
48. Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansfield, Dina Demner-Fushma… — [Large Language Models Encode Clinical Knowledge](https://md.research.elamur.ai/free/en/papers/singhal-2022-large-language-models-encode-clinical/)
49. Cynthia L Smith and Janan T Eppig. The mammalian phenotype ontology: enabling robust annotation and comparative analysis. Wiley Interdisciplinary Reviews: Systems Biology and Medicine, 1(3):390–399, 2009. [find](https://scholar.google.com/scholar?q=Cynthia+L+Smith+and+Janan+T+Eppig%2E+The+mammalian+phenotype+ontology%3A+enabling+robust+annotation+and+comparative+analysis%2E+Wiley+Interdisciplinary+Reviews%3A+Systems+Biology+and+Medicine%2C+1%283%29%3A390%E2%80%93399%2C+2)
50. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W Kocurek, Ali Safaya, Ali Tazarv, Alice X… — [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
51. Felix Strieth-Kalthoff, Han Hao, Vandana Rathore, Joshua Derasp, Théophile Gaudin, Nicholas H Angello, Martin Seifrid, Ekaterina Trushina, Mason Guy, Junliang Liu, Xun Tang, Masashi Mamada, Wesley Wang, Tuul Tsagaantsooj, Cyrille Lavigne, Robert Pollice, Tony C Wu, Kazuhiro Hotta, Leticia Bodo, Shan… [find](https://scholar.google.com/scholar?q=Felix+Strieth%2DKalthoff%2C+Han+Hao%2C+Vandana+Rathore%2C+Joshua+Derasp%2C+Th%C3%A9ophile+Gaudin%2C+Nicholas+H+Angello%2C+Martin+Seifrid%2C+Ekaterina+Trushina%2C+Mason+Guy%2C+Junliang+Liu%2C+Xun+Tang%2C+Masashi+Mamada%2C+Wesley+Wan)
52. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. [Galactica: A large language model for science](https://md.research.elamur.ai/free/en/papers/taylor-2022-galactica-large-language-model-science/). arXiv, 2022.
53. Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eisenach, Woody Ahern, Andrew J Borst, Robert J Ragotte, Lukas F Milles, Basile I M Wicky, Nikita Hanikel, Samuel J Pellock, Alexis Courbet, William Sheffler, Jue Wang, Preetham Venkatesh, Isaac Sappington, Susa… [find](https://scholar.google.com/scholar?q=Joseph+L+Watson%2C+David+Juergens%2C+Nathaniel+R+Bennett%2C+Brian+L+Trippe%2C+Jason+Yim%2C+Helen+E+Eisenach%2C+Woody+Ahern%2C+Andrew+J+Borst%2C+Robert+J+Ragotte%2C+Lukas+F+Milles%2C+Basile+I+M+Wicky%2C+Nikita+Hanikel%2C+Samu)
54. Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306.13063, 2023. [find](https://scholar.google.com/scholar?q=Miao+Xiong%2C+Zhiyuan+Hu%2C+Xinyang+Lu%2C+Yifei+Li%2C+Jie+Fu%2C+Junxian+He%2C+and+Bryan+Hooi%2E+Can+llms+express+their+uncertainty%3F+an+empirical+evaluation+of+confidence+elicitation+in+llms%2E+arXiv+preprint+arXiv%3A23)
55. Ivan Yevshin, Ruslan Sharipov, Tagir Valeev, Alexander Kel, and Fedor Kolpakov. Gtrd: a database of transcription factor binding sites identified by chip-seq experiments. Nucleic acids research, page gkw951, 2016. [find](https://scholar.google.com/scholar?q=Ivan+Yevshin%2C+Ruslan+Sharipov%2C+Tagir+Valeev%2C+Alexander+Kel%2C+and+Fedor+Kolpakov%2E+Gtrd%3A+a+database+of+transcription+factor+binding+sites+identified+by+chip%2Dseq+experiments%2E+Nucleic+acids+research%2C+page+)
56. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=Chujie+Zheng%2C+Hao+Zhou%2C+Fandong+Meng%2C+Jie+Zhou%2C+and+Minlie+Huang%2E+Large+language+models+are+not+robust+multiple+choice+selectors%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2C+2)
57. Zhiqiang Zhong, Kuangyu Zhou, and Davide Mottin. Benchmarking large language models for molecule prediction tasks, 2024. [find](https://scholar.google.com/scholar?q=Zhiqiang+Zhong%2C+Kuangyu+Zhou%2C+and+Davide+Mottin%2E+Benchmarking+large+language+models+for+molecule+prediction+tasks%2C+2024%2E)

*Updated 2 November 2026*