---
title: "CURIE — paper by Cui et al. (2025) — summary and ideas"
description: "Most science benchmarks for LLMs, such as GPQA or PubmedQA, use short questions with… 67 references, cited by 14 papers here."
authors: ["Hao Cui", "Zahra Shamsi", "Gowoon Cheon", "Xuejian Ma", "Shutong Li", "Maria Tikhanovskaya", "Peter Norgaard", "Nayantara Mudur", "Martyna Plomecka", "Paul Raccuglia", "Yasaman Bahri", "Victor V. Albert", "Pranesh Srinivasan", "Haining Pan", "Philippe Faist", "Brian Rohr", "Ekin Dogus Cubuk", "Muratahan Aykol", "Amil Merchant", "Michael J. Statt", "Dan Morris", "Drew Purves", "Elise Kleeman", "Ruth Alcantara", "Matthew Abraham", "Muqthar Mohammad", "Ean Phing VanLee", "Chenfei Jiang", "Elizabeth Dorfman", "Eun-Ah Kim", "Michael P Brenner", "Viren Jain", "Sameera Ponda", "Subhashini Venugopalan"]
year: 2025
venue: "arXiv:2503.13517"
source: "https://arxiv.org/abs/2503.13517"
doi: "10.48550/arxiv.2503.13517"
arxiv: "2503.13517"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/cui-2025-curie-evaluating-llms-multitask-scientific/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cui-2025-curie-evaluating-llms-multitask-scientific/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/6/) / CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoning

# CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoning

**Authors:** [Hao Cui](https://md.research.elamur.ai/free/en/authors/hao-cui/), [Zahra Shamsi](https://md.research.elamur.ai/free/en/authors/zahra-shamsi/), Gowoon Cheon, Xuejian Ma, Shutong Li, Maria Tikhanovskaya, [Peter Norgaard](https://md.research.elamur.ai/free/en/authors/peter-norgaard/), Nayantara Mudur, Martyna Plomecka, Paul Raccuglia, Yasaman Bahri, Victor V. Albert, Pranesh Srinivasan, Haining Pan, Philippe Faist, Brian Rohr, Ekin Dogus Cubuk, Muratahan Aykol, Amil Merchant, Michael J. Statt, Dan Morris, Drew Purves, Elise Kleeman, Ruth Alcantara, Matthew Abraham, Muqthar Mohammad, Ean Phing VanLee, Chenfei Jiang, Elizabeth Dorfman, Eun-Ah Kim, Michael P Brenner, Viren Jain, Sameera Ponda and Subhashini Venugopalan · **Year:** 2025 · **Venue:** arXiv:2503.13517 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2503.13517) · [PDF](https://arxiv.org/pdf/2503.13517) · [DOI 10.48550/arxiv.2503.13517](https://doi.org/10.48550/arxiv.2503.13517) · [arXiv 2503.13517](https://arxiv.org/abs/2503.13517) · [OpenAlex](https://openalex.org/W6966931784)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [14](https://md.research.elamur.ai/free/en/papers/cui-2025-curie-evaluating-llms-multitask-scientific/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most science benchmarks for LLMs, such as GPQA or PubmedQA, use short questions with multiple-choice answers. Real research work is different: scientists read whole papers and pull information from them. CURIE addresses this gap with a benchmark built on full-length scientific documents. PhD-level experts created 580 problem–solution pairs from 429 research papers. These are organized into ten tasks spanning six disciplines: proteins, biodiversity, analysis of geospatial data, quantum computation, the physics of condensed matter, and the study of materials.

The tasks test extraction, concept tracking, aggregation, algebraic derivation, map and figure comprehension, and cross-domain skills such as writing code for DFT calculations. Input contexts average roughly 15k words, and expected answers are long and heterogeneous.

The authors evaluated eight open and closed models with context windows of at least 32k tokens. Gemini Flash 2.0 and Claude-3 were consistently strong, and Command-R+ led the open models. Even so, the top score was only about 32%. The benchmark shows that current LLMs remain far from dependable assistants for long-context scientific work.

## Key points
- **Scale and sources:** 10 tasks, 580 examples, 429 papers, 6 disciplines. Input contexts average about 15k words and outputs about 1k words. Code and data are at github.com/google/curie.
- **Hardest tasks:** The materials tasks were the toughest for every model. They require exhaustively finding and combining values scattered throughout a paper.
- **Protein sequencing failures:** GPT-4o collapsed on protein sequence reconstruction (PDB) by repeating subsequences and never stopping. Gemini Flash 2.0 instead produced working Python code in about half of the cases.
- **PDB task setup:** 64 proteins from 15 species. Scoring uses Biopython pairwise alignment, with the identity ratio as the metric. Accuracy declines as sequences get longer.
- **Common protein errors:** Models skipped residues, undercounted repeated amino acids, and misapplied three-to-one letter codes.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Hao Cui, Zahra Shamsi, Gowoon Cheon, Xuejian Ma, Shutong Li, Maria Tikhanovskaya, Peter Norgaard, Nayantara Mudur, Martyna Plomecka, Paul Raccuglia, Yasaman Bahri, Victor V. Albert, Pranesh Srinivasan, Haining Pan, Philippe Faist, Brian Rohr, Ekin Dogus Cubuk, Muratahan Aykol, Amil Merchant, Michael J. Statt, Dan Morris, Drew Purves, Elise Kleeman, Ruth Alcantara, Matthew Abraham, Muqthar Mohammad, Ean Phing VanLee, Chenfei Jiang, Elizabeth Dorfman, Eun-Ah Kim, Michael P Brenner, Viren Jain, Sameera Ponda and Subhashini Venugopalan. 2025. CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoning. arXiv:2503.13517.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{cui2025curie,
  title = {{CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoning}},
  author = {Hao Cui, Zahra Shamsi, Gowoon Cheon, Xuejian Ma, Shutong Li, Maria Tikhanovskaya, Peter Norgaard, Nayantara Mudur, Martyna Plomecka, Paul Raccuglia, Yasaman Bahri, Victor V. Albert, Pranesh Srinivasan, Haining Pan, Philippe Faist, Brian Rohr, Ekin Dogus Cubuk, Muratahan Aykol, Amil Merchant, Michael J. Statt, Dan Morris, Drew Purves, Elise Kleeman, Ruth Alcantara, Matthew Abraham, Muqthar Mohammad, Ean Phing VanLee, Chenfei Jiang, Elizabeth Dorfman, Eun-Ah Kim, Michael P Brenner, Viren Jain, Sameera Ponda and Subhashini Venugopalan},
  year = {2025},
  doi = {10.48550/arxiv.2503.13517},
  eprint = {2503.13517},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2503.13517}
}
```

- [← previous ChemAgent: Self-updating Library in Large Language Models Improves Chemical Reasoning](https://md.research.elamur.ai/free/en/papers/tang-2025-chemagent-self-updating-library-large/)
- [next → Debate-to-Write: A Persona-Driven Multi-Agent Framework for Diverse Argument Generation](https://md.research.elamur.ai/free/en/papers/hu-2025-debate-write-persona-driven-multi/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/6/)

## References

67 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Materialsproject.org. [https://docs.materialsproject.org/methodology/materials-methodology/electronic-structure-accuracy-of-band-structures](https://docs.materialsproject.org/methodology/materials-methodology/electronic-structure-accuracy-of-band-structures), 2017. [↗](https://docs.materialsproject.org/methodology/materials-methodology/electronic-structure-accuracy-of-band-structures)
2. Victor V. Albert and Philippe Faist (eds.). The Error Correction Zoo. 2024. URL [https://errorcorrectionzoo.org/](https://errorcorrectionzoo.org/). [↗](https://errorcorrectionzoo.org/)
3. Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. [L-eval: Instituting standardized evaluation for long context language models](https://md.research.elamur.ai/free/en/papers/an-2023-l-eval-instituting-standardized-evaluation/). arXiv preprint arXiv:2307.11088, 2023.
4. Anthropic. The Claude 3 Model Family: Opus, Sonnet, Haiku. Available online at: — [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://md.research.elamur.ai/free/en/papers/bai-2023-longbench-bilingual-multitask-benchmark/) [https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/](https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/) Model_Card_Claude_3.pdf. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou,… [↗](https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/)
5. Iz Beltagy, Kyle Lo, and Arman Cohan. [Scibert: A pretrained language model for scientific text](https://md.research.elamur.ai/free/en/papers/beltagy-2019-scibert-pretrained-language-model-scientific/). EMNLP, 2019.
6. Thomas Camara, Harold Ollivier, and Jean-Pierre Tillich. A class of quantum ldpc codes: construction and performances under iterative decoding. 2007 IEEE International Symposium on Information Theory, pp. 811–815, 2007. URL [https://api.semanticscholar.org/](https://api.semanticscholar.org/) CorpusID:13899261. [↗](https://api.semanticscholar.org/)
7. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374, 2021.
8. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv, abs/1803.05457, 2018. URL [https://api.semanticscholar.org/](https://api.semanticscholar.org/) CorpusID:3922816. [↗](https://api.semanticscholar.org/)
9. Peter JA Cock, Tiago Antao, Jeffrey T Chang, Brad A Chapman, Cymon J Cox, Andrew Dalke, Iddo Friedberg, Thomas Hamelryck, Frank Kauff, Bartek Wilczynski, et al. Biopython: freely available python tools for computational molecular biology and bioinformatics. Bioinformatics, 25(11):1422, 2009. [find](https://scholar.google.com/scholar?q=Peter+JA+Cock%2C+Tiago+Antao%2C+Jeffrey+T+Chang%2C+Brad+A+Chapman%2C+Cymon+J+Cox%2C+Andrew+Dalke%2C+Iddo+Friedberg%2C+Thomas+Hamelryck%2C+Frank+Kauff%2C+Bartek+Wilczynski%2C+et+al%2E+Biopython%3A+freely+available+python+tool)
10. Arman Cohan, Waleed Ammar, Madeleine Van Zuylen, and Field Cady. [Structural scaffolds for citation intent classification in scientific publications](https://md.research.elamur.ai/free/en/papers/cohan-2019-structural-scaffolds-citation-intent-classification/). arXiv preprint arXiv:1904.01608, 2019.
11. Cohere. Introducing Command R+: A Scalable LLM Built for Business. Available online at: — [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://md.research.elamur.ai/free/en/papers/dasigi-2021-dataset-information-seeking-questions-answers/) [https://cohere.com/blog/command-r-plus-microsoft-azure](https://cohere.com/blog/command-r-plus-microsoft-azure). Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of information-seeking questions and answers anchored in researc… [↗](https://cohere.com/blog/command-r-plus-microsoft-azure)
12. Qingyang Dong and Jacqueline M Cole. Auto-generated database of semiconductor band gaps using chemdataextractor. Scientific Data, 9(1):193, 2022a. [find](https://scholar.google.com/scholar?q=Qingyang+Dong+and+Jacqueline+M+Cole%2E+Auto%2Dgenerated+database+of+semiconductor+band+gaps+using+chemdataextractor%2E+Scientific+Data%2C+9%281%29%3A193%2C+2022a%2E)
13. Qingyang Dong and Jacqueline M Cole. Auto-generated database of semiconductor band gaps using chemdataextractor. Scientific Data, 9(1):193, 2022b. [find](https://scholar.google.com/scholar?q=Qingyang+Dong+and+Jacqueline+M+Cole%2E+Auto%2Dgenerated+database+of+semiconductor+band+gaps+using+chemdataextractor%2E+Scientific+Data%2C+9%281%29%3A193%2C+2022b%2E)
14. Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. [Bamboo: A comprehensive benchmark for evaluating long text modeling capacities of large language models](https://md.research.elamur.ai/free/en/papers/dong-2024-bamboo-comprehensive-benchmark-evaluating-long/). arXiv preprint arXiv:2309.13345, 2023.
15. David Donoho. Data science at the singularity. Harvard Data Science Review, 6(1), 2024. [find](https://scholar.google.com/scholar?q=David+Donoho%2E+Data+science+at+the+singularity%2E+Harvard+Data+Science+Review%2C+6%281%29%2C+2024%2E)
16. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. [Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/). arXiv preprint arXiv:1903.00161, 2019.
17. Alexander Dunn, John Dagdelen, Nicholas Walker, Sanghoon Lee, Andrew S Rosen, Gerbrand Ceder, Kristin Persson, and Anubhav Jain. Structured information extraction from complex scientific text with fine-tuned large language models. arXiv preprint arXiv:2212.05238, 2022. — [arXiv:2212.05238](https://arxiv.org/abs/2212.05238) [find](https://scholar.google.com/scholar?q=Alexander+Dunn%2C+John+Dagdelen%2C+Nicholas+Walker%2C+Sanghoon+Lee%2C+Andrew+S+Rosen%2C+Gerbrand+Ceder%2C+Kristin+Persson%2C+and+Anubhav+Jain%2E+Structured+information+extraction+from+complex+scientific+text+with+fin)
18. Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. [Domain-specific language model pretraining for biomedical natural language processing](https://md.research.elamur.ai/free/en/papers/gu-2020-domain-specific-language-model-pretraining/). ACM Transactions on Computing for Healthcare (HEALTH), 3(1): 1–23, 2021.
19. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300, 2020.
20. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.[06654, 2024](https://md.research.elamur.ai/free/en/papers/hsieh-2024-ruler-s-real-context-size/).
21. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+et+al%2E+Mixtral+of+expe)
22. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. [Swe-bench: Can language models resolve real-world github issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/) arXiv preprint arXiv:2310.06770, 2023.
23. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. [Pubmedqa: A dataset for biomedical research question answering](https://md.research.elamur.ai/free/en/papers/jin-2019-pubmedqa-dataset-biomedical-research-question/). In EMNLP, pp. 2567–2577, 2019.
24. David Jurgens, Srijan Kumar, Raine Hoover, Dan McFarland, and Dan Jurafsky. [Measuring the evolution of a scientific field through citation frames](https://md.research.elamur.ai/free/en/papers/jurgens-2018-measuring-evolution-scientific-field-through/). Transactions of the Association for Computational Linguistics, 6:391–406, 2018.
25. Ehsan Kamalloo, Nouha Dziri, Charles LA Clarke, and Davood Rafiei. [Evaluating open-domain question answering in the era of large language models](https://md.research.elamur.ai/free/en/papers/kamalloo-2023-evaluating-open-domain-question-answering/). arXiv preprint arXiv:2305.06984, 2023.
26. Gregory Kamradt. Needle in a haystack - pressure [https://github.com/gkamradt/LLMTest](https://github.com/gkamradt/LLMTest) NeedleInAHaystack/tree/main, 2023. testing LLMs. [↗](https://github.com/gkamradt/LLMTest)
27. J-D Kim, Tomoko Ohta, Yuka Tateisi, and Jun’ichi Tsujii. Genia corpus—a semantically annotated corpus for bio-textmining. Bioinformatics, 19(suppl 1):i180–i182, 2003. [find](https://scholar.google.com/scholar?q=J%2DD+Kim%2C+Tomoko+Ohta%2C+Yuka+Tateisi%2C+and+Jun%E2%80%99ichi+Tsujii%2E+Genia+corpus%E2%80%94a+semantically+annotated+corpus+for+bio%2Dtextmining%2E+Bioinformatics%2C+19%28suppl+1%29%3Ai180%E2%80%93i182%2C+2003%2E)
28. Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. [The narrativeqa reading comprehension challenge](https://md.research.elamur.ai/free/en/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/). Transactions of the Association for Computational Linguistics, 6:317–328, 2018.
29. Martin Krallinger, Anastasia Krithara, Anastasios Nentidis, Georgios Paliouras, and Marta Villegas. Bioasq at clef2020: Large-scale biomedical semantic indexing and question answering. In Advances in Information Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 1… [find](https://scholar.google.com/scholar?q=Martin+Krallinger%2C+Anastasia+Krithara%2C+Anastasios+Nentidis%2C+Georgios+Paliouras%2C+and+Marta+Villegas%2E+Bioasq+at+clef2020%3A+Large%2Dscale+biomedical+semantic+indexing+and+question+answering%2E+In+Advances+in+)
30. Jens Kringelum, Sonny Kim Kjaerulff, Søren Brunak, Ole Lund, Tudor I Oprea, and Olivier Taboureau. Chemprot-3.0: a global chemical biology diseases mapping. Database, 2016:bav123, 2016. [find](https://scholar.google.com/scholar?q=Jens+Kringelum%2C+Sonny+Kim+Kjaerulff%2C+S%C3%B8ren+Brunak%2C+Ole+Lund%2C+Tudor+I+Oprea%2C+and+Olivier+Taboureau%2E+Chemprot%2D3%2E0%3A+a+global+chemical+biology+diseases+mapping%2E+Database%2C+2016%3Abav123%2C+2016%2E)
31. Wojciech Kryściński, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. Booksum: A collection of datasets for long-form narrative summarization. arXiv preprint arXiv:2105.08209, 2021. [find](https://scholar.google.com/scholar?q=Wojciech+Krys%CC%81cin%CC%81ski%2C+Nazneen+Rajani%2C+Divyansh+Agarwal%2C+Caiming+Xiong%2C+and+Dragomir+Radev%2E+Booksum%3A+A+collection+of+datasets+for+long%2Dform+narrative+summarization%2E+arXiv+preprint+arXiv%3A2105%2E08209%2C+20)
32. Myer Kutz. Handbook of materials selection. John Wiley & Sons, 2002. [find](https://scholar.google.com/scholar?q=Myer+Kutz%2E+Handbook+of+materials+selection%2E+John+Wiley+%26+Sons%2C+2002%2E)
33. Wai-Chung Kwan, Xingshan Zeng, Yufei Wang, Yusen Sun, Liangyou Li, Lifeng Shang, Qun Liu, and Kam-Fai Wong. M4le: A multi-ability multi-range multi-task multi-domain long-context evaluation benchmark for large language models. arXiv preprint arXiv:2310.19240, 2023. [find](https://scholar.google.com/scholar?q=Wai%2DChung+Kwan%2C+Xingshan+Zeng%2C+Yufei+Wang%2C+Yusen+Sun%2C+Liangyou+Li%2C+Lifeng+Shang%2C+Qun+Liu%2C+and+Kam%2DFai+Wong%2E+M4le%3A+A+multi%2Dability+multi%2Drange+multi%2Dtask+multi%2Ddomain+long%2Dcontext+evaluation+benchmark+)
34. Seongyun Lee, Seungone Kim, Sue Hyun Park, Geewook Kim, and Minjoon Seo. Prometheus-vision: Vision-language model as a judge for fine-grained evaluation. arXiv preprint arXiv:2401.06591, 2024. [find](https://scholar.google.com/scholar?q=Seongyun+Lee%2C+Seungone+Kim%2C+Sue+Hyun+Park%2C+Geewook+Kim%2C+and+Minjoon+Seo%2E+Prometheus%2Dvision%3A+Vision%2Dlanguage+model+as+a+judge+for+fine%2Dgrained+evaluation%2E+arXiv+preprint+arXiv%3A2401%2E06591%2C+2024%2E)
35. Yoonjoo Lee, Kyungjae Lee, Sunghyun Park, Dasol Hwang, Jaehyeon Kim, Hong-in Lee, and Moontae Lee. Qasa: advanced question answering on scientific articles. In ICML, pp. 19036– 19052. PMLR, 2023. [find](https://scholar.google.com/scholar?q=Yoonjoo+Lee%2C+Kyungjae+Lee%2C+Sunghyun+Park%2C+Dasol+Hwang%2C+Jaehyeon+Kim%2C+Hong%2Din+Lee%2C+and+Moontae+Lee%2E+Qasa%3A+advanced+question+answering+on+scientific+articles%2E+In+ICML%2C+pp%2E+19036%E2%80%93+19052%2E+PMLR%2C+2023%2E)
36. Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. Biocreative v cdr task corpus: a resource for chemical disease relation extraction. Database, 2016, 2016. [find](https://scholar.google.com/scholar?q=Jiao+Li%2C+Yueping+Sun%2C+Robin+J+Johnson%2C+Daniela+Sciaky%2C+Chih%2DHsuan+Wei%2C+Robert+Leaman%2C+Allan+Peter+Davis%2C+Carolyn+J+Mattingly%2C+Thomas+C+Wiegers%2C+and+Zhiyong+Lu%2E+Biocreative+v+cdr+task+corpus%3A+a+resourc)
37. Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. [Loogle: Can long-context language models understand long contexts?](https://md.research.elamur.ai/free/en/papers/li-2023-loogle-long-context-language-models/) arXiv preprint arXiv:2311.04939, 2023.
38. Dong Liang, Huadong Guo, Lu Zhang, Yun Cheng, Qi Zhu, and Xuting Liu. Time-series snowmelt detection over the antarctic using sentinel-1 sar images on google earth engine. Remote Sensing of Environment, 256:112318, 2021. [find](https://scholar.google.com/scholar?q=Dong+Liang%2C+Huadong+Guo%2C+Lu+Zhang%2C+Yun+Cheng%2C+Qi+Zhu%2C+and+Xuting+Liu%2E+Time%2Dseries+snowmelt+detection+over+the+antarctic+using+sentinel%2D1+sar+images+on+google+earth+engine%2E+Remote+Sensing+of+Environmen)
39. Chin-Yew Lin. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out, pp. 74–81, 2004.
40. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). Transactions of the Association for Computational Linguistics, 12:157–173, 2024.
41. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. [Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/). arXiv preprint arXiv:2310.02255, 2023.
42. Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. [Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction](https://md.research.elamur.ai/free/en/papers/luan-2018-multi-task-identification-entities-relations/). EMNLP, 2018.
43. Oscar Mañas, Benno Krojer, and Aishwarya Agrawal. Improving automatic vqa evaluation using large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4171–4179, 2024. [find](https://scholar.google.com/scholar?q=Oscar+Man%CC%83as%2C+Benno+Krojer%2C+and+Aishwarya+Agrawal%2E+Improving+automatic+vqa+evaluation+using+large+language+models%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volume+38%2C+pp%2E+4171)
44. Juraj Mavracic, Callum J Court, Taketomo Isazawa, Stephen R Elliott, and Jacqueline M Cole. Chemdataextractor 2.0: Autopopulated ontologies for materials science. Journal of Chemical Information and Modeling, 61(9):4280–4289, 2021a. [find](https://scholar.google.com/scholar?q=Juraj+Mavracic%2C+Callum+J+Court%2C+Taketomo+Isazawa%2C+Stephen+R+Elliott%2C+and+Jacqueline+M+Cole%2E+Chemdataextractor+2%2E0%3A+Autopopulated+ontologies+for+materials+science%2E+Journal+of+Chemical+Information+and+M)
45. Juraj Mavracic, Callum J Court, Taketomo Isazawa, Stephen R Elliott, and Jacqueline M Cole. Chemdataextractor 2.0: Autopopulated ontologies for materials science. Journal of Chemical Information and Modeling, 61(9):4280–4289, 2021b. [find](https://scholar.google.com/scholar?q=Juraj+Mavracic%2C+Callum+J+Court%2C+Taketomo+Isazawa%2C+Stephen+R+Elliott%2C+and+Jacqueline+M+Cole%2E+Chemdataextractor+2%2E0%3A+Autopopulated+ontologies+for+materials+science%2E+Journal+of+Chemical+Information+and+M)
46. James Mayfield, Eugene Yang, Dawn Lawrie, Sean MacAvaney, Paul McNamee, Douglas W. Oard, Luca Soldaini, Ian Soboroff, Orion Weller, Efsun Kayi, Kate Sanders, Marc Mason, and Noah Hibbler. On the Evaluation of Machine-Generated Reports, May 2024. URL http://arxiv.org/abs/2405.00982. arXiv:2405.00982… — [arXiv:2405.00982](https://arxiv.org/abs/2405.00982) [↗](http://arxiv.org/abs/2405.00982)
47. Nicolas C. Menicucci. Fault-tolerant measurement-based quantum computing with continuous-variable cluster states. Phys. Rev. Lett., 112:120504, Mar 2014. doi: 10.1103/PhysRevLett. 112.120504. URL [https://link.aps.org/doi/10.1103/PhysRevLett.112](https://link.aps.org/doi/10.1103/PhysRevLett.112). 120504. — [doi:10.1103/physrevlett](https://doi.org/10.1103/physrevlett) [↗](https://link.aps.org/doi/10.1103/PhysRevLett.112)
48. Benjamin Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang, Iain J Marshall, Ani Nenkova, and Byron C Wallace. [A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature](https://md.research.elamur.ai/free/en/papers/nye-2018-corpus-multi-level-annotations-patients/). In Proceedings of the conference. Association for Computation…
49. NIH Public Access, 2018. [find](https://scholar.google.com/scholar?q=NIH+Public+Access%2C+2018%2E)
50. OpenAI. Hello GPT-4o. hello-gpt-4o/. Available online at: [https://openai.com/index/Haining](https://openai.com/index/Haining) Pan, Nayantara Mudur, Will Taranto, Maria Tikhanovskaya, Subhashini Venugopalan, Yasaman Bahri, Michael P. Brenner, and Eun-Ah Kim. Quantum many-body physics calculations with large language models. arxiv 2403… — [arXiv:2403.03154](https://arxiv.org/abs/2403.03154) [↗](https://openai.com/index/Haining)
51. Maciej Polak and Dane Morgan. Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering -Example of ChatGPT. URL — [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/) [https://arxiv.org/pdf/2303.05352.pdf](https://arxiv.org/pdf/2303.05352.pdf). Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-ba… [↗](https://arxiv.org/pdf/2303.05352.pdf)
52. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. [Gpqa: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). arXiv preprint arXiv:2311.12022, 2023.
53. Ari Seifter, Alison Schwarzwalder, Kate Geis, and John Aucott. The utility of google trends for epidemiological research: Lyme disease as an example. Geospatial health, 4(2):135–137, 2010. [find](https://scholar.google.com/scholar?q=Ari+Seifter%2C+Alison+Schwarzwalder%2C+Kate+Geis%2C+and+John+Aucott%2E+The+utility+of+google+trends+for+epidemiological+research%3A+Lyme+disease+as+an+example%2E+Geospatial+health%2C+4%282%29%3A135%E2%80%93137%2C+2010%2E)
54. Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. [Zeroscrolls: A zero-shot benchmark for long text understanding](https://md.research.elamur.ai/free/en/papers/shaham-2023-zeroscrolls-zero-shot-benchmark-long/). arXiv preprint arXiv:2305.14196, 2023.
55. Matthew C Swain and Jacqueline M Cole. Chemdataextractor: a toolkit for automated extraction of chemical information from the scientific literature. Journal of chemical information and modeling, 56(10):1894–1904, 2016. [find](https://scholar.google.com/scholar?q=Matthew+C+Swain+and+Jacqueline+M+Cole%2E+Chemdataextractor%3A+a+toolkit+for+automated+extraction+of+chemical+information+from+the+scientific+literature%2E+Journal+of+chemical+information+and+modeling%2C+56%2810)
56. Szymon Tworkowski, Konrad Staniszewski, Mikołaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Miłoś. Focused transformer: Contrastive training for context scaling. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Szymon+Tworkowski%2C+Konrad+Staniszewski%2C+Miko%C5%82aj+Pacek%2C+Yuhuai+Wu%2C+Henryk+Michalewski%2C+and+Piotr+Mi%C5%82os%CC%81%2E+Focused+transformer%3A+Contrastive+training+for+context+scaling%2E+Advances+in+Neural+Information+Pr)
57. Chonghua Wang, Haodong Duan, Songyang Zhang, Dahua Lin, and Kai Chen. Ada-leval: Evaluating long-context llms with length-adaptable benchmarks. arXiv preprint arXiv:2404.06480, 2024. [find](https://scholar.google.com/scholar?q=Chonghua+Wang%2C+Haodong+Duan%2C+Songyang+Zhang%2C+Dahua+Lin%2C+and+Kai+Chen%2E+Ada%2Dleval%3A+Evaluating+long%2Dcontext+llms+with+length%2Dadaptable+benchmarks%2E+arXiv+preprint+arXiv%3A2404%2E06480%2C+2024%2E)
58. Gerhard Welsch, Rodney Boyer, and EW Collings. Materials properties handbook: titanium alloys. ASM international, 1993. [find](https://scholar.google.com/scholar?q=Gerhard+Welsch%2C+Rodney+Boyer%2C+and+EW+Collings%2E+Materials+properties+handbook%3A+titanium+alloys%2E+ASM+international%2C+1993%2E)
59. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+arXiv+preprint+arXiv%3A1905%2E07830%2C+2019%2E)
60. Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. [MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?](https://md.research.elamur.ai/free/en/papers/zhang-2024-mathverse-multi-modal-llm-truly/) arXiv preprint arXiv:2403.14624, 2024.
61. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+arXiv+preprint+arXiv%3A1904%2E09675%2C+2019%2E)
62. Xiang Zhang, Zichun Zhou, Chen Ming, and Yi-Yang Sun. Gpt-assisted learning of structure-property relationships by graph neural networks: Application to rare-earth doped phosphors. arXiv preprint arXiv:2306.14238, 2023. — [arXiv:2306.14238](https://arxiv.org/abs/2306.14238) [find](https://scholar.google.com/scholar?q=Xiang+Zhang%2C+Zichun+Zhou%2C+Chen+Ming%2C+and+Yi%2DYang+Sun%2E+Gpt%2Dassisted+learning+of+structure%2Dproperty+relationships+by+graph+neural+networks%3A+Application+to+rare%2Dearth+doped+phosphors%2E+arXiv+preprint+arXi)
63. Yuzhou Zhang, Gabriel Milinovich, Zhiwei Xu, Hilary Bambrick, Kerrie Mengersen, Shilu Tong, and Wenbiao Hu. Monitoring pertussis infections using internet search queries. Scientific reports, 7(1):10437, 2017. [find](https://scholar.google.com/scholar?q=Yuzhou+Zhang%2C+Gabriel+Milinovich%2C+Zhiwei+Xu%2C+Hilary+Bambrick%2C+Kerrie+Mengersen%2C+Shilu+Tong%2C+and+Wenbiao+Hu%2E+Monitoring+pertussis+infections+using+internet+search+queries%2E+Scientific+reports%2C+7%281%29%3A1043)
64. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Advances in Neural Information Processing Systems, 36, 2024.
65. Zhiling Zheng, Oufan Zhang, Christian Borgs, Jennifer T Chayes, and Omar M Yaghi. Chatgpt chemistry assistant for text mining and prediction of mof synthesis. arXiv preprint arXiv:2306.11296, 2023. [find](https://scholar.google.com/scholar?q=Zhiling+Zheng%2C+Oufan+Zhang%2C+Christian+Borgs%2C+Jennifer+T+Chayes%2C+and+Omar+M+Yaghi%2E+Chatgpt+chemistry+assistant+for+text+mining+and+prediction+of+mof+synthesis%2E+arXiv+preprint+arXiv%3A2306%2E11296%2C+2023%2E)
66. Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Chunting+Zhou%2C+Pengfei+Liu%2C+Puxin+Xu%2C+Srinivasan+Iyer%2C+Jiao+Sun%2C+Yuning+Mao%2C+Xuezhe+Ma%2C+Avia+Efrat%2C+Ping+Yu%2C+Lili+Yu%2C+et+al%2E+Lima%3A+Less+is+more+for+alignment%2E+Advances+in+Neural+Information+Processing)
67. Andrew Zhu, Alyssa Hwang, Liam Dugan, and Chris Callison-Burch. Fanoutqa: Multi-hop, multi-document question answering for large language models. arXiv preprint arXiv:2402.14116, 2024. — [arXiv:2402.14116](https://arxiv.org/abs/2402.14116) [find](https://scholar.google.com/scholar?q=Andrew+Zhu%2C+Alyssa+Hwang%2C+Liam+Dugan%2C+and+Chris+Callison%2DBurch%2E+Fanoutqa%3A+Multi%2Dhop%2C+multi%2Ddocument+question+answering+for+large+language+models%2E+arXiv+preprint+arXiv%3A2402%2E14116%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E1)

*Updated 1 November 2026*