---
title: "MMSci — paper by Li et al. (2024) — summary and ideas"
description: "Large Vision Language Models (LVLMs) are increasingly pitched as research assistants, but existing… 61 references, cited by 19 papers here."
authors: ["Zekun Li", "Xianjun Yang", "Kyuri Choi", "Wanrong Zhu", "Ryan Hsieh", "HyeonJung Kim", "Jin Hyuk Lim", "Sungyoung Ji", "Byungju Lee", "Xifeng Yan", "Linda Ruth Petzold", "Stephen D. Wilson", "Woosang Lim", "William Yang Wang"]
year: 2024
venue: "arXiv:2407.04903"
source: "https://arxiv.org/abs/2407.04903"
doi: "10.48550/arxiv.2407.04903"
arxiv: "2407.04903"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2024-mmsci-dataset-graduate-level-multi/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2024-mmsci-dataset-graduate-level-multi/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/) / MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding

# MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding

**Authors:** [Zekun Li](https://md.research.elamur.ai/free/en/authors/zekun-li/), [Xianjun Yang](https://md.research.elamur.ai/free/en/authors/xianjun-yang/), Kyuri Choi, [Wanrong Zhu](https://md.research.elamur.ai/free/en/authors/wanrong-zhu/), Ryan Hsieh, HyeonJung Kim, Jin Hyuk Lim, Sungyoung Ji, Byungju Lee, [Xifeng Yan](https://md.research.elamur.ai/free/en/authors/xifeng-yan/), Linda Ruth Petzold, Stephen D. Wilson, Woosang Lim and [William Yang Wang](https://md.research.elamur.ai/free/en/authors/william-yang-wang/) · **Year:** 2024 · **Venue:** arXiv:2407.04903 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2407.04903) · [PDF](https://arxiv.org/pdf/2407.04903) · [DOI 10.48550/arxiv.2407.04903](https://doi.org/10.48550/arxiv.2407.04903) · [arXiv 2407.04903](https://arxiv.org/abs/2407.04903) · [OpenAlex](https://openalex.org/W4400517453)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [19](https://md.research.elamur.ai/free/en/papers/li-2024-mmsci-dataset-graduate-level-multi/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large Vision Language Models (LVLMs) are increasingly pitched as research assistants, but existing benchmarks mostly test simple charts or figures from a narrow set of fields. They rarely probe the expert-level reading of diagrams, micrographs or experimental plots that real papers demand. MMSci fills this gap with a corpus of open-access, peer-reviewed Nature Communications articles (CC BY 4.0) spanning 72 disciplines. Text, figures and captions were collected directly from article webpages, and multi-panel captions were split into sub-captions with regular expressions.

On top of this corpus, the authors built figure-captioning and multiple-choice benchmarks. They tested 19 proprietary and open models and also collected human expert annotations. Many open models scored near chance on multiple choice, while GPT-4o and Claude-3.5-Sonnet led and matched or beat domain experts. Captioning remained difficult for every model.

The data also proved valuable for training. Instruction fine-tuning lifted Qwen2-VL-7B to the best multiple-choice accuracy overall. Continued pre-training of LLaMA2-7B on interleaved article and figure data improved a downstream materials-generation task.

## Key points
- The source corpus contains 131,393 articles and 742,273 figures from 72 subjects, crawled up to 15 April 2024. Regex splitting yielded 514,054 sub-figure/sub-caption pairs.
- Figures fall into seven types. Quantitative charts make up about 53.5%, followed by microscopy (14.7%), schematics (13.2%) and experimental result visuals such as Western blots (8.8%). Maps, macroscopic photos and simulated images form the remainder.
- Among open models, Qwen2-VL-7B and MiniCPM-V-2.6 performed on par with Gemini-1.5-Flash and Claude-3-Opus. GPT-4o and Claude-3.5-Sonnet were the strongest models overall.
- After fine-tuning on MMSci task data, Qwen2-VL-7B surpassed both GPT-4o and human experts on multiple choice. Its captioning performance improved only marginally.
- LLaMA2-7B pre-trained on MMSci plus MMC4 interleaved data, then fine-tuned, achieved the best metastability and stability scores (checked with M3GNet and DFT) in crystal material generation. GPT-4o failed at this task without specialised training.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of California, Santa Barbara](https://md.research.elamur.ai/free/en/universities/university-of-california-santa-barbara/) — University, Santa Barbara, United States · 57 works · 9,752 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 657 works

## Cite

Cite as: Zekun Li, Xianjun Yang, Kyuri Choi, Wanrong Zhu, Ryan Hsieh, HyeonJung Kim, Jin Hyuk Lim, Sungyoung Ji, Byungju Lee, Xifeng Yan, Linda Ruth Petzold, Stephen D. Wilson, Woosang Lim and William Yang Wang. 2024. MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding. arXiv:2407.04903.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{li2024mmsci,
  title = {{MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding}},
  author = {Zekun Li, Xianjun Yang, Kyuri Choi, Wanrong Zhu, Ryan Hsieh, HyeonJung Kim, Jin Hyuk Lim, Sungyoung Ji, Byungju Lee, Xifeng Yan, Linda Ruth Petzold, Stephen D. Wilson, Woosang Lim and William Yang Wang},
  year = {2024},
  doi = {10.48550/arxiv.2407.04903},
  eprint = {2407.04903},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.04903}
}
```

- [← previous Mixat: A Data Set of Bilingual Emirati-English Speech](https://md.research.elamur.ai/free/en/papers/ali-2024-mixat-data-set-bilingual-emirati/)
- [next → Qalam: A Multimodal LLM for Arabic Optical Character and Handwriting Recognition](https://md.research.elamur.ai/free/en/papers/bhatia-2024-qalam-multimodal-llm-arabic-optical/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/)

## References

61 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. AI will transform science - now researchers must tame it. Nature, 621(7980):658, September 2023. [find](https://scholar.google.com/scholar?q=AI+will+transform+science+%2D+now+researchers+must+tame+it%2E+Nature%2C+621%287980%29%3A658%2C+September+2023%2E)
2. Achiam, O. J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shapiro, G., Berner, C., Bogdonof… [find](https://scholar.google.com/scholar?q=Achiam%2C+O%2E+J%2E%2C+Adler%2C+S%2E%2C+Agarwal%2C+S%2E%2C+Ahmad%2C+L%2E%2C+Akkaya%2C+I%2E%2C+Aleman%2C+F%2E+L%2E%2C+Almeida%2C+D%2E%2C+Altenschmidt%2C+J%2E%2C+Altman%2C+S%2E%2C+Anadkat%2C+S%2E%2C+Avila%2C+R%2E%2C+Babuschkin%2C+I%2E%2C+Balaji%2C+S%2E%2C+Balcom%2C+V%2E%2C+Baltescu%2C+P%2E%2C+Ba)
3. Anthropic. Claude 3.5 sonnet, June 2024b. [https://www.anthropic.com/news/](https://www.anthropic.com/news/) claude-3-5-sonnet. [↗](https://www.anthropic.com/news/)
4. URL Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.[12966, 2023a](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/).
5. Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. [Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/). 2023b.
6. Banerjee, S. and Lavie, A. [Meteor: An automatic metric for mt evaluation with improved correlation with human judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/). In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65–72, 2005.
7. Guo, Y., Peng, B., Lu, G., Dong, G., Yang, G., Chen, B., Qiu, R., Liu, H., Zhang, B., Yao, Y., et al. Remarkable flexibility in freestanding single-crystalline antiferroelec-tric pbzro3 membranes. Nature Communications, 15(1): 4414, 2024. [find](https://scholar.google.com/scholar?q=Guo%2C+Y%2E%2C+Peng%2C+B%2E%2C+Lu%2C+G%2E%2C+Dong%2C+G%2E%2C+Yang%2C+G%2E%2C+Chen%2C+B%2E%2C+Qiu%2C+R%2E%2C+Liu%2C+H%2E%2C+Zhang%2C+B%2E%2C+Yao%2C+Y%2E%2C+et+al%2E+Remarkable+flexibility+in+freestanding+single%2Dcrystalline+antiferroelec%2Dtric+pbzro3+membranes%2E+Nat)
8. Chen, C. and Ong, S. P. A universal graph deep learning interatomic potential for the periodic table. Nature Computational Science, 2(11):718–728, 2022. [find](https://scholar.google.com/scholar?q=Chen%2C+C%2E+and+Ong%2C+S%2E+P%2E+A+universal+graph+deep+learning+interatomic+potential+for+the+periodic+table%2E+Nature+Computational+Science%2C+2%2811%29%3A718%E2%80%93728%2C+2022%2E)
9. Hafner, J. Ab-initio simulations of materials using vasp: Density-functional theory and beyond. Journal of computational chemistry, 29(13):2044–2078, 2008. [find](https://scholar.google.com/scholar?q=Hafner%2C+J%2E+Ab%2Dinitio+simulations+of+materials+using+vasp%3A+Density%2Dfunctional+theory+and+beyond%2E+Journal+of+computational+chemistry%2C+29%2813%29%3A2044%E2%80%932078%2C+2008%2E)
10. Chen, C., Zhang, R., Koh, E., Kim, S., Cohen, S., and Rossi, R. Figure captioning with relation maps for reasoning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1537–1545, 2020. [find](https://scholar.google.com/scholar?q=Chen%2C+C%2E%2C+Zhang%2C+R%2E%2C+Koh%2C+E%2E%2C+Kim%2C+S%2E%2C+Cohen%2C+S%2E%2C+and+Rossi%2C+R%2E+Figure+captioning+with+relation+maps+for+reasoning%2E+In+Proceedings+of+the+IEEE%2FCVF+Winter+Conference+on+Applications+of+Computer+Vision%2C)
11. Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations, 2021.
12. Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Wang, J., Qiao, Y., Lin, D., et al. [Are we on the right way for evaluating large vision-language models?](https://md.research.elamur.ai/free/en/papers/chen-2024-right-way-evaluating-large-vision/) arXiv preprint arXiv:2403.20330, 2024a.
13. Jain, A., Ong, S. P., Hautier, G., Chen, W., Richards, W. D., Dacek, S., Cholia, S., Gunter, D., Skinner, D., Ceder, G., et al. Commentary: The materials project: A materials genome approach to accelerating materials innovation. APL materials, 1(1), 2013. [find](https://scholar.google.com/scholar?q=Jain%2C+A%2E%2C+Ong%2C+S%2E+P%2E%2C+Hautier%2C+G%2E%2C+Chen%2C+W%2E%2C+Richards%2C+W%2E+D%2E%2C+Dacek%2C+S%2E%2C+Cholia%2C+S%2E%2C+Gunter%2C+D%2E%2C+Skinner%2C+D%2E%2C+Ceder%2C+G%2E%2C+et+al%2E+Commentary%3A+The+materials+project%3A+A+materials+genome+approach+to+accele)
14. Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., et al. [How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites](https://md.research.elamur.ai/free/en/papers/chen-2024-far-gpt-4v-closing/). arXiv preprint arXiv:2404.16821, 2024b.
15. Kafle, K., Price, B., Cohen, S., and Kanan, C. Dvqa: Understanding data visualizations via question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5648–5656, 2018. [find](https://scholar.google.com/scholar?q=Kafle%2C+K%2E%2C+Price%2C+B%2E%2C+Cohen%2C+S%2E%2C+and+Kanan%2C+C%2E+Dvqa%3A+Understanding+data+visualizations+via+question+answering%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pp%2E+564)
16. Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al. [Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks](https://md.research.elamur.ai/free/en/papers/chen-2023-internvl-scaling-up-vision/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185–24198,…
17. Kahou, S. E., Michalski, V., Atkinson, A., Kádár, Á., Trischler, A., and Bengio, Y. Figureqa: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300, 2017. [find](https://scholar.google.com/scholar?q=Kahou%2C+S%2E+E%2E%2C+Michalski%2C+V%2E%2C+Atkinson%2C+A%2E%2C+Ka%CC%81da%CC%81r%2C+A%CC%81%2E%2C+Trischler%2C+A%2E%2C+and+Bengio%2C+Y%2E+Figureqa%3A+An+annotated+figure+dataset+for+visual+reasoning%2E+arXiv+preprint+arXiv%3A1710%2E07300%2C+2017%2E)
18. Kang, B., Jang, M., Chung, Y., Kim, H., Kwak, S. K., Oh, J. H., and Cho, K. Enhancing 2d growth of organic semiconductor thin films with macroporous structures via a small-molecule heterointerface. Nature Communications, 5(1):4752, Aug 2014. ISSN 2041-1723. doi: 10.1038/ncomms5752. URL [https://doi.o…](https://doi.o…) — [doi:10.1038/ncomms5752](https://doi.org/10.1038/ncomms5752) [↗](https://doi.o…)
19. Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207–1216, Stanford, CA, 2000. Morgan Kaufmann. [find](https://scholar.google.com/scholar?q=Langley%2C+P%2E+Crafting+papers+on+machine+learning%2E+In+Langley%2C+P%2E+%28ed%2E%29%2C+Proceedings+of+the+17th+International+Conference+on+Machine+Learning+%28ICML+2000%29%2C+pp%2E+1207%E2%80%931216%2C+Stanford%2C+CA%2C+2000%2E+Morgan+Kaufm)
20. Laurençon, H., Marafioti, A., Sanh, V., and Tronchon, L. Building and better understanding vision-language models: insights and future directions., 2024a. [find](https://scholar.google.com/scholar?q=Laurenc%CC%A7on%2C+H%2E%2C+Marafioti%2C+A%2E%2C+Sanh%2C+V%2E%2C+and+Tronchon%2C+L%2E+Building+and+better+understanding+vision%2Dlanguage+models%3A+insights+and+future+directions%2E%2C+2024a%2E)
21. Davies, D. W., Butler, K. T., Jackson, A. J., Skelton, J. M., Morita, K., and Walsh, A. Smact: Semiconducting materials by analogy and chemical theory. Journal of Open Source Software, 4(38):1361, 2019. [find](https://scholar.google.com/scholar?q=Davies%2C+D%2E+W%2E%2C+Butler%2C+K%2E+T%2E%2C+Jackson%2C+A%2E+J%2E%2C+Skelton%2C+J%2E+M%2E%2C+Morita%2C+K%2E%2C+and+Walsh%2C+A%2E+Smact%3A+Semiconducting+materials+by+analogy+and+chemical+theory%2E+Journal+of+Open+Source+Software%2C+4%2838%29%3A1361%2C+201)
22. Laurençon, H., Tronchon, L., Cord, M., and Sanh, V. [What matters when building vision-language models?](https://md.research.elamur.ai/free/en/papers/lauren-ccon-2024-matters-building-vision-language-models/), 2024b.
23. Dettmers, T., Lewis, M., Shleifer, S., and Zettlemoyer, L. [8-bit optimizers via block-wise quantization](https://md.research.elamur.ai/free/en/papers/dettmers-2021-8-bit-optimizers-block-wise/). arXiv preprint arXiv:2110.02861, 2021.
24. Li, J., Li, D., Savarese, S., and Hoi, S. [Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). In International conference on machine learning, pp. 19730–19742. PMLR, 2023.
25. Gruver, N., Sriram, A., Madotto, A., Wilson, A. G., Zitnick, C. L., and Ulissi, Z. Fine-tuned language models generate stable inorganic materials as text. arXiv preprint arXiv:2402.04379, 2024. — [arXiv:2402.04379](https://arxiv.org/abs/2402.04379) [find](https://scholar.google.com/scholar?q=Gruver%2C+N%2E%2C+Sriram%2C+A%2E%2C+Madotto%2C+A%2E%2C+Wilson%2C+A%2E+G%2E%2C+Zitnick%2C+C%2E+L%2E%2C+and+Ulissi%2C+Z%2E+Fine%2Dtuned+language+models+generate+stable+inorganic+materials+as+text%2E+arXiv+preprint+arXiv%3A2402%2E04379%2C+2024%2E+%E2%80%94+%5BarX)
26. Li, L., Wang, Y., Xu, R., Wang, P., Feng, X., Kong, L., and Liu, Q. [Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models](https://md.research.elamur.ai/free/en/papers/li-2024-multimodal-arxiv-dataset-improving-scientific/). arXiv preprint arXiv:2403.00231, 2024. et al. Learning transferable visual models from natural language supervision. In Internationa…
27. Lin, C.-Y. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out, pp. 74–81, 2004.
28. Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lillicrap, T., Alayrac, J.-b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., et al. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint arXiv:2403.05530, 2024.
29. Lin, J., Yin, H., Ping, W., Lu, Y., Molchanov, P., Tao, A., Mao, H., Kautz, J., Shoeybi, M., and Han, S. [Vila: On pre-training for visual language models](https://md.research.elamur.ai/free/en/papers/lin-2023-vila-pre-training-visual-language/). arXiv preprint arXiv:2312.07533, 2023.
30. Roberts, J., Han, K., Houlsby, N., and Albanie, S. Scifibench: Benchmarking large multimodal models for scientific figure interpretation. arXiv preprint arXiv:2405.08807, 2024. — [arXiv:2405.08807](https://arxiv.org/abs/2405.08807) [find](https://scholar.google.com/scholar?q=Roberts%2C+J%2E%2C+Han%2C+K%2E%2C+Houlsby%2C+N%2E%2C+and+Albanie%2C+S%2E+Scifibench%3A+Benchmarking+large+multimodal+models+for+scientific+figure+interpretation%2E+arXiv+preprint+arXiv%3A2405%2E08807%2C+2024%2E+%E2%80%94+%5BarXiv%3A2405%2E08807%5D%28ht)
31. Liu, H., Li, C., Li, Y., and Lee, Y. J. [Improved baselines with visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/). arXiv preprint arXiv:2310.03744, 2023a.
32. Rubungo, A. N., Arnold, C., Rand, B. P., and Dieng, A. B. Llm-prop: Predicting physical and electronic properties of crystalline solids from their text descriptions. arXiv preprint arXiv:2310.14029, 2023. — [arXiv:2310.14029](https://arxiv.org/abs/2310.14029) [find](https://scholar.google.com/scholar?q=Rubungo%2C+A%2E+N%2E%2C+Arnold%2C+C%2E%2C+Rand%2C+B%2E+P%2E%2C+and+Dieng%2C+A%2E+B%2E+Llm%2Dprop%3A+Predicting+physical+and+electronic+properties+of+crystalline+solids+from+their+text+descriptions%2E+arXiv+preprint+arXiv%3A2310%2E14029%2C+2)
33. Liu, H., Li, C., Wu, Q., and Lee, Y. J. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). Advances in neural information processing systems, 36, 2024.
34. Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., and Zhu, C. [G-eval: Nlg evaluation using gpt-4 with better human alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/). arXiv preprint arXiv:2303.16634, 2023b.
35. Siegel, N., Horvitz, Z., Levin, R., Divvala, S., and Farhadi, A. Figureseer: Parsing result-figures in research papers. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part VII 14, pp. 664–680. [find](https://scholar.google.com/scholar?q=Siegel%2C+N%2E%2C+Horvitz%2C+Z%2E%2C+Levin%2C+R%2E%2C+Divvala%2C+S%2E%2C+and+Farhadi%2C+A%2E+Figureseer%3A+Parsing+result%2Dfigures+in+research+papers%2E+In+Computer+Vision%E2%80%93ECCV+2016%3A+14th+European+Conference%2C+Amsterdam%2C+The+Netherlan)
36. Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521, 2022. [find](https://scholar.google.com/scholar?q=Lu%2C+P%2E%2C+Mishra%2C+S%2E%2C+Xia%2C+T%2E%2C+Qiu%2C+L%2E%2C+Chang%2C+K%2E%2DW%2E%2C+Zhu%2C+S%2E%2DC%2E%2C+Tafjord%2C+O%2E%2C+Clark%2C+P%2E%2C+and+Kalyan%2C+A%2E+Learn+to+explain%3A+Multimodal+reasoning+via+thought+chains+for+science+question+answering%2E+Advance)
37. Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
38. Masry, A., Long, D. X., Tan, J. Q., Joty, S., and Hoque, E. [Chartqa: A benchmark for question answering about charts with visual and logical reasoning](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/). arXiv preprint arXiv:2203.10244, 2022.
39. Team, M. L. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models, Sep 2024. URL — [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://md.research.elamur.ai/free/en/papers/min-2023-factscore-fine-grained-atomic-evaluation/) [https://ai.meta.com/blog/](https://ai.meta.com/blog/) llama-3-2-connect-2024-vision-edge-mobile-devices Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-t., Koh, P. W., Iyyer, M., Zettlemoyer, L., and Hajishirzi, H. Factsco… [↗](https://ai.meta.com/blog/)
40. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
41. Miret, S. and Krishnan, N. Are llms ready for real-world materials discovery? arXiv preprint arXiv:2402.05200, 2024. [find](https://scholar.google.com/scholar?q=Miret%2C+S%2E+and+Krishnan%2C+N%2E+Are+llms+ready+for+real%2Dworld+materials+discovery%3F+arXiv+preprint+arXiv%3A2402%2E05200%2C+2024%2E)
42. Vedantam, R., Lawrence Zitnick, C., and Parikh, D. [Cider: Consensus-based image description evaluation](https://md.research.elamur.ai/free/en/papers/vedantam-2014-cider-consensus-based-image-description/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566–4575, 2015.
43. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311–318, 2002.
44. Vert, J.-P. How will generative ai disrupt data science in drug discovery? Nature Biotechnology, 41(6):750– 751, Jun 2023. ISSN 1546-1696. doi: 10.1038/ s41587-023-01789-6. URL https://doi.org/10. 1038/s41587-023-01789-6. [find](https://scholar.google.com/scholar?q=Vert%2C+J%2E%2DP%2E+How+will+generative+ai+disrupt+data+science+in+drug+discovery%3F+Nature+Biotechnology%2C+41%286%29%3A750%E2%80%93+751%2C+Jun+2023%2E+ISSN+1546%2D1696%2E+doi%3A+10%2E1038%2F+s41587%2D023%2D01789%2D6%2E+URL+https%3A%2F%2Fdoi%2Eorg%2F10%2E+103)
45. Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., and Wei, F. [Kosmos-2: Grounding multimodal large language models to the world](https://md.research.elamur.ai/free/en/papers/peng-2023-kosmos-2-grounding-multimodal-large/). arXiv preprint arXiv:2306.14824, 2023.
46. Walker, N., Trewartha, A., Huo, H., Lee, S., Cruse, K., Dagdelen, J., Dunn, A., Persson, K., Ceder, G., and Jain, A. The impact of domain-specific pre-training on named entity recognition tasks in materials science. Available at SSRN 3950755, 2021. [find](https://scholar.google.com/scholar?q=Walker%2C+N%2E%2C+Trewartha%2C+A%2E%2C+Huo%2C+H%2E%2C+Lee%2C+S%2E%2C+Cruse%2C+K%2E%2C+Dagdelen%2C+J%2E%2C+Dunn%2C+A%2E%2C+Persson%2C+K%2E%2C+Ceder%2C+G%2E%2C+and+Jain%2C+A%2E+The+impact+of+domain%2Dspecific+pre%2Dtraining+on+named+entity+recognition+tasks+in+mat)
47. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Q… — [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://md.research.elamur.ai/free/en/papers/wang-2024-qwen2-vl-enhancing-vision/)
48. Yue, X., Zheng, T., Ni, Y., Wang, Y., Zhang, K., Tong, S., Sun, Y., Yin, M., Yu, B., Zhang, G., et al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2409.02813, 2024. [find](https://scholar.google.com/scholar?q=Yue%2C+X%2E%2C+Zheng%2C+T%2E%2C+Ni%2C+Y%2E%2C+Wang%2C+Y%2E%2C+Zhang%2C+K%2E%2C+Tong%2C+S%2E%2C+Sun%2C+Y%2E%2C+Yin%2C+M%2E%2C+Yu%2C+B%2E%2C+Zhang%2C+G%2E%2C+et+al%2E+Mmmu%2Dpro%3A+A+more+robust+multi%2Ddiscipline+multimodal+understanding+benchmark%2E+arXiv+preprint+arXiv)
49. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2019. [find](https://scholar.google.com/scholar?q=Zhang%2C+T%2E%2C+Kishore%2C+V%2E%2C+Wu%2C+F%2E%2C+Weinberger%2C+K%2E+Q%2E%2C+and+Artzi%2C+Y%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+In+International+Conference+on+Learning+Representations%2C+2019%2E)
50. Wang, X., Hu, Z., Lu, P., Zhu, Y., Zhang, J., Subramaniam, S., Loomba, A. R., Zhang, S., Sun, Y., and Wang, W. [Scibench: Evaluating college-level scientific problem-solving abilities of large language models](https://md.research.elamur.ai/free/en/papers/wang-2023-scibench-evaluating-college-level-scientific/). arXiv preprint arXiv:2307.10635, 2023.
51. Zheng, Y., Zhang, R., Zhang, J., Ye, Y., Luo, Z., Feng, Z., and Ma, Y. [Llamafactory: Unified efficient fine-tuning of 100+ language models](https://md.research.elamur.ai/free/en/papers/zheng-2024-llamafactory-unified-efficient-fine-tuning/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Associati…
52. Wang, Y.-C., Chin, K.-H., Tu, Z.-L., He, J., Jones, C. J., Sanchez, D. Z., Yildiz, F. H., Galperin, M. Y., and Chou, S.-H. Nucleotide binding by the widespread high-affinity cyclic di-gmp receptor mshen domain. Nature Communications, 7(1):12481, Aug 2016. ISSN 2041-1723. doi: 10.1038/ncomms12481. UR… — [doi:10.1038/ncomms12481](https://doi.org/10.1038/ncomms12481) [find](https://scholar.google.com/scholar?q=Wang%2C+Y%2E%2DC%2E%2C+Chin%2C+K%2E%2DH%2E%2C+Tu%2C+Z%2E%2DL%2E%2C+He%2C+J%2E%2C+Jones%2C+C%2E+J%2E%2C+Sanchez%2C+D%2E+Z%2E%2C+Yildiz%2C+F%2E+H%2E%2C+Galperin%2C+M%2E+Y%2E%2C+and+Chou%2C+S%2E%2DH%2E+Nucleotide+binding+by+the+widespread+high%2Daffinity+cyclic+di%2Dgmp+receptor+msh)
53. Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. [Minigpt-4: Enhancing vision-language understanding with advanced large language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/). arXiv preprint arXiv:2304.10592, 2023.
54. Wang, Z., Xia, M., He, L., Chen, H., Liu, Y., Zhu, R., Liang, K., Wu, X., Liu, H., Malladi, S., et al. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. arXiv preprint arXiv:2406.18521, 2024b. [find](https://scholar.google.com/scholar?q=Wang%2C+Z%2E%2C+Xia%2C+M%2E%2C+He%2C+L%2E%2C+Chen%2C+H%2E%2C+Liu%2C+Y%2E%2C+Zhu%2C+R%2E%2C+Liang%2C+K%2E%2C+Wu%2C+X%2E%2C+Liu%2C+H%2E%2C+Malladi%2C+S%2E%2C+et+al%2E+Charxiv%3A+Charting+gaps+in+realistic+chart+understanding+in+multimodal+llms%2E+arXiv+preprint+arXiv%3A)
55. Zhu, W., Hessel, J., Awadalla, A., Gadre, S. Y., Dodge, J., Fang, A., Yu, Y., Schmidt, L., Wang, W. Y., and Choi, Y. Multimodal c4: An open, billion-scale corpus of images interleaved with text. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Zhu%2C+W%2E%2C+Hessel%2C+J%2E%2C+Awadalla%2C+A%2E%2C+Gadre%2C+S%2E+Y%2E%2C+Dodge%2C+J%2E%2C+Fang%2C+A%2E%2C+Yu%2C+Y%2E%2C+Schmidt%2C+L%2E%2C+Wang%2C+W%2E+Y%2E%2C+and+Choi%2C+Y%2E+Multimodal+c4%3A+An+open%2C+billion%2Dscale+corpus+of+images+interleaved+with+text%2E+Advan)
56. Ward, L., Dunn, A., Faghaninia, A., Zimmermann, N. E., Bajaj, S., Wang, Q., Montoya, J., Chen, J., Bystrom, K., Dylla, M., et al. Matminer: An open source toolkit for materials data mining. Computational Materials Science, 152:60–69, 2018. [find](https://scholar.google.com/scholar?q=Ward%2C+L%2E%2C+Dunn%2C+A%2E%2C+Faghaninia%2C+A%2E%2C+Zimmermann%2C+N%2E+E%2E%2C+Bajaj%2C+S%2E%2C+Wang%2C+Q%2E%2C+Montoya%2C+J%2E%2C+Chen%2C+J%2E%2C+Bystrom%2C+K%2E%2C+Dylla%2C+M%2E%2C+et+al%2E+Matminer%3A+An+open+source+toolkit+for+materials+data+mining%2E+Computatio)
57. White, A. D. The future of chemistry is language. Nature Reviews Chemistry, 7(7):457–458, 2023. [find](https://scholar.google.com/scholar?q=White%2C+A%2E+D%2E+The+future+of+chemistry+is+language%2E+Nature+Reviews+Chemistry%2C+7%287%29%3A457%E2%80%93458%2C+2023%2E)
58. Xie, T., Fu, X., Ganea, O.-E., Barzilay, R., and Jaakkola, T. Crystal diffusion variational autoencoder for periodic material generation. arXiv preprint arXiv:2110.06197, 2021. [find](https://scholar.google.com/scholar?q=Xie%2C+T%2E%2C+Fu%2C+X%2E%2C+Ganea%2C+O%2E%2DE%2E%2C+Barzilay%2C+R%2E%2C+and+Jaakkola%2C+T%2E+Crystal+diffusion+variational+autoencoder+for+periodic+material+generation%2E+arXiv+preprint+arXiv%3A2110%2E06197%2C+2021%2E)
59. Yang, Z., Dabre, R., Tanaka, H., and Okazaki, N. Scicap+: A knowledge augmented dataset to study the challenges of scientific figure captioning. arXiv preprint arXiv:2306.03491, 2023. — [arXiv:2306.03491](https://arxiv.org/abs/2306.03491) [find](https://scholar.google.com/scholar?q=Yang%2C+Z%2E%2C+Dabre%2C+R%2E%2C+Tanaka%2C+H%2E%2C+and+Okazaki%2C+N%2E+Scicap%2B%3A+A+knowledge+augmented+dataset+to+study+the+challenges+of+scientific+figure+captioning%2E+arXiv+preprint+arXiv%3A2306%2E03491%2C+2023%2E+%E2%80%94+%5BarXiv%3A2306%2E03)
60. Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He, Z., et al. [Minicpm-v: A gpt-4v level mllm on your phone](https://md.research.elamur.ai/free/en/papers/yao-2024-minicpm-v-gpt-4v/). arXiv preprint arXiv:2408.01800, 2024.
61. Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. [Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/). arXiv preprint arXiv:2311.16502, 2023.

*Updated 1 November 2026*