---
title: "Language Models as Science Tutors — Chevalier 2024"
description: "The work offers a new way to test free-form, long-context science help, plus an open recipe for… 47 references, cited by 23 papers here."
authors: ["Alexis Chevalier", "Jiayi Geng", "Alexander Wettig", "Howard Chen", "Sebastian Mizera", "Toni Annala", "Max Jameson Aragon", "Arturo Rodriguez Fanlo", "Simon Frieder", "Simon Machado", "Akshara Prabhakar", "Ellie Thieu", "Jiachen T. Wang", "Zirui Wang", "Xindi Wu", "Mengzhou Xia", "Wenhan Xia", "Jiatong Yu", "Jun-Jie Zhu", "Zhiyong Jason Ren", "Sanjeev Arora", "Danqi Chen"]
year: 2024
venue: "arXiv:2402.11111"
source: "https://arxiv.org/abs/2402.11111"
doi: "10.48550/arxiv.2402.11111"
arxiv: "2402.11111"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chevalier-2024-language-models-science-tutors/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chevalier-2024-language-models-science-tutors/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) / Language Models as Science Tutors

# Language Models as Science Tutors

**Authors:** [Alexis Chevalier](https://md.research.elamur.ai/free/en/authors/alexis-chevalier/), Jiayi Geng, [Alexander Wettig](https://md.research.elamur.ai/free/en/authors/alexander-wettig/), [Howard Chen](https://md.research.elamur.ai/free/en/authors/howard-chen/), Sebastian Mizera, Toni Annala, Max Jameson Aragon, Arturo Rodriguez Fanlo, [Simon Frieder](https://md.research.elamur.ai/free/en/authors/simon-frieder/), Simon Machado, [Akshara Prabhakar](https://md.research.elamur.ai/free/en/authors/akshara-prabhakar/), Ellie Thieu, Jiachen T. Wang, [Zirui Wang](https://md.research.elamur.ai/free/en/authors/zirui-wang/), Xindi Wu, [Mengzhou Xia](https://md.research.elamur.ai/free/en/authors/mengzhou-xia/), Wenhan Xia, Jiatong Yu, Jun-Jie Zhu, Zhiyong Jason Ren, [Sanjeev Arora](https://md.research.elamur.ai/free/en/authors/sanjeev-arora/) and [Danqi Chen](https://md.research.elamur.ai/free/en/authors/danqi-chen/) · **Year:** 2024 · **Venue:** arXiv:2402.11111 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.11111) · [PDF](https://arxiv.org/pdf/2402.11111) · [DOI 10.48550/arxiv.2402.11111](https://doi.org/10.48550/arxiv.2402.11111) · [arXiv 2402.11111](https://arxiv.org/abs/2402.11111) · [OpenAlex](https://openalex.org/W4392011902)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [23](https://md.research.elamur.ai/free/en/papers/chevalier-2024-language-models-science-tutors/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most evaluations of scientific language models only check final answers to short problems. That says little about whether a model can help a student working through a long textbook chapter. To measure this, the authors built TutorEval: 834 questions written by 17 STEM researchers with teaching experience, tied to 202 chapters of open textbooks from libretexts.org. For each question, the annotator also sketched the key points a good answer should cover, and GPT-4 grades model replies against those points.

Base models fine-tuned on existing dialogue datasets scored poorly on TutorEval. The team therefore created TutorChat, about 80,000 synthetic long conversations about textbook chapters, generated with GPT-3.5 and GPT-4. They fine-tuned Llemma (7B and 34B, 32K-token context) on MathMix, which blends the STEM portion of TutorChat with MetaMath. The resulting tutors do well on TutorEval and stay strong on GSM8K and MATH. The 7B model is competitive with Mistral-7B-V2. The work offers a new way to test free-form, long-context science help, plus an open recipe for building such tutors.

## Key points
- TutorEval's questions span five fields (the life sciences, mathematics, environmental science, physics and computer science), though math and CS make up most of them. Chapters average about 1,800 words and reach roughly 6,100. Its 370 closed-book questions also allow testing of short-context models (TutorEval-ClosedBook).
- The GPT-4 grader agrees well with human judgments. 14 of 17 annotators agreed on an overall model ranking, and GPT-4 reproduced that ranking on 12 of the 17 question sets.
- TutorChat alone, or combined with MetaMath, did not improve math scores. Only the TutorChat-STEM + MetaMath mix (MathMix) gave a clear boost.
- In human evaluation, 16 of 17 annotators preferred Llemma-7B-32K-MathMix over Llemma-7B-32K-UltraChat for correctness and presentation. The MathMix model was also observed to resist misleading questions.
- Models, data and evaluation code are released at github.com/princeton-nlp/LM-Science-Tutor. The paper appeared at ICML 2024.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Alexis Chevalier, Jiayi Geng, Alexander Wettig, Howard Chen, Sebastian Mizera, Toni Annala, Max Jameson Aragon, Arturo Rodriguez Fanlo, Simon Frieder, Simon Machado, Akshara Prabhakar, Ellie Thieu, Jiachen T. Wang, Zirui Wang, Xindi Wu, Mengzhou Xia, Wenhan Xia, Jiatong Yu, Jun-Jie Zhu, Zhiyong Jason Ren, Sanjeev Arora and Danqi Chen. 2024. Language Models as Science Tutors. arXiv:2402.11111.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{chevalier2024language,
  title = {{Language Models as Science Tutors}},
  author = {Alexis Chevalier, Jiayi Geng, Alexander Wettig, Howard Chen, Sebastian Mizera, Toni Annala, Max Jameson Aragon, Arturo Rodriguez Fanlo, Simon Frieder, Simon Machado, Akshara Prabhakar, Ellie Thieu, Jiachen T. Wang, Zirui Wang, Xindi Wu, Mengzhou Xia, Wenhan Xia, Jiatong Yu, Jun-Jie Zhu, Zhiyong Jason Ren, Sanjeev Arora and Danqi Chen},
  year = {2024},
  doi = {10.48550/arxiv.2402.11111},
  eprint = {2402.11111},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.11111}
}
```

- [← previous Jamba-1.5: Hybrid Transformer-Mamba Models at Scale](https://md.research.elamur.ai/free/en/papers/lenz-2024-jamba-1-5-hybrid-transformer/)
- [next → MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling](https://md.research.elamur.ai/free/en/papers/limisiewicz-2024-myte-morphology-driven-byte-encoding/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/)

## References

47 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732, 2021.
2. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). ArXiv, abs/1803.05457, 2018. URL [https://api.se](https://api.se) manticscholar.org/CorpusID:3922816. [↗](https://api.se)
3. Azerbayev, Z., Schoelkopf, H., Paster, K., Santos, M. D., McAleer, S., Jiang, A. Q., Deng, J., Biderman, S., and Welleck, S. [Llemma: An open language model for mathematics](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-llemma-open-language-model-mathematics/), 2023.
4. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
5. Collins, K. M., Jiang, A. Q., Frieder, S., Wong, L., Zilka, M., Bhatt, U., Lukasiewicz, T., Wu, Y., Tenenbaum, J. B., Hart, W., Gowers, T., Li, W., Weller, A., and Jamnik, M. Evaluating language models for mathematics through interactions, 2023. [find](https://scholar.google.com/scholar?q=Collins%2C+K%2E+M%2E%2C+Jiang%2C+A%2E+Q%2E%2C+Frieder%2C+S%2E%2C+Wong%2C+L%2E%2C+Zilka%2C+M%2E%2C+Bhatt%2C+U%2E%2C+Lukasiewicz%2C+T%2E%2C+Wu%2C+Y%2E%2C+Tenenbaum%2C+J%2E+B%2E%2C+Hart%2C+W%2E%2C+Gowers%2C+T%2E%2C+Li%2C+W%2E%2C+Weller%2C+A%2E%2C+and+Jamnik%2C+M%2E+Evaluating+language+model)
6. Huang, Q., Vora, J., Liang, P., and Leskovec, J. Benchmarking large language models as ai research agents. arXiv preprint arXiv:2310.[03302, 2023](https://md.research.elamur.ai/free/en/papers/huang-2023-mlagentbench-evaluating-language-agents/).
7. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [find](https://scholar.google.com/scholar?q=Jiang%2C+A%2E+Q%2E%2C+Sablayrolles%2C+A%2E%2C+Mensch%2C+A%2E%2C+Bamford%2C+C%2E%2C+Chaplot%2C+D%2E+S%2E%2C+Casas%2C+D%2E+d%2E+l%2E%2C+Bressand%2C+F%2E%2C+Lengyel%2C+G%2E%2C+Lample%2C+G%2E%2C+Saulnier%2C+L%2E%2C+et+al%2E+Mistral+7b%2E+arXiv+preprint+arXiv%3A2310%2E06825%2C+2023%2E)
8. Dao, T. [FlashAttention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). 2023.
9. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=Dao%2C+T%2E%2C+Fu%2C+D%2E+Y%2E%2C+Ermon%2C+S%2E%2C+Rudra%2C+A%2E%2C+and+R%C3%A9%2C+C%2E+FlashAttention%3A+Fast+and+memory%2Defficient+exact+attention+with+IO%2Dawareness%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2022%2E)
10. Kingma, D. P. and Ba, J. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6980. [↗](http://arxiv.org/abs/1412.6980)
11. Ding, N., Chen, Y., Xu, B., Qin, Y., Hu, S., Liu, Z., Sun, M., and Zhou, B. [Enhancing chat language models by scaling high-quality instructional conversations](https://md.research.elamur.ai/free/en/papers/ding-2023-enhancing-chat-language-models-scaling/). In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3…
12. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., and Petrov, S. [Natural questions: A benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). T…
13. Frieder, S., Pinchetti, L., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P. C., Chevalier, A., and Berner, J. [Mathematical capabilities of chatgpt](https://md.research.elamur.ai/free/en/papers/frieder-2023-mathematical-capabilities-chatgpt/). arXiv preprint arXiv:2301.13867, 2023.
14. Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et al. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/). Advances in Neural Information Processing Systems, 35:3843–3857, 2022.
15. Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac’h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework… [find](https://scholar.google.com/scholar?q=Gao%2C+L%2E%2C+Tow%2C+J%2E%2C+Abbasi%2C+B%2E%2C+Biderman%2C+S%2E%2C+Black%2C+S%2E%2C+DiPofi%2C+A%2E%2C+Foster%2C+C%2E%2C+Golding%2C+L%2E%2C+Hsu%2C+J%2E%2C+Le+Noac%E2%80%99h%2C+A%2E%2C+Li%2C+H%2E%2C+McDonell%2C+K%2E%2C+Muennighoff%2C+N%2E%2C+Ociepa%2C+C%2E%2C+Phang%2C+J%2E%2C+Reynolds%2C+L%2E%2C+Schoelko)
16. Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/tatsu-lab/alpaca_](https://github.com/tatsu-lab/alpaca_) eval, 2023a. [↗](https://github.com/tatsu-lab/alpaca_)
17. Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Del Giorno, A., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644, 2023. [find](https://scholar.google.com/scholar?q=Gunasekar%2C+S%2E%2C+Zhang%2C+Y%2E%2C+Aneja%2C+J%2E%2C+Mendes%2C+C%2E+C%2E+T%2E%2C+Del+Giorno%2C+A%2E%2C+Gopi%2C+S%2E%2C+Javaheripi%2C+M%2E%2C+Kauffmann%2C+P%2E%2C+de+Rosa%2C+G%2E%2C+Saarikivi%2C+O%2E%2C+et+al%2E+Textbooks+are+all+you+need%2E+arXiv+preprint+arXiv%3A2306)
18. Li, Y., Bubeck, S., Eldan, R., Del Giorno, A., Gunasekar, S., and Lee, Y. T. [Textbooks are all you need ii: phi-1.5 technical report](https://md.research.elamur.ai/free/en/papers/li-2023-textbooks-all-need-ii-phi/). arXiv preprint arXiv:2309.05463, 2023b.
19. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
20. Li, Y., Qu, S., Shen, J., Min, S., and Yu, Z. Curriculum-driven edubot: A framework for developing language learning chatbots through synthesizing conversational data, 2023c. [find](https://scholar.google.com/scholar?q=Li%2C+Y%2E%2C+Qu%2C+S%2E%2C+Shen%2C+J%2E%2C+Min%2C+S%2E%2C+and+Yu%2C+Z%2E+Curriculum%2Ddriven+edubot%3A+A+framework+for+developing+language+learning+chatbots+through+synthesizing+conversational+data%2C+2023c%2E)
21. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In Vanschoren, J. and Yeung, S. (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volum…
22. Lian, W., Goodson, B., Pentland, E., Cook, A., Vong, C., and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces. https://https://huggingface.co/Open-Orca/OpenOrca, 2023. [find](https://scholar.google.com/scholar?q=Lian%2C+W%2E%2C+Goodson%2C+B%2E%2C+Pentland%2C+E%2E%2C+Cook%2C+A%2E%2C+Vong%2C+C%2E%2C+and+%22Teknium%22%2E+Openorca%3A+An+open+dataset+of+gpt+augmented+flan+reasoning+traces%2E+https%3A%2F%2Fhttps%3A%2F%2Fhuggingface%2Eco%2FOpen%2DOrca%2FOpenOrca%2C+2023%2E)
23. Luo, H., Sun, Q., Xu, C., Zhao, P., Lou, J., Tao, C., Geng, X., Lin, Q., Chen, S., and Zhang, D. [Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct](https://md.research.elamur.ai/free/en/papers/luo-2023-wizardmath-empowering-mathematical-reasoning-large/). arXiv preprint arXiv:2308.09583, 2023.
24. Findings of the Association for Computational Linguistics: ACL 2023, pp. 13387–13434, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.847. URL https://ac lanthology.org/2023.findings-acl.847. — [doi:10.18653/v1/2023.findings-acl.847](https://doi.org/10.18653/v1/2023.findings-acl.847) [find](https://scholar.google.com/scholar?q=Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2023%2C+pp%2E+13387%E2%80%9313434%2C+Toronto%2C+Canada%2C+July+2023%2E+Association+for+Computational+Linguistics%2E+doi%3A+10%2E18653%2Fv1%2F2023%2Efindings%2Dacl%2E847%2E+URL)
25. Macina, J., Daheim, N., Chowdhury, S., Sinha, T., Kapur, M., Gurevych, I., and Sachan, M. [MathDial: A dialogue tutoring dataset with rich pedagogical properties grounded in math reasoning problems](https://md.research.elamur.ai/free/en/papers/macina-2023-mathdial-dialogue-tutoring-dataset-rich/). In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computational Linguisti…
26. Peskoff, D. and Stewart, B. Credible without credit: Domain experts assess generative language models. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 427–438, Toronto, Canad… [find](https://scholar.google.com/scholar?q=Peskoff%2C+D%2E+and+Stewart%2C+B%2E+Credible+without+credit%3A+Domain+experts+assess+generative+language+models%2E+In+Rogers%2C+A%2E%2C+Boyd%2DGraber%2C+J%2E%2C+and+Okazaki%2C+N%2E+%28eds%2E%29%2C+Proceedings+of+the+61st+Annual+Meeting+of)
27. Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al. [Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/). arXiv preprint arXiv:2308.12950, 2023.
28. Mukherjee, S., Mitra, A., Jawahar, G., Agarwal, S., Palangi, H., and Awadallah, A. [Orca: Progressive learning from complex explanation traces of gpt-4](https://md.research.elamur.ai/free/en/papers/mukherjee-2023-orca-progressive-learning-complex-explanation/). arXiv preprint arXiv:2306.02707, 2023.
29. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99–106, aug 2021. ISSN 0001-0782. doi: 10.1145/3474381. URL https://doi.org/10.1145/3474381. [↗](https://doi.org/10.1145/3474381)
30. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023.
31. Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N. Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/). In Erk, K. and Smith, N. A. (eds.), Proceedings of the 54th Annual Meeting of the Association for C…
32. Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Neurocomputing, 568:127063, 2024.
33. Sun, L., Han, Y., Zhao, Z., Ma, D., Shen, Z., Chen, B., Chen, L., and Yu, K. Scieval: A multi-level large language model evaluation benchmark for scientific research, 2023. [find](https://scholar.google.com/scholar?q=Sun%2C+L%2E%2C+Han%2C+Y%2E%2C+Zhao%2C+Z%2E%2C+Ma%2C+D%2E%2C+Shen%2C+Z%2E%2C+Chen%2C+B%2E%2C+Chen%2C+L%2E%2C+and+Yu%2C+K%2E+Scieval%3A+A+multi%2Dlevel+large+language+model+evaluation+benchmark+for+scientific+research%2C+2023%2E)
34. Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., and Stojnic, R. [Galactica: A large language model for science](https://md.research.elamur.ai/free/en/papers/taylor-2022-galactica-large-language-model-science/). arXiv preprint arXiv:2211.09085, 2022.
35. Paster, K., Santos, M. D., Azerbayev, Z., and Ba, J. [Openwebmath: An open dataset of high-quality mathematical web text](https://md.research.elamur.ai/free/en/papers/paster-2023-openwebmath-open-dataset-high-quality/). arXiv preprint arXiv:2310.06786, 2023.
36. Perez, E., Ringer, S., Lukosiute, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., Jones, A., Chen, A., Mann, B., Israel, B., Seethor, B., McKinnon, C., Olah, C., Yan, D., Amodei, D., Amodei, D., Drain, D., Li, D., Tran-Johnson, E., Khundadze, G., Kernion, J., L… — [Discovering Language Model Behaviors with Model-Written Evaluations](https://md.research.elamur.ai/free/en/papers/perez-2022-discovering-language-model-behaviors-model/)
37. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30, 2017. in the wild. In The Twelfth International Conference on Learning Representations, 2024. URL https://ope…
38. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J., and Stoica, I. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). ArXiv, abs/2306.05685, 2023. URL [https://api.semanticscholar.org/CorpusID:259129398](https://api.semanticscholar.org/CorpusID:259129398). [↗](https://api.semanticscholar.org/CorpusID:259129398)
39. Wang, X., Hu, Z., Lu, P., Zhu, Y., Zhang, J., Subramaniam, S., Loomba, A. R., Zhang, S., Sun, Y., and Wang, W. [Scibench: Evaluating college-level scientific problem-solving abilities of large language models](https://md.research.elamur.ai/free/en/papers/wang-2023-scibench-evaluating-college-level-scientific/). arXiv preprint arXiv:2307.10635, 2023.
40. Welbl, J., Liu, N. F., and Gardner, M. [Crowdsourcing multiple choice science questions](https://md.research.elamur.ai/free/en/papers/welbl-2017-crowdsourcing-multiple-choice-science-questions/). In Derczynski, L., Xu, W., Ritter, A., and Baldwin, T. (eds.), Proceedings of the 3rd Workshop on Noisy User-generated Text, pp. 94–106, Copenhagen, Denmark, September 2017. Association for Computational Linguist…
41. Wu, C., Gan, Y., Ge, Y., Lu, Z., Wang, J., Feng, Y., Luo, P., and Shan, Y. Llama pro: Progressive llama with block expansion, 2024. [find](https://scholar.google.com/scholar?q=Wu%2C+C%2E%2C+Gan%2C+Y%2E%2C+Ge%2C+Y%2E%2C+Lu%2C+Z%2E%2C+Wang%2C+J%2E%2C+Feng%2C+Y%2E%2C+Luo%2C+P%2E%2C+and+Shan%2C+Y%2E+Llama+pro%3A+Progressive+llama+with+block+expansion%2C+2024%2E)
42. Wu, Y., Henriksson, A., Duneld, M., and Nouri, J. Towards improving the reliability and transparency of chatgpt for educational question answering. In Viberg, O., Jivet, I., Muñoz-Merino, P. J., Perifanou, M., and Papathoma, T. (eds.), Responsive and Sustainable Educational Futures, pp. 475–488, Cha… [find](https://scholar.google.com/scholar?q=Wu%2C+Y%2E%2C+Henriksson%2C+A%2E%2C+Duneld%2C+M%2E%2C+and+Nouri%2C+J%2E+Towards+improving+the+reliability+and+transparency+of+chatgpt+for+educational+question+answering%2E+In+Viberg%2C+O%2E%2C+Jivet%2C+I%2E%2C+Mu%C3%B1oz%2DMerino%2C+P%2E+J%2E%2C+Perif)
43. Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J. T., Li, Z., Weller, A., and Liu, W. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). arXiv preprint arXiv:2309.12284, 2023.
44. Yue, X., Qu, X., Zhang, G., Fu, Y., Huang, W., Sun, H., Su, Y., and Chen, W. [Mammoth: Building math generalist models through hybrid instruction tuning](https://md.research.elamur.ai/free/en/papers/yue-2023-mammoth-building-math-generalist-models/). arXiv preprint arXiv:2309.05653, 2023.
45. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can a machine really finish your sentence? In Korhonen, A., Traum, D., and Màrquez, L. (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791– 4800, Florence, Italy, July 2019… [find](https://scholar.google.com/scholar?q=Zellers%2C+R%2E%2C+Holtzman%2C+A%2E%2C+Bisk%2C+Y%2E%2C+Farhadi%2C+A%2E%2C+and+Choi%2C+Y%2E+HellaSwag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Korhonen%2C+A%2E%2C+Traum%2C+D%2E%2C+and+M%C3%A0rquez%2C+L%2E+%28eds%2E%29%2C+Proceedings+of+the+57th+Annual+)
46. Zeng, Z., Yu, J., Gao, T., Meng, Y., Goyal, T., and Chen, D. Evaluating large language models at evaluating instruction following. arXiv preprint arXiv:2310.07641, 2023. [find](https://scholar.google.com/scholar?q=Zeng%2C+Z%2E%2C+Yu%2C+J%2E%2C+Gao%2C+T%2E%2C+Meng%2C+Y%2E%2C+Goyal%2C+T%2E%2C+and+Chen%2C+D%2E+Evaluating+large+language+models+at+evaluating+instruction+following%2E+arXiv+preprint+arXiv%3A2310%2E07641%2C+2023%2E)
47. Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., and Deng, Y. (inthe)wildchat: 570k chatGPT interaction logs [find](https://scholar.google.com/scholar?q=Zhao%2C+W%2E%2C+Ren%2C+X%2E%2C+Hessel%2C+J%2E%2C+Cardie%2C+C%2E%2C+Choi%2C+Y%2E%2C+and+Deng%2C+Y%2E+%28inthe%29wildchat%3A+570k+chatGPT+interaction+logs)

*Updated 30 October 2026*