---
title: "A Careful Examination of Large Language Model Performance on"
description: "The work offers a template for detecting benchmark overfitting and argues for releasing test sets only… 32 references, cited by 83 papers here."
authors: ["Hugh Zhang", "Jeff Da", "Dean Lee", "Vaughn Robinson", "Catherine Wu", "Will Song", "Tiffany Zhao", "Pranav Raja", "Charlotte Zhuang", "Dylan Slack", "Qin Lyu", "Sean Hendryx", "Russell Kaplan", "Michele Lunati", "Summer Yue"]
year: 2024
venue: "arXiv:2405.00332"
source: "https://arxiv.org/abs/2405.00332"
doi: "10.48550/arxiv.2405.00332"
arxiv: "2405.00332"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2024-careful-examination-large-language-model/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2024-careful-examination-large-language-model/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / A Careful Examination of Large Language Model Performance on Grade School Arithmetic

# A Careful Examination of Large Language Model Performance on Grade School Arithmetic

**Authors:** [Hugh Zhang](https://md.research.elamur.ai/free/en/authors/hugh-zhang/), [Jeff Da](https://md.research.elamur.ai/free/en/authors/jeff-da/), Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, Qin Lyu, [Sean Hendryx](https://md.research.elamur.ai/free/en/authors/sean-hendryx/), Russell Kaplan, Michele Lunati and Summer Yue · **Year:** 2024 · **Venue:** [arXiv:2405.00332](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2024](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2024/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2405.00332) · [PDF](https://arxiv.org/pdf/2405.00332) · [DOI 10.48550/arxiv.2405.00332](https://doi.org/10.48550/arxiv.2405.00332) · [arXiv 2405.00332](https://arxiv.org/abs/2405.00332) · [OpenAlex](https://openalex.org/W4415797772)

**Citations:** 18 citations (OpenAlex, 2026-10-14) · [83](https://md.research.elamur.ai/free/en/papers/zhang-2024-careful-examination-large-language-model/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Strong LLM scores on math benchmarks may partly come from contamination: training data that resembles or contains test items. To measure this, Scale AI researchers built GSM1k, a fresh set of 1205 grade-school word problems. Human annotators wrote the problems with no LLM help, and the set was designed to match GSM8k in solution step count, answer magnitude and human solve rate. Each problem went through three review layers, including an independent blind re-solve. The authors then compared many open and closed models on GSM8k and GSM1k. Some families, notably Phi, Mistral and parts of Llama, scored up to 8 points lower on the new set, and this held across most of their sizes. Gemini, GPT and Claude showed almost no gap. Models that assign higher likelihood to GSM8k text tended to have larger gaps, which points to partial memorization. Even so, every model could solve unseen problems. The work offers a template for detecting benchmark overfitting and argues for releasing test sets only after a delay.

## Key points
- **Dataset construction:** GSM1k started from 2108 human-written candidates. Of these, 1419 survived the blind re-solve and 1375 passed the quality audit. The final 1205 were chosen to mirror GSM8k's answer-size distribution. All answers are positive integers.
- **Memorization signal:** The authors measured each model's per-character log-likelihood of the GSM8k test set. Its Spearman correlation with the GSM8k–GSM1k gap was 0.36 (p = 0.03), alongside Kendall's τ = 0.29.
- **Contamination is not the whole story:** Math-Shepherd-Mistral-7B-RL is heavily overfit but has low likelihood on GSM8k. Llemma models show high likelihood yet barely overfit.
- **Overfit models still reason:** Phi-2 drops about 6 points but still solves over half of GSM1k. That is comparable to Llama2-70B, which has more than 25 times as many parameters.
- **Delayed release:** GSM1k is withheld for now. It will be published under the MIT license once three open-source models from distinct lineages reach 95%, or in June 2025, whichever comes first. The evaluation code, a fork of EleutherAI's LM Evaluation Harness, will be open-sourced (NeurIPS 2024 Datasets and Benchmarks).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati and Summer Yue. 2024. A Careful Examination of Large Language Model Performance on Grade School Arithmetic. arXiv:2405.00332.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhang2024careful,
  title = {{A Careful Examination of Large Language Model Performance on Grade School Arithmetic}},
  author = {Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati and Summer Yue},
  year = {2024},
  doi = {10.48550/arxiv.2405.00332},
  eprint = {2405.00332},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.00332}
}
```

- [← previous How do Large Language Models Handle Multilingualism?](https://md.research.elamur.ai/free/en/papers/zhao-2024-large-language-models-handle-multilingualism/)
- [next → Debating with More Persuasive LLMs Leads to More Truthful Answers](https://md.research.elamur.ai/free/en/papers/khan-2024-debating-more-persuasive-llms-leads/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

32 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Alli… — [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/)
2. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. [Program Synthesis with Large Language Models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/), August 2021. URL http://arxiv.org/abs/2108.07732. arXiv:2108.07732 [cs]. [↗](http://arxiv.org/abs/2108.07732)
3. Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. [Llemma: An Open Language Model For Mathematics](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-llemma-open-language-model-mathematics/), March 2024. URL http://arxiv.org/abs/2310.10631. arXiv:2310.10631 [cs]. [↗](http://arxiv.org/abs/2310.10631)
4. Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondřej Dušek. [Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs](https://md.research.elamur.ai/free/en/papers/balloccu-2024-leak-cheat-repeat-data-contamination/), February 2024. URL http://arxiv.org/abs/2402.03927. arXiv:2402.03927 [cs]. [↗](http://arxiv.org/abs/2402.03927)
5. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-20B: An Open-Source Autoregressive… — [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/)
6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretchen)
7. Curran Associates, Inc., 2020. URL [https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf](https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf). [↗](https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf)
8. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying Memorization Across Neural Language Models, March 2023. URL http://arxiv.org/abs/2202.07646. arXiv:2202.07646 [cs]. [↗](http://arxiv.org/abs/2202.07646)
9. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
10. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. [Training Verifiers to Solve Math Word Problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/), November 2021. URL http://arxiv.org/abs/2110.14168. arXiv:2110… [↗](http://arxiv.org/abs/2110.14168)
11. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Ani… [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Baber+Abbasi%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Alain+Le+Noac%E2%80%99h%2C+Haonan+Li%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Chris+Oci)
12. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL: Program-aided Language Models, January 2023b. URL http://arxiv.org/abs/2211.10435. arXiv:2211.10435 [cs]. [↗](http://arxiv.org/abs/2211.10435)
13. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi… [find](https://scholar.google.com/scholar?q=Suriya+Gunasekar%2C+Yi+Zhang%2C+Jyoti+Aneja%2C+Caio+C%C3%A9sar+Teodoro+Mendes%2C+Allie+Del+Giorno%2C+Sivakanth+Gopi%2C+Mojan+Javaheripi%2C+Piero+Kauffmann%2C+Gustavo+de+Rosa%2C+Olli+Saarikivi%2C+Adil+Salim%2C+Shital+Shah%2C+Harki)
14. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring Massive Multitask Language Understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/), January 2021a. URL http://arxiv.org/abs/2009.03300. arXiv:2009.03300 [cs]. [↗](http://arxiv.org/abs/2009.03300)
15. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring Mathematical Problem Solving with the MATH Dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). NeurIPS, 2021b.
16. Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natu… [find](https://scholar.google.com/scholar?q=Alon+Jacovi%2C+Avi+Caciularu%2C+Omer+Goldman%2C+and+Yoav+Goldberg%2E+Stop+Uploading+Test+Data+in+Plain+Text%3A+Practical+Strategies+for+Mitigating+Data+Contamination+by+Evaluation+Benchmarks%2E+In+Houda+Bouamor%2C+)
17. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
18. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guill)
19. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/), April 2024. URL http://arxiv.org/abs/2310.06770. arXiv:2310.06770 [cs]. [↗](http://arxiv.org/abs/2310.06770)
20. Inbal Magar and Roy Schwartz. Data Contamination: From Memorization to Exploitation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 157–165, Dublin, Ireland,… [find](https://scholar.google.com/scholar?q=Inbal+Magar+and+Roy+Schwartz%2E+Data+Contamination%3A+From+Memorization+to+Exploitation%2E+In+Smaranda+Muresan%2C+Preslav+Nakov%2C+and+Aline+Villavicencio%2C+editors%2C+Proceedings+of+the+60th+Annual+Meeting+of+the)
21. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… — [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
22. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. page 24, 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+Language+Models+are+Unsupervised+Multitask+Learners%2E+page+24%2C+2019%2E)
23. Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet Classifiers Generalize to ImageNet?, June 2019. URL http://arxiv.org/abs/1902.10811. arXiv:1902.10811 [cs, stat]. [↗](http://arxiv.org/abs/1902.10811)
24. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/), November 2023. URL https://arxiv.org/abs/2311.12022v1. [↗](https://arxiv.org/abs/2311.12022v1)
25. Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. [NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark](https://md.research.elamur.ai/free/en/papers/sainz-2023-nlp-evaluation-trouble-need-measure/). In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Ling…
26. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting Pretraining Data from Large Language Models, March 2024. URL http://arxiv.org/abs/2310.16789. arXiv:2310.16789 [cs]. [↗](http://arxiv.org/abs/2310.16789)
27. Saurabh Srivastava, Annarose M. B, Anto P V, Shashank Menon, Ajay Sukumar, Adwaith Samod T, Alan Philipose, Stevin Prince, and Sooraj Thomas. Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap, February 2024. URL [http://arxiv.org/](http://arxiv.org/) abs/2402.19450. arXiv:2402.1… [↗](http://arxiv.org/)
28. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazari… — [Gemini: A Family of Highly Capable Multimodal Models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/)
29. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. [LLaMA: Open and Efficient Foundation Language Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/), February 2023a…
30. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
31. Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. [Benchmarking Benchmark Leakage in Large Language Models](https://md.research.elamur.ai/free/en/papers/xu-2024-benchmarking-benchmark-leakage-large-language/), April 2024. URL http://arxiv.org/abs/2404.18824. arXiv:2404.18824 [cs]. [↗](http://arxiv.org/abs/2404.18824)
32. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. [MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/), October 2023. URL http://arxiv.org/abs/2309. 12284. arXiv:2309.12284 [cs]. [↗](http://arxiv.org/abs/2309)

*Updated 2 November 2026*