---
title: "OpenWebMath: An Open Dataset of High-Quality — Paster 2023"
description: "Large language models get noticeably better at quantitative reasoning when trained on math-heavy text."
authors: ["Keiran Paster", "Marco Dos Santos", "Zhangir Azerbayev", "Jimmy Ba"]
year: 2023
venue: "arXiv:2310.06786"
source: "https://arxiv.org/abs/2310.06786"
doi: "10.48550/arxiv.2310.06786"
arxiv: "2310.06786"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/paster-2023-openwebmath-open-dataset-high-quality/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/paster-2023-openwebmath-open-dataset-high-quality/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text

# OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text

**Authors:** [Keiran Paster](https://md.research.elamur.ai/free/en/authors/keiran-paster/), [Marco Dos Santos](https://md.research.elamur.ai/free/en/authors/marco-dos-santos/), [Zhangir Azerbayev](https://md.research.elamur.ai/free/en/authors/zhangir-azerbayev/), [Jimmy Ba](https://md.research.elamur.ai/free/en/authors/jimmy-ba/) · **Year:** 2023 · **Venue:** arXiv:2310.06786 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2310.06786) · [PDF](https://arxiv.org/pdf/2310.06786) · [DOI 10.48550/arxiv.2310.06786](https://doi.org/10.48550/arxiv.2310.06786) · [arXiv 2310.06786](https://arxiv.org/abs/2310.06786) · [OpenAlex](https://openalex.org/W4387561465)

**Citations:** 4 citations (openalex-snapshot · 2026-10-14) · [120](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Large language models get noticeably better at quantitative reasoning when trained on math-heavy text. Minerva showed this, but its web corpus was never released. Existing open web datasets also mangle mathematical notation during HTML-to-text conversion, so open researchers had no comparable resource. The authors build OpenWebMath as a public alternative to Minerva's Math Web Pages. They process every Common Crawl snapshot available before May 2023 through a multi-stage pipeline: a cheap math prefilter, English language identification, a math-content classifier (MathScore), perplexity-based quality filtering, near-duplicate removal and a final manual filtering pass. Their custom HTML extraction keeps LaTeX equations intact and strips boilerplate. Throughout, they tuned the pipeline to favour recall, so a little non-math content was accepted to avoid losing relevant pages. To test the corpus, they trained 1.4B-parameter models on it. These models outperformed models trained on more than 20 times as much general-domain text. The dataset, pipeline code and a full datasheet are public, which makes math-focused pretraining reproducible outside large labs.

## Key points
- **Size and sources:** OpenWebMath contains about 6.3M documents (14.7B tokens) filtered from Common Crawl shards dated 2013–2023. It is hosted on the Hugging Face Hub under the ODC-By 1.0 license.
- **Pipeline stages:** prefilter, language ID, MathScore classifier, perplexity filter, deduplication and manual filtering. The extraction step preserves LaTeX.
- **Content:** Q&A forums are the largest source, followed by educational, reference, paper and blog pages. Mathematics is the leading subject, with sizeable shares of physics, computer science and statistics.
- **Comparison:** Minerva's closed Math Web Pages corpus held 17.5B tokens. Related open resources include Proof-Pile, AMPS and NaturalProofs.
- **Limitation:** the corpus is English-only, so it is of limited direct use for math models in other languages, including Ukrainian.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Mathematics & statistics](https://md.research.elamur.ai/free/en/shelves/mathematics/)

## Cited by

Cited by

120 papers of our library cite it (of the 140884 whose bibliographies we read); 13 of them have a page here, 107 not yet.

- [Fanar: An Arabic-Centric Multimodal Generative AI Platform](https://md.research.elamur.ai/free/en/papers/abbas-2025-fanar-arabic-centric-multimodal-generative/)
- [Quiet-STaR: Language models can teach themselves to think before speaking](https://md.research.elamur.ai/free/en/papers/al-2024-quiet-star-language-models-teach/)
- [SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model](https://md.research.elamur.ai/free/en/papers/allal-2025-smollm2-smol-goes-big-data/)
- [Llemma: An Open Language Model For Mathematics](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-llemma-open-language-model-mathematics/)
- [Olmo 3](https://md.research.elamur.ai/free/en/papers/ettinger-2025-olmo-3/)
- [Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs](https://md.research.elamur.ai/free/en/papers/gandhi-2025-cognitive-behaviors-that-enable-self/)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://md.research.elamur.ai/free/en/papers/kimi-2025-team-kimi-k1-5-scaling-reinforcement/)
- [EuroLLM: Multilingual Language Models for Europe](https://md.research.elamur.ai/free/en/papers/martins-2024-eurollm-multilingual-language-models-europe/)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/)
- [LIMO: Less is More for Reasoning](https://md.research.elamur.ai/free/en/papers/ye-2025-limo-less-more-reasoning/)
- [Demystifying Long Chain-of-Thought Reasoning in LLMs](https://md.research.elamur.ai/free/en/papers/yeo-2025-demystifying-long-chain-thought-reasoning/)
- [MAmmoTH2: Scaling Instructions from the Web](https://md.research.elamur.ai/free/en/papers/yue-2024-mammoth2-scaling-instructions-web/)
- [Instruction Tuning for Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/zhang-2023-instruction-tuning-large-language-models/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{paster2023openwebmath,
  title = {{OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text}},
  author = {Paster, Keiran and Santos, Marco Dos and Azerbayev, Zhangir and Ba, Jimmy},
  year = {2023},
  doi = {10.48550/arxiv.2310.06786},
  eprint = {2310.06786},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.06786}
}
```

- [← previous OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models](https://md.research.elamur.ai/free/en/papers/awadalla-2023-openflamingo-open-source-framework-training/)
- [next → Optimizing the Neural Network Training for OCR Error Correction of Historical Hebrew Texts](https://md.research.elamur.ai/free/en/papers/hbo-2023-suissa-optimizing-neural-network-training-ocr/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

38 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Jason Phang, Shivanshu Purohit, Hailey Schoelkopf, Dashiell Stander, Tri Songz, Curt Tigges, Benjamin Thérien, Phil Wang, and… [find](https://scholar.google.com/scholar?q=Alex+Andonian%2C+Quentin+Anthony%2C+Stella+Biderman%2C+Sid+Black%2C+Preetham+Gali%2C+Leo+Gao%2C+Eric+Hallahan%2C+Josh+Levy%2DKramer%2C+Connor+Leahy%2C+Lucas+Nestler%2C+Kip+Parker%2C+Michael+Pieler%2C+Jason+Phang%2C+Shivanshu+Pur)
2. Zhangir Azerbayev, Bartosz Piotrowski, Hailey Schoelkopf, Edward W Ayers, Dragomir Radev, and Jeremy Avigad. Proofnet: Autoformalizing and formally proving undergraduate-level mathematics. arXiv preprint arXiv:2302.12433, 2023. [find](https://scholar.google.com/scholar?q=Zhangir+Azerbayev%2C+Bartosz+Piotrowski%2C+Hailey+Schoelkopf%2C+Edward+W+Ayers%2C+Dragomir+Radev%2C+and+Jeremy+Avigad%2E+Proofnet%3A+Autoformalizing+and+formally+proving+undergraduate%2Dlevel+mathematics%2E+arXiv+prepr) [our card](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-proofnet-autoformalizing-formally-proving-undergraduate/)
3. Adrien Barbaresi. Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction. In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing… [find](https://scholar.google.com/scholar?q=Adrien+Barbaresi%2E+Trafilatura%3A+A+Web+Scraping+Library+and+Command%2DLine+Tool+for+Text+Discovery+and+Extraction%2E+In+Proceedings+of+the+Joint+Conference+of+the+59th+Annual+Meeting+of+the+Association+for+)
4. Janek Bevendorff, Benno Stein, Matthias Hagen, and Martin Potthast. Elastic ChatNoir: Search Engine for the ClueWeb and the Common Crawl. In Leif Azzopardi, Allan Hanbury, Gabriella Pasi, and Benjamin Piwowarski (eds.), Advances in Information Retrieval. 40th European Conference on IR Research (ECIR… [find](https://scholar.google.com/scholar?q=Janek+Bevendorff%2C+Benno+Stein%2C+Matthias+Hagen%2C+and+Martin+Potthast%2E+Elastic+ChatNoir%3A+Search+Engine+for+the+ClueWeb+and+the+Common+Crawl%2E+In+Leif+Azzopardi%2C+Allan+Hanbury%2C+Gabriella+Pasi%2C+and+Benjamin)
5. Janek Bevendorff, Martin Potthast, and Benno Stein. FastWARC: Optimizing Large-Scale Web Archive Analytics. In Andreas Wagner, Christian Guetl, Michael Granitzer, and Stefan Voigt (eds.), 3rd International Symposium on Open Search Technology (OSSYM 2021). [find](https://scholar.google.com/scholar?q=Janek+Bevendorff%2C+Martin+Potthast%2C+and+Benno+Stein%2E+FastWARC%3A+Optimizing+Large%2DScale+Web+Archive+Analytics%2E+In+Andreas+Wagner%2C+Christian+Guetl%2C+Michael+Granitzer%2C+and+Stefan+Voigt+%28eds%2E%29%2C+3rd+Internat)
6. International Open Search Symposium, October 2021. [find](https://scholar.google.com/scholar?q=International+Open+Search+Symposium%2C+October+2021%2E)
7. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across train… [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Quentin+Gregory+Anthony%2C+Herbie+Bradley%2C+Kyle+O%E2%80%99Brien%2C+Eric+Hallahan%2C+Mohammad+Aflah+Khan%2C+Shivanshu+Purohit%2C+USVSN+Sai+Prashanth%2C+Edward+Raff%2C+Aviya+Skowron%2C+Linta)
8. PMLR, 2023. URL [https://proceedings.mlr.press/v202/biderman23a.html](https://proceedings.mlr.press/v202/biderman23a.html). [↗](https://proceedings.mlr.press/v202/biderman23a.html)
9. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… [find](https://scholar.google.com/scholar?q=Tom+B%2E+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche)
10. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… [find](https://scholar.google.com/scholar?q=Aakanksha+Chowdhery%2C+Sharan+Narang%2C+Jacob+Devlin%2C+Maarten+Bosma%2C+Gaurav+Mishra%2C+Adam+Roberts%2C+Paul+Barham%2C+Hyung+Won+Chung%2C+Charles+Sutton%2C+Sebastian+Gehrmann%2C+Parker+Schuh%2C+Kensen+Shi%2C+Sasha+Tsvyashc)
11. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. [find](https://scholar.google.com/scholar?q=Karl+Cobbe%2C+Vineet+Kosaraju%2C+Mohammad+Bavarian%2C+Mark+Chen%2C+Heewoo+Jun%2C+Lukasz+Kaiser%2C+Matthias+Plappert%2C+Jerry+Tworek%2C+Jacob+Hilton%2C+Reiichiro+Nakano%2C+et+al%2E+Training+verifiers+to+solve+math+word+prob) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
12. Katherine M Collins, Albert Q Jiang, Simon Frieder, Lionel Wong, Miri Zilka, Umang Bhatt, Thomas Lukasiewicz, Yuhuai Wu, Joshua B Tenenbaum, William Hart, et al. Evaluating language models for mathematics through interactions. arXiv preprint arXiv:2306.01694, 2023. [find](https://scholar.google.com/scholar?q=Katherine+M+Collins%2C+Albert+Q+Jiang%2C+Simon+Frieder%2C+Lionel+Wong%2C+Miri+Zilka%2C+Umang+Bhatt%2C+Thomas+Lukasiewicz%2C+Yuhuai+Wu%2C+Joshua+B+Tenenbaum%2C+William+Hart%2C+et+al%2E+Evaluating+language+models+for+mathema)
13. István Endrédy and Attila Novák. More effective boilerplate removal-the goldminer algorithm. Polibits, 48:79–83, 12 2013. doi: 10.17562/PB-48-10. — [doi:10.17562/pb-48-10](https://doi.org/10.17562/pb-48-10) [find](https://scholar.google.com/scholar?q=Istva%CC%81n+Endre%CC%81dy+and+Attila+Nova%CC%81k%2E+More+effective+boilerplate+removal%2Dthe+goldminer+algorithm%2E+Polibits%2C+48%3A79%E2%80%9383%2C+12+2013%2E+doi%3A+10%2E17562%2FPB%2D48%2D10%2E+%E2%80%94+%5Bdoi%3A10%2E17562%2Fpb%2D48%2D10%5D%28https%3A%2F%2Fdoi%2Eorg%2F10%2E17562%2F)
14. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+et+al%2E+The+pile%3A+An+800gb+dataset+of+diverse+text+for+language+)
15. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III au2, and Kate Crawford. Datasheets for datasets, 2021. [find](https://scholar.google.com/scholar?q=Timnit+Gebru%2C+Jamie+Morgenstern%2C+Briana+Vecchione%2C+Jennifer+Wortman+Vaughan%2C+Hanna+Wallach%2C+Hal+Daume%CC%81+III+au2%2C+and+Kate+Crawford%2E+Datasheets+for+datasets%2C+2021%2E)
16. Xinyang Geng and Hao Liu. Openllama: An open reproduction of llama, May 2023. URL [https://github.com/openlm-research/open_llama](https://github.com/openlm-research/open_llama). [↗](https://github.com/openlm-research/open_llama)
17. Kenneth Heafield. Kenlm: Faster and smaller language model queries. In Proceedings of the sixth workshop on statistical machine translation, pp. 187–197, 2011. [find](https://scholar.google.com/scholar?q=Kenneth+Heafield%2E+Kenlm%3A+Faster+and+smaller+language+model+queries%2E+In+Proceedings+of+the+sixth+workshop+on+statistical+machine+translation%2C+pp%2E+187%E2%80%93197%2C+2011%2E) [our card](https://md.research.elamur.ai/free/en/papers/heafield-2011-kenlm-faster-smaller-language-model/)
18. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2C+Collin+Burns%2C+Steven+Basart%2C+Andy+Zou%2C+Mantas+Mazeika%2C+Dawn+Song%2C+and+Jacob+Steinhardt%2E+Measuring+massive+multitask+language+understanding%2E+arXiv+preprint+arXiv%3A2009%2E03300%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
19. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. CoRR, abs/2103.03874, 2021. URL https://arxiv.org/abs/2103.03874. [↗](https://arxiv.org/abs/2103.03874) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/)
20. Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. Fasttext.zip: Compressing text classification models. arXiv preprint arXiv:1612.03651, 2016. [find](https://scholar.google.com/scholar?q=Armand+Joulin%2C+Edouard+Grave%2C+Piotr+Bojanowski%2C+Matthijs+Douze%2C+He%CC%81rve+Je%CC%81gou%2C+and+Tomas+Mikolov%2E+Fasttext%2Ezip%3A+Compressing+text+classification+models%2E+arXiv+preprint+arXiv%3A1612%2E03651%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/joulin-2016-fasttext-zip-compressing-text-classification/)
21. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843–3857, 2022. [find](https://scholar.google.com/scholar?q=Aitor+Lewkowycz%2C+Anders+Andreassen%2C+David+Dohan%2C+Ethan+Dyer%2C+Henryk+Michalewski%2C+Vinay+Ramasesh%2C+Ambrose+Slone%2C+Cem+Anil%2C+Imanol+Schlag%2C+Theo+Gutman%2DSolo%2C+et+al%2E+Solving+quantitative+reasoning+problem) [our card](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/)
22. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. CoRR, abs/2305.20050, 2023. doi: 10.48550/arXiv.2305.20050. URL https://doi.org/10.48550/arXiv.2305.20050. [↗](https://doi.org/10.48550/arXiv.2305.20050) [our card](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/)
23. Gurmeet Singh Manku, Arvind Jain, and Anish Das Sarma. Detecting near-duplicates for web crawling. In Proceedings of the 16th International Conference on World Wide Web, WWW ’07, pp. 141–150, New York, NY, USA, 2007. Association for Computing Machinery. ISBN 9781595936547. doi: 10.1145/1242572.12425… [find](https://scholar.google.com/scholar?q=Gurmeet+Singh+Manku%2C+Arvind+Jain%2C+and+Anish+Das+Sarma%2E+Detecting+near%2Dduplicates+for+web+crawling%2E+In+Proceedings+of+the+16th+International+Conference+on+World+Wide+Web%2C+WWW+%E2%80%9907%2C+pp%2E+141%E2%80%93150%2C+New+York)
24. Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, et al. Lila: A unified benchmark for mathematical reasoning. arXiv preprint arXiv:2210.17517, 2022. [find](https://scholar.google.com/scholar?q=Swaroop+Mishra%2C+Matthew+Finlayson%2C+Pan+Lu%2C+Leonard+Tang%2C+Sean+Welleck%2C+Chitta+Baral%2C+Tanmay+Rajpurohit%2C+Oyvind+Tafjord%2C+Ashish+Sabharwal%2C+Peter+Clark%2C+et+al%2E+Lila%3A+A+unified+benchmark+for+mathematical) [our card](https://md.research.elamur.ai/free/en/papers/mishra-2022-lila-unified-benchmark-mathematical-reasoning/)
25. Chenghao Mou, Chris Ha, Kenneth Enevoldsen, and Peiyuan Liu. Chenghaomou/text-dedup: Reference snapshot, September 2023. URL https://doi.org/10.5281/zenodo.8364980. — [doi:10.5281/zenodo.8364980](https://doi.org/10.5281/zenodo.8364980) [↗](https://doi.org/10.5281/zenodo.8364980)
26. OpenAI. Gpt-4 technical report, 2023. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Gpt%2D4+technical+report%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
27. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon LLM: outperforming curated corpora with web data, and web data only. CoRR, abs/2306.01116, 2023. doi… [find](https://scholar.google.com/scholar?q=Guilherme+Penedo%2C+Quentin+Malartic%2C+Daniel+Hesslow%2C+Ruxandra+Cojocaru%2C+Alessandro+Cappelli%2C+Hamza+Alobeidli%2C+Baptiste+Pannier%2C+Ebtesam+Almazrouei%2C+and+Julien+Launay%2E+The+refinedweb+dataset+for+falcon+)
28. Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem proving. arXiv preprint arXiv:2009.03393, 2020. [find](https://scholar.google.com/scholar?q=Stanislas+Polu+and+Ilya+Sutskever%2E+Generative+language+modeling+for+automated+theorem+proving%2E+arXiv+preprint+arXiv%3A2009%2E03393%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/polu-2020-generative-language-modeling-automated-theorem/)
29. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-S… [find](https://scholar.google.com/scholar?q=Jack+W%2E+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+H%2E+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+Eliza+Rutherford%2C+Tom+Hennigan%2C+Jacob+Menick)
30. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J+Liu%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext+trans)
31. Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https://arxiv.org/abs/1908.10084. [↗](https://arxiv.org/abs/1908.10084)
32. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Thibaut+Lavril%2C+Gautier+Izacard%2C+Xavier+Martinet%2C+Marie%2DAnne+Lachaux%2C+Timothe%CC%81e+Lacroix%2C+Baptiste+Rozie%CC%80re%2C+Naman+Goyal%2C+Eric+Hambro%2C+Faisal+Azhar%2C+Aure%CC%81lien+Rodriguez%2C+Armand+Joulin%2C+Ed)
33. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023b. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Thibaut+Lavril%2C+Gautier+Izacard%2C+Xavier+Martinet%2C+Marie%2DAnne+Lachaux%2C+Timothe%CC%81e+Lacroix%2C+Baptiste+Rozie%CC%80re%2C+Naman+Goyal%2C+Eric+Hambro%2C+Faisal+Azhar%2C+et+al%2E+Llama%3A+Open+and+efficient+found)
34. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+Dan+Bikel%2C+Lukas+Blecher%2C+Cristian+Canton%2DFe)
35. Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers, 2020. [find](https://scholar.google.com/scholar?q=Wenhui+Wang%2C+Furu+Wei%2C+Li+Dong%2C+Hangbo+Bao%2C+Nan+Yang%2C+and+Ming+Zhou%2E+Minilm%3A+Deep+self%2Dattention+distillation+for+task%2Dagnostic+compression+of+pre%2Dtrained+transformers%2C+2020%2E)
36. Sean Welleck, Jiacheng Liu, Ronan Le Bras, Hannaneh Hajishirzi, Yejin Choi, and Kyunghyun Cho. Naturalproofs: Mathematical theorem proving in natural language. arXiv preprint arXiv:2104.01112, 2021. [find](https://scholar.google.com/scholar?q=Sean+Welleck%2C+Jiacheng+Liu%2C+Ronan+Le+Bras%2C+Hannaneh+Hajishirzi%2C+Yejin+Choi%2C+and+Kyunghyun+Cho%2E+Naturalproofs%3A+Mathematical+theorem+proving+in+natural+language%2E+arXiv+preprint+arXiv%3A2104%2E01112%2C+2021%2E)
37. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359, 2019. [find](https://scholar.google.com/scholar?q=Guillaume+Wenzek%2C+Marie%2DAnne+Lachaux%2C+Alexis+Conneau%2C+Vishrav+Chaudhary%2C+Francisco+Guzma%CC%81n%2C+Armand+Joulin%2C+and+Edouard+Grave%2E+Ccnet%3A+Extracting+high+quality+monolingual+datasets+from+web+crawl+data%2E+a) [our card](https://md.research.elamur.ai/free/en/papers/wenzek-2020-ccnet-extracting-high-quality-monolingual/)
38. Method Runtime (s) Resiliparse HTML-Text Inscripts BoilerPy jusText HTML2Text BeautifulSoup Trafilatura ExtractNet 3.99 10.75 19.14 24.94 31.17 37.17 38.42 63.90 299.67 [https://github.com/chatnoir-eu/chatnoir-resiliparse](https://github.com/chatnoir-eu/chatnoir-resiliparse) https://github.com/TeamHG-Memex/html-text https://github.com/weblyzard/inscript… [↗](https://github.com/chatnoir-eu/chatnoir-resiliparse)