---
title: "Unearthing Large Scale Domain-Specific Knowledge from Public"
description: "The authors propose Retrieve-from-CC, a largely automatic alternative. 61 references, cited by 10 papers here."
authors: ["Zhaoye Fei", "Yunfan Shao", "Linyang Li", "Zhiyuan Zeng", "Conghui He", "Qipeng Guo", "Hang Yan", "Dahua Lin", "Xipeng Qiu"]
year: 2024
venue: "arXiv:2401.14624"
source: "https://arxiv.org/abs/2401.14624"
doi: "10.48550/arxiv.2401.14624"
arxiv: "2401.14624"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/fei-2024-unearthing-large-scale-domain-specific/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/fei-2024-unearthing-large-scale-domain-specific/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/) / Unearthing Large Scale Domain-Specific Knowledge from Public Corpora

# Unearthing Large Scale Domain-Specific Knowledge from Public Corpora

**Authors:** [Zhaoye Fei](https://md.research.elamur.ai/free/en/authors/zhaoye-fei/), [Yunfan Shao](https://md.research.elamur.ai/free/en/authors/yunfan-shao/), [Linyang Li](https://md.research.elamur.ai/free/en/authors/linyang-li/), [Zhiyuan Zeng](https://md.research.elamur.ai/free/en/authors/zhiyuan-zeng/), [Conghui He](https://md.research.elamur.ai/free/en/authors/conghui-he/), [Qipeng Guo](https://md.research.elamur.ai/free/en/authors/qipeng-guo/), [Hang Yan](https://md.research.elamur.ai/free/en/authors/hang-yan/), [Dahua Lin](https://md.research.elamur.ai/free/en/authors/dahua-lin/) and [Xipeng Qiu](https://md.research.elamur.ai/free/en/authors/xipeng-qiu/) · **Year:** 2024 · **Venue:** arXiv:2401.14624 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2401.14624) · [PDF](https://arxiv.org/pdf/2401.14624) · [DOI 10.48550/arxiv.2401.14624](https://doi.org/10.48550/arxiv.2401.14624) · [arXiv 2401.14624](https://arxiv.org/abs/2401.14624) · [OpenAlex](https://openalex.org/W4391335099)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [10](https://md.research.elamur.ai/free/en/papers/fei-2024-unearthing-large-scale-domain-specific/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Building domain-specific corpora for LLMs usually requires experts to pick sources and engineers to scrape them. This takes months and handles poorly the knowledge that is scattered thinly across the web. The authors propose Retrieve-from-CC, a largely automatic alternative. An LLM receives a few seed keywords per domain and iteratively writes new related questions (Question Extension, for breadth). It then answers them with step-by-step reasoning (Thought Generation, for depth). Both the questions and the reasoning serve as search queries. Generated answers may contain errors, but they only need to locate genuine web text. BM25 search over a cleaned Common Crawl index returns candidate documents, which are then quality-filtered and deduplicated. The result is Retrieve-Pile, covering STEM, humanities, social sciences, medical sciences and general knowledge. Continued pre-training of Llama and Mistral on it improves mathematics, knowledge, professional-exam and reasoning benchmarks. The method offers a cheap, scalable way to assemble topical corpora from public crawls.

## Key points
- Released dataset: **Retrieve-Pile** on Hugging Face (`Query-of-CC/Retrieve-Pile`). The enhanced models are called Llama-QoC and Mistral-QoC.
- Retrieval index: Common Crawl WARC data from 2016–2023, cleaned into several billion records (~50TB). It is served by Elasticsearch with BM25, returns the top 1000 documents per query, and answers each query in about 100ms.
- Post-processing: a BERT-base-uncased quality classifier keeps documents scoring at least 0.8. MinHash-LSH deduplication uses 13-grams, a similarity threshold of 0.8, and num_perm = 128.
- Llama gains at least 10 points on average in every MMLU category. Mistral gains less, mostly in STEM (53.23 → 56.6), while other categories improve by about one point.
- AGIEval results for Mistral-QoC: SAT-Math rises from 31.36 to 45 and LogiQA from 39.32 to 45.01. SAT-English drops slightly (68.93 → 65.53).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) — University, Shanghai, China · 167 works · 20,987 citations
- [Shanghai Artificial Intelligence Laboratory](https://md.research.elamur.ai/free/en/labs/shanghai-artificial-intelligence-laboratory/) — Research institute, Shanghai, China · 117 works · 11,281 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

## Cite

Cite as: Zhaoye Fei, Yunfan Shao, Linyang Li, Zhiyuan Zeng, Conghui He, Qipeng Guo, Hang Yan, Dahua Lin and Xipeng Qiu. 2024. Unearthing Large Scale Domain-Specific Knowledge from Public Corpora. arXiv:2401.14624.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{fei2024unearthing,
  title = {{Unearthing Large Scale Domain-Specific Knowledge from Public Corpora}},
  author = {Zhaoye Fei, Yunfan Shao, Linyang Li, Zhiyuan Zeng, Conghui He, Qipeng Guo, Hang Yan, Dahua Lin and Xipeng Qiu},
  year = {2024},
  doi = {10.48550/arxiv.2401.14624},
  eprint = {2401.14624},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.14624}
}
```

- [← previous Towards Zero-Shot Text-To-Speech for Arabic Dialects](https://md.research.elamur.ai/free/en/papers/doan-2024-zero-shot-text-speech-arabic/)
- [next → Usefulness of LLMs as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experiment](https://md.research.elamur.ai/free/en/papers/goldberg-2024-usefulness-llms-author-checklist-assistant/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/)

## References

61 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. [Llemma: An open language model for mathematics](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-llemma-open-language-model-mathematics/). In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria…
2. OpenReview.net, 2024. URL [https://openreview.net/forum?id=4WnqRR915j](https://openreview.net/forum?id=4WnqRR915j). [↗](https://openreview.net/forum?id=4WnqRR915j)
3. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, P… — [Qwen Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/)
4. Andrei Z. Broder. On the resemblance and containment of documents. In Bruno Carpentieri, Alfredo De Santis, Ugo Vaccaro, and James A. Storer (eds.), Compression and Complexity of SEQUENCES 1997, Positano, Amalfitan Coast, Salerno, Italy, June 11-13, 1997, Proceedings, pp. 21–29. [find](https://scholar.google.com/scholar?q=Andrei+Z%2E+Broder%2E+On+the+resemblance+and+containment+of+documents%2E+In+Bruno+Carpentieri%2C+Alfredo+De+Santis%2C+Ugo+Vaccaro%2C+and+James+A%2E+Storer+%28eds%2E%29%2C+Compression+and+Complexity+of+SEQUENCES+1997%2C+Posit)
5. IEEE, 1997. doi: 10.1109/SEQUEN.1997.666900. URL https://doi.org/10. 1109/SEQUEN.1997.666900. [find](https://scholar.google.com/scholar?q=IEEE%2C+1997%2E+doi%3A+10%2E1109%2FSEQUEN%2E1997%2E666900%2E+URL+https%3A%2F%2Fdoi%2Eorg%2F10%2E+1109%2FSEQUEN%2E1997%2E666900%2E)
6. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
7. Qiaoling Chen, Qinghao Hu, Zhisheng Ye, Guoteng Wang, Peng Sun, Yonggang Wen, and Tianwei Zhang. AMSP: super-scaling LLM training via advanced model states partitioning. CoRR, abs/2311.00257, 2023. doi: 10.48550/ARXIV.2311.00257. URL https://doi.org/10. 48550/arXiv.2311.00257. [find](https://scholar.google.com/scholar?q=Qiaoling+Chen%2C+Qinghao+Hu%2C+Zhisheng+Ye%2C+Guoteng+Wang%2C+Peng+Sun%2C+Yonggang+Wen%2C+and+Tianwei+Zhang%2E+AMSP%3A+super%2Dscaling+LLM+training+via+advanced+model+states+partitioning%2E+CoRR%2C+abs%2F2311%2E00257%2C+2023%2E+do)
8. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
9. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). CoRR, abs/2110.14168, 2021. URL https://arxiv.org/abs/2110.1416… [↗](https://arxiv.org/abs/2110.1416…)
10. Together Computer. [Redpajama: an open dataset for training large language models](https://md.research.elamur.ai/free/en/papers/weber-2024-redpajama-open-dataset-training-large/), 2023. URL [https://github.com/togethercomputer/RedPajama-Data](https://github.com/togethercomputer/RedPajama-Data). [↗](https://github.com/togethercomputer/RedPajama-Data)
11. Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. Chatlaw: Open-source legal large language model with integrated external knowledge bases. CoRR, abs/2306.[16092, 2023](https://md.research.elamur.ai/free/en/papers/cui-2023-chatlaw-multi-agent-legal-assistant/). doi: 10. 48550/ARXIV.2306.16092. URL https://doi.org/10.48550/arXiv.2306.16092. [↗](https://doi.org/10.48550/arXiv.2306.16092)
12. Tri Dao. [Flashattention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024.
13. OpenReview.net, 2024. URL [https://openreview.net/forum?id=](https://openreview.net/forum?id=) mZn2Xyh9Ec. [↗](https://openreview.net/forum?id=)
14. Tim Dettmers. openassistant-guanaco, May 2023. URL [https://huggingface.co/](https://huggingface.co/) datasets/timdettmers/openassistant-guanaco. [↗](https://huggingface.co/)
15. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101.00027, 2021. URL https://arxiv.org/abs/2101.00027… [↗](https://arxiv.org/abs/2101.00027…)
16. Weihao Gao, Zhuo Deng, Zhiyuan Niu, Fuju Rong, Chucheng Chen, Zheng Gong, Wenze Zhang, Daimin Xiao, Fang Li, Zhenjie Cao, Zhaoyi Ma, Wenbin Wei, and Lan Ma. Ophglm: Training an ophthalmology large language-and-vision assistant based on instructions and dialogue. CoRR, abs/2306.12174, 2023. doi: 10.4… — [doi:10.48550/arxiv.2306.12174](https://doi.org/10.48550/arxiv.2306.12174) [find](https://scholar.google.com/scholar?q=Weihao+Gao%2C+Zhuo+Deng%2C+Zhiyuan+Niu%2C+Fuju+Rong%2C+Chucheng+Chen%2C+Zheng+Gong%2C+Wenze+Zhang%2C+Daimin+Xiao%2C+Fang+Li%2C+Zhenjie+Cao%2C+Zhaoyi+Ma%2C+Wenbin+Wei%2C+and+Lan+Ma%2E+Ophglm%3A+Training+an+ophthalmology+large+lan)
17. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanz… [find](https://scholar.google.com/scholar?q=Suriya+Gunasekar%2C+Yi+Zhang%2C+Jyoti+Aneja%2C+Caio+Ce%CC%81sar+Teodoro+Mendes%2C+Allie+Del+Giorno%2C+Sivakanth+Gopi%2C+Mojan+Javaheripi%2C+Piero+Kauffmann%2C+Gustavo+de+Rosa%2C+Olli+Saarikivi%2C+Adil+Salim%2C+Shital+Shah%2C+Hark)
18. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
19. OpenReview.net, 2021a. URL [https://openreview.net/forum?id=d7KBjmI3GmQ](https://openreview.net/forum?id=d7KBjmI3GmQ). [↗](https://openreview.net/forum?id=d7KBjmI3GmQ)
20. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the MATH dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Dataset…
21. InternLM-Team. Internlm: A multilingual language model with progressively enhanced capabilities. [https://github.com/InternLM/InternLM](https://github.com/InternLM/InternLM), 2023. [↗](https://github.com/InternLM/InternLM)
22. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroi… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+Las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+Le%CC%81lio+Renard+La)
23. Minhao Jiang, Ken Ziyu Liu, Ming Zhong, Rylan Schaeffer, Siru Ouyang, Jiawei Han, and Sanmi Koyejo. Investigating data contamination for pre-training language models. CoRR, abs/2401.06059, 2024. doi: 10.48550/ARXIV.2401.06059. URL https://doi.org/10.48550/arXiv. 2401.06059. — [doi:10.48550/arxiv.2401.06059](https://doi.org/10.48550/arxiv.2401.06059) [↗](https://doi.org/10.48550/arXiv)
24. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. [Dense passage retrieval for open-domain question answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/). In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in…
25. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/). In NeurIPS, 2022. UR…
26. Xiaonan Li and Xipeng Qiu. Mot: Memory-of-thought enables chatgpt to self-improve. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 6354–6374. Association for C… [find](https://scholar.google.com/scholar?q=Xiaonan+Li+and+Xipeng+Qiu%2E+Mot%3A+Memory%2Dof%2Dthought+enables+chatgpt+to+self%2Dimprove%2E+In+Houda+Bouamor%2C+Juan+Pino%2C+and+Kalika+Bali+%28eds%2E%29%2C+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+Natur)
27. Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. [Unified demonstration retriever for in-context learning](https://md.research.elamur.ai/free/en/papers/li-2023-unified-demonstration-retriever-context-learning/). In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computati…
28. Ha-Thanh Nguyen. A brief report on lawgpt 1.0: A virtual legal assistant based on GPT-3. CoRR, abs/2302.05729, 2023. doi: 10.48550/ARXIV.2302.05729. URL https://doi.org/10. 48550/arXiv.2302.05729. [find](https://scholar.google.com/scholar?q=Ha%2DThanh+Nguyen%2E+A+brief+report+on+lawgpt+1%2E0%3A+A+virtual+legal+assistant+based+on+GPT%2D3%2E+CoRR%2C+abs%2F2302%2E05729%2C+2023%2E+doi%3A+10%2E48550%2FARXIV%2E2302%2E05729%2E+URL+https%3A%2F%2Fdoi%2Eorg%2F10%2E+48550%2FarXiv%2E2302%2E05729%2E)
29. OpenAI. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). CoRR, abs/2303.08774, 2023. doi: 10.48550/ARXIV.2303.08774. URL https://doi.org/10.48550/arXiv.2303.08774. [↗](https://doi.org/10.48550/arXiv.2303.08774)
30. Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. [Openwebmath: An open dataset of high-quality mathematical web text](https://md.research.elamur.ai/free/en/papers/paster-2023-openwebmath-open-dataset-high-quality/). In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024.
31. OpenReview.net, 2024. URL [https://openreview.net/forum?id=jKHmjlpViu](https://openreview.net/forum?id=jKHmjlpViu). [↗](https://openreview.net/forum?id=jKHmjlpViu)
32. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). J. Mach. Learn. Res., 21:140:1–140:67, 2020. URL [http://jmlr.org/](http://jmlr.org/) papers/v21/20-074.html. [↗](http://jmlr.org/)
33. Stephen E. Robertson and Steve Walker. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In W. Bruce Croft and C. J. van Rijsbergen (eds.), Proceedings of the 17th Annual International ACM-SIGIR Conference on Research and Development in Information Ret… — [doi:10.1007/978-1-4471-2099-5\](https://doi.org/10.1007/978-1-4471-2099-5\) [find](https://scholar.google.com/scholar?q=Stephen+E%2E+Robertson+and+Steve+Walker%2E+Some+simple+effective+approximations+to+the+2%2Dpoisson+model+for+probabilistic+weighted+retrieval%2E+In+W%2E+Bruce+Croft+and+C%2E+J%2E+van+Rijsbergen+%28eds%2E%29%2C+Proceedings+)
34. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Fai… — [Code Llama: Open Foundation Models for Code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)
35. Tianxiang Sun, Xiaotian Zhang, Zhengfu He, Peng Li, Qinyuan Cheng, Hang Yan, Xiangyang Liu, Yunfan Shao, Qiong Tang, Xingjian Zhao, Ke Chen, Yining Zheng, Zhejian Zhou, Ruixiao Li, Jun Zhan, Yunhua Zhou, Linyang Li, Xiaogui Yang, Lingling Wu, Zhangyue Yin, Xuanjing Huang, and Xipeng Qiu. Moss: Train… [find](https://scholar.google.com/scholar?q=Tianxiang+Sun%2C+Xiaotian+Zhang%2C+Zhengfu+He%2C+Peng+Li%2C+Qinyuan+Cheng%2C+Hang+Yan%2C+Xiangyang+Liu%2C+Yunfan+Shao%2C+Qiong+Tang%2C+Xingjian+Zhao%2C+Ke+Chen%2C+Yining+Zheng%2C+Zhejian+Zhou%2C+Ruixiao+Li%2C+Jun+Zhan%2C+Yunhua+Zh)
36. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed Chi, Denny Zhou, and Jason Wei. [Challenging big-bench tasks and whether chain-of-thought can solve them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/). In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.),…
37. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. [Galactica: A large language model for science](https://md.research.elamur.ai/free/en/papers/taylor-2022-galactica-large-language-model-science/). CoRR, abs/2211.09085, 2022. doi: 10.48550/ARXIV.2211.09085. URL https://doi.org/10.48550/arXiv.2211.09085… [↗](https://doi.org/10.48550/arXiv.2211.09085…)
38. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). CoRR, abs/2…
39. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
40. Ken Tsui. llm-data-textbook-quality-fasttext-classifier-v2, 2024. URL [https://huggingface.co/kenhktsui/llm-data-textbook-quality-fasttext-classifier-v2](https://huggingface.co/kenhktsui/llm-data-textbook-quality-fasttext-classifier-v2). [↗](https://huggingface.co/kenhktsui/llm-data-textbook-quality-fasttext-classifier-v2)
41. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023.
42. OpenReview.net, 2023a. URL [https://openreview.net/](https://openreview.net/) pdf?id=1PL1NIMMrw. [↗](https://openreview.net/)
43. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the A… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A%2E+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+Self%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%2E+In+Anna+Roger)
44. Zengzhi Wang, Rui Xia, and Liu Pengfei. Generative ai for math: Part i – mathpile: A billion-token-scale pretraining corpus for math. arXiv preprint arXiv:2312.17120, 2023c. [find](https://scholar.google.com/scholar?q=Zengzhi+Wang%2C+Rui+Xia%2C+and+Liu+Pengfei%2E+Generative+ai+for+math%3A+Part+i+%E2%80%93+mathpile%3A+A+billion%2Dtoken%2Dscale+pretraining+corpus+for+math%2E+arXiv+preprint+arXiv%3A2312%2E17120%2C+2023c%2E)
45. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In NeurIPS, 2022. URL [http://papers.nips.cc/paper_files/paper/2022/hash/](http://papers.nips.cc/paper_files/paper/2022/hash/) 9d5609613524ecf4f15af0f7b31abca4-Abstr… [↗](http://papers.nips.cc/paper_files/paper/2022/hash/)
46. Shufa Wei, Xiaolong Xu, Xianbiao Qi, Xi Yin, Jun Xia, Jingyi Ren, Peijun Tang, Yuxiang Zhong, Yihao Chen, Xiaoqin Ren, Yuxin Liang, Liankai Huang, Kai Xie, Weikang Gui, Wei Tan, Shuanglong Sun, Yongquan Hu, Qinxian Liu, Nanjin Li, Chihao Dai, Lihua Wang, Xiaohui Liu, Lei Zhang, and Yutao Xie. Academ… — [doi:10.48550/arxiv.2311.12315](https://doi.org/10.48550/arxiv.2311.12315) [find](https://scholar.google.com/scholar?q=Shufa+Wei%2C+Xiaolong+Xu%2C+Xianbiao+Qi%2C+Xi+Yin%2C+Jun+Xia%2C+Jingyi+Ren%2C+Peijun+Tang%2C+Yuxiang+Zhong%2C+Yihao+Chen%2C+Xiaoqin+Ren%2C+Yuxin+Liang%2C+Liankai+Huang%2C+Kai+Xie%2C+Weikang+Gui%2C+Wei+Tan%2C+Shuanglong+Sun%2C+Yongqu)
47. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. [Ccnet: Extracting high quality monolingual datasets from web crawl data](https://md.research.elamur.ai/free/en/papers/wenzek-2020-ccnet-extracting-high-quality-monolingual/). In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thier…
48. European Language Resources Association, 2020. [URL https://aclanthology.org/2020.lrec-1.494/](https://md.research.elamur.ai/free/en/papers/wenzek-2020-ccnet-extracting-high-quality-monolingual/). [↗](https://aclanthology.org/2020.lrec-1.494/)
49. Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. Qurating: Selecting high-quality data for training language models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. [find](https://scholar.google.com/scholar?q=Alexander+Wettig%2C+Aatmik+Gupta%2C+Saumya+Malik%2C+and+Danqi+Chen%2E+Qurating%3A+Selecting+high%2Dquality+data+for+training+language+models%2E+In+Forty%2Dfirst+International+Conference+on+Machine+Learning%2C+ICML+2024)
50. OpenReview.net, 2024. URL [https://openreview.net/forum?id=GLGYYqPwjy](https://openreview.net/forum?id=GLGYYqPwjy). [↗](https://openreview.net/forum?id=GLGYYqPwjy)
51. Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David S. Rosenberg, and Gideon Mann. [Bloomberggpt: A large language model for finance](https://md.research.elamur.ai/free/en/papers/wu-2023-bloomberggpt-large-language-model-finance/). CoRR, abs/2303.17564, 2023. doi: 10.48550/ARXIV.2303.17564. URL https://doi.org/10.48550/arXiv.2303.17564. [↗](https://doi.org/10.48550/arXiv.2303.17564)
52. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. [Wizardlm: Empowering large pre-trained language models to follow complex instructions](https://md.research.elamur.ai/free/en/papers/xu-2023-wizardlm-empowering-large-pre-trained/). In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May…
53. OpenReview.net, 2024. URL [https://openreview.net/](https://openreview.net/) forum?id=CfXh93NDgH. [↗](https://openreview.net/)
54. Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang, Fei Deng, Feng Wang, Feng Liu, Guangwei Ai, Guosheng Dong, Haizhou Zhao, Hang Xu, Haoze Sun, Hongda Zhang, Hui Liu, Jiaming Ji, Jian Xie, Juntao Dai, Kun Fang, Lei Su, Liang Song,… — [Baichuan 2: Open Large-scale Language Models](https://md.research.elamur.ai/free/en/papers/yang-2023-baichuan-2-open-large-scale/)
55. Xi Yang, Nima M. Pournejatian, Hoo Chang Shin, Kaleb E. Smith, Christopher Parisien, Colin Compas, Cheryl Martin, Mona G. Flores, Ying Zhang, Tanja Magoc, Christopher A. Harle, Gloria P. Lipori, Duane A. Mitchell, William R. Hogan, Elizabeth A. Shenkman, Jiang Bian, and Yonghui Wu. Gatortron: A larg… — [doi:10.48550/arxiv.2203.03540](https://doi.org/10.48550/arxiv.2203.03540) [find](https://scholar.google.com/scholar?q=Xi+Yang%2C+Nima+M%2E+Pournejatian%2C+Hoo+Chang+Shin%2C+Kaleb+E%2E+Smith%2C+Christopher+Parisien%2C+Colin+Compas%2C+Cheryl+Martin%2C+Mona+G%2E+Flores%2C+Ying+Zhang%2C+Tanja+Magoc%2C+Christopher+A%2E+Harle%2C+Gloria+P%2E+Lipori%2C+Duane)
56. Xingcheng Yao, Yanan Zheng, Xiaocong Yang, and Zhilin Yang. NLP from scratch without large-scale pretraining: A simple and efficient framework. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2… [find](https://scholar.google.com/scholar?q=Xingcheng+Yao%2C+Yanan+Zheng%2C+Xiaocong+Yang%2C+and+Zhilin+Yang%2E+NLP+from+scratch+without+large%2Dscale+pretraining%3A+A+simple+and+efficient+framework%2E+In+Kamalika+Chaudhuri%2C+Stefanie+Jegelka%2C+Le+Song%2C+Csaba+)
57. PMLR, 2022. URL [https://proceedings.mlr.press/v162/yao22c.html](https://proceedings.mlr.press/v162/yao22c.html). [↗](https://proceedings.mlr.press/v162/yao22c.html)
58. Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. [Mammoth2: Scaling instructions from the web](https://md.research.elamur.ai/free/en/papers/yue-2024-mammoth2-scaling-instructions-web/), 2024.
59. Xuanyu Zhang and Qing Yang. [Xuanyuan 2.0: A large chinese financial chat model with hundreds of billions parameters](https://md.research.elamur.ai/free/en/papers/zhang-2023-xuanyuan-2-0-large-chinese/). In Ingo Frommholz, Frank Hopfgartner, Mark Lee, Michael Oakes, Mounia Lalmas, Min Zhang, and Rodrygo L. T. Santos (eds.), Proceedings of the 32nd ACM International Conference on Infor…
60. ACM, 2023. doi: 10.1145/3583780.3615285. URL https://doi.org/10.1145/3583780.3615285. — [doi:10.1145/3583780.3615285](https://doi.org/10.1145/3583780.3615285) [↗](https://doi.org/10.1145/3583780.3615285)
61. Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. [Agieval: A human-centric benchmark for evaluating foundation models](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/). In Kevin Duh, Helena Gómez-Adorno, and Steven Bethard (eds.), Findings of the Association for Computational Linguist…

*Updated 31 October 2026*