--- title: "BaichuanSEED — paper by Dong et al. (2024)" description: "Pre-training corpora largely determine what a large language model can do. 2 references, cited by 2 papers here." authors: ["Guosheng Dong", "Da Pan", "Yiding Sun", "Shusen Zhang", "Zheng Liang", "Xin Wu", "Yanjun Shen", "Fan Yang", "Haoze Sun", "Tianpeng Li", "Mingan Lin", "Jianhua Xu", "Yufan Zhang", "Xiaonan Nie", "Lei Su", "Bingning Wang", "Wentao Zhang", "Jiaxin Mao", "Zenan Zhou", "Weipeng Chen"] year: 2024 venue: "arXiv:2408.15079" source: "https://arxiv.org/abs/2408.15079" doi: "10.48550/arxiv.2408.15079" arxiv: "2408.15079" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/dong-2024-baichuanseed-sharing-potential-extensive/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/dong-2024-baichuanseed-sharing-potential-extensive/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/12/) / BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and Deduplication by Introducing a Competitive Large Language Model Baseline # BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and Deduplication by Introducing a Competitive Large Language Model Baseline **Authors:** [Guosheng Dong](https://md.research.elamur.ai/free/en/authors/guosheng-dong/), [Da Pan](https://md.research.elamur.ai/free/en/authors/da-pan/), [Yiding Sun](https://md.research.elamur.ai/free/en/authors/yiding-sun/), Shusen Zhang, [Zheng Liang](https://md.research.elamur.ai/free/en/authors/zheng-liang/), [Xin Wu](https://md.research.elamur.ai/free/en/authors/xin-wu/), [Yanjun Shen](https://md.research.elamur.ai/free/en/authors/yanjun-shen/), [Fan Yang](https://md.research.elamur.ai/free/en/authors/fan-yang/), [Haoze Sun](https://md.research.elamur.ai/free/en/authors/haoze-sun/), [Tianpeng Li](https://md.research.elamur.ai/free/en/authors/tianpeng-li/), [Mingan Lin](https://md.research.elamur.ai/free/en/authors/mingan-lin/), [Jianhua Xu](https://md.research.elamur.ai/free/en/authors/jianhua-xu/), Yufan Zhang, Xiaonan Nie, Lei Su, [Bingning Wang](https://md.research.elamur.ai/free/en/authors/bingning-wang/), [Wentao Zhang](https://md.research.elamur.ai/free/en/authors/wentao-zhang/), Jiaxin Mao, Zenan Zhou and [Weipeng Chen](https://md.research.elamur.ai/free/en/authors/weipeng-chen/) · **Year:** 2024 · **Venue:** arXiv:2408.15079 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2408.15079) · [PDF](https://arxiv.org/pdf/2408.15079) · [DOI 10.48550/arxiv.2408.15079](https://doi.org/10.48550/arxiv.2408.15079) · [arXiv 2408.15079](https://arxiv.org/abs/2408.15079) · [OpenAlex](https://openalex.org/W4402705130) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/dong-2024-baichuanseed-sharing-potential-extensive/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Pre-training corpora largely determine what a large language model can do. Yet many developers, including some that release open weights, keep their data collection and mixing recipes private. Some models are also tuned toward particular benchmarks, which hides their real abilities. The Baichuan team addresses this by publishing a complete data pipeline built on two stages: - **Broad collection** from trusted public sources: CommonCrawl web pages, knowledge-intensive books and papers, code, and domain data. Synthetic data and benchmark material are deliberately left out. - **Reweighting** through a global multi-granularity deduplication scheme that works at document and sentence level and also removes PII and harmful content. Fine-grained data selection is intentionally skipped. With this data they pretrain BaichuanSEED, a 7B model with a Baichuan2/Llama-style architecture, on 3T bilingual Chinese–English tokens. A simple SFT stage then produces BaichuanSEED-SFT. The model behaves consistently and predictably during training. It matches models such as Qwen1.5 and Llama3 on broad benchmarks but lags on mathematics. Its value is as a transparent, unoptimized baseline for measuring how much specific optimizations actually help. ## Key points - **Architecture:** 32 layers and 32 attention heads, hidden size 4,096, FFN size 11,008, with SwiGLU, RMSNorm and RoPE. - **Data processing:** - More than 10T tokens were gathered, including 94 CommonCrawl batches parsed with Trafilatura and language-tagged with CLD3. - Global deduplication discarded about 88% of CommonCrawl tokens. - **Code data:** about 750B tokens drawn from StarCoder, license-filtered GitHub and StackExchange. Roughly 62% is organized at repository level, with files ordered by dependency in the style of Deepseek-coder. - **Knowledge-intensive data (KID):** experiments on a 2B model found that a 25% KID share gave the lowest perplexity, both when training from scratch (13.68) and in continued pre-training (11.73). This 25% share was adopted for BaichuanSEED. - **Math experiment:** raising math data to 10% during 300B tokens of continued pre-training, starting from a 1.44T checkpoint, improved GSM8K by 7.4% and MATH by 65.6% (likely relative gains). MMLU moved by only about 0.3%. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 244 works · 39,359 citations - [Renmin University of China](https://md.research.elamur.ai/free/en/universities/renmin-university-of-china/) — University, Beijing, China · 114 works · 16,118 citations - [Baichuan](https://md.research.elamur.ai/free/en/companies/baichuan/) — Company, Beijing, China · 7 works · 218 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works ## Cite Cite as: Guosheng Dong, Da Pan, Yiding Sun, Shusen Zhang, Zheng Liang, Xin Wu, Yanjun Shen, Fan Yang, Haoze Sun, Tianpeng Li, Mingan Lin, Jianhua Xu, Yufan Zhang, Xiaonan Nie, Lei Su, Bingning Wang, Wentao Zhang, Jiaxin Mao, Zenan Zhou and Weipeng Chen. 2024. BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and Deduplication by Introducing a Competitive Large Language Model Baseline. arXiv:2408.15079. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{dong2024baichuanseed, title = {{BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and Deduplication by Introducing a Competitive Large Language Model Baseline}}, author = {Guosheng Dong, Da Pan, Yiding Sun, Shusen Zhang, Zheng Liang, Xin Wu, Yanjun Shen, Fan Yang, Haoze Sun, Tianpeng Li, Mingan Lin, Jianhua Xu, Yufan Zhang, Xiaonan Nie, Lei Su, Bingning Wang, Wentao Zhang, Jiaxin Mao, Zenan Zhou and Weipeng Chen}, year = {2024}, doi = {10.48550/arxiv.2408.15079}, eprint = {2408.15079}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2408.15079} } ``` - [← previous BabelBot at AraFinNLP2024: Fine-tuning T5 for Multi-dialect Intent Detection with Synthetic Data and Model Ensembling](https://md.research.elamur.ai/free/en/papers/fares-2024-babelbot-arafinnlp2024-fine-tuning-t5/) - [next → BiasGanda at FIGNEWS 2024 Shared Task: A Quest to Uncover Biased Views in News Coverage](https://md.research.elamur.ai/free/en/papers/wardi-2024-biasganda-fignews-2024-shared-task/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/12/) ## References 2 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. and Dao, 2023), and optimization for vertical downstream tasks (DeepSeek-AI et al., 2024b; Shao et al., 2024). As a baseline model, BaichuanSEED focus on the exploration of the limits of model capabilities through data collection and deduplication, avoiding any specific optimization. Nevertheless, o… [find](https://scholar.google.com/scholar?q=and+Dao%2C+2023%29%2C+and+optimization+for+vertical+downstream+tasks+%28DeepSeek%2DAI+et+al%2E%2C+2024b%3B+Shao+et+al%2E%2C+2024%29%2E+As+a+baseline+model%2C+BaichuanSEED+focus+on+the+exploration+of+the+limits+of+model+capabil) 2. Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat S. Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra,… — [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/) *Updated 31 October 2026*