---
title: "360Zhinao Technical Report — paper (2024)"
description: "The 360Zhinao Team describes how it built 360Zhinao-7B, an open 7B-parameter LLM released in variants with 4K, 32K and… 89 references."
year: 2024
venue: "arXiv:2405.13386"
source: "https://arxiv.org/abs/2405.13386"
doi: "10.48550/arxiv.2405.13386"
arxiv: "2405.13386"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/360zhinao-team-2024-360zhinao-technical-report/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/360zhinao-team-2024-360zhinao-technical-report/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/14/) / 360Zhinao Technical Report

# 360Zhinao Technical Report

**Year:** 2024 · **Venue:** arXiv:2405.13386 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2405.13386) · [PDF](https://arxiv.org/pdf/2405.13386) · [DOI 10.48550/arxiv.2405.13386](https://doi.org/10.48550/arxiv.2405.13386) · [arXiv 2405.13386](https://arxiv.org/abs/2405.13386) · [OpenAlex](https://openalex.org/W4398795285)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The 360Zhinao Team describes how it built 360Zhinao-7B, an open 7B-parameter LLM released in variants with 4K, 32K and 360K context windows. The report addresses two pretraining difficulties: improving the information density of a huge web corpus, and judging data choices reliably with small models, since OpenCompass proved noisy and insensitive at that scale. The team built a four-step web-data pipeline (preparation, cleaning, deduplication, mixing) and used it to produce 3.4T tokens for 360Zhinao-7B-Base. To guide these decisions, they created a small-model ablation setup with custom benchmarks tied to downstream skills (360Eval plus an SFT-based evaluation). Alignment was also data-driven. SFT data was first collected for volume, then refined for quality. Context was extended by changing the RoPE base and training on purpose-built data. RMs and RLHF were stabilised and applied to selected tasks. The resulting chat and base models perform competitively against similarly sized peers. The report matters because it documents intermediate production details that are rarely published, and because the models are openly released.

## Key points
- The models are released at github.com/Qihoo360/360zhinao. The base model was pretrained on 3.4T tokens.
- Document-level preparation removed 12.66% of the data, using URL/keyword black- and whitelists, exact deduplication and FastText language ID. Cleaning then removed a further 53.65%.
- The cleaning tools are:
  - a FastText junk-paragraph classifier (F1 99%);
  - a BERT quality classifier trained on 60k manually labelled samples (F1 85%);
  - regex-based PII masking.
- In ablations, keeping 30% as the paragraph-deduplication level raised the SFT-evaluation average from 47.92 to 53.04. Document deduplication improved SFT scores by 1.47%, compared with only 0.02% on OpenCompass.
- Context extension to 32K and 360K uses a simple RoPE-base change plus tailored data. The trained RM is also reused for data selection and judging.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

## Cite

Cite as: 2024. 360Zhinao Technical Report. arXiv:2405.13386.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{anon2024360zhinao,
  title = {{360Zhinao Technical Report}},
  year = {2024},
  doi = {10.48550/arxiv.2405.13386},
  eprint = {2405.13386},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.13386}
}
```

- [← previous “It was just so hard”: ineffable <em>just</em> as a mixed expressive](https://md.research.elamur.ai/free/en/papers/manetta-2024-it-was-just-so-hard/)
- [next → A Comparison of Conjunctive Cohesion between Pakistani and British English Newspapers’ Opinion Articles](https://md.research.elamur.ai/free/en/papers/khalid-2024-comparison-conjunctive-cohesion-between-pakistani/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/14/)

## References

89 references, 47 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. [A survey on data selection for language models](https://md.research.elamur.ai/free/en/papers/albalak-2024-survey-data-selection-language-models/). arXiv preprint arXiv:2402.16827, 2024.
2. Arize-AI. [https://github.com/Arize-ai/LLMTest_NeedleInAHaystack](https://github.com/Arize-ai/LLMTest_NeedleInAHaystack), 2024. [↗](https://github.com/Arize-ai/LLMTest_NeedleInAHaystack)
3. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732, 2021.
4. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609, 2023a.
5. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+et+al%2E+Training+a+helpful+and+harmless+assistant+with+reinforcem)
6. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. [Longbench: A bilingual, multitask benchmark for long context understanding](https://md.research.elamur.ai/free/en/papers/bai-2023-longbench-bilingual-multitask-benchmark/). arXiv preprint arXiv:2308.14508, 2023b.
7. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024. [find](https://scholar.google.com/scholar?q=Xiao+Bi%2C+Deli+Chen%2C+Guanting+Chen%2C+Shanhuang+Chen%2C+Damai+Dai%2C+Chengqi+Deng%2C+Honghui+Ding%2C+Kai+Dong%2C+Qiushi+Du%2C+Zhe+Fu%2C+et+al%2E+Deepseek+llm%3A+Scaling+open%2Dsource+language+models+with+longtermism%2E+arXiv+)
8. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. [Enriching word vectors with subword information](https://md.research.elamur.ai/free/en/papers/bojanowski-2017-enriching-word-vectors-subword-information/). Transactions of the association for computational linguistics, 5:135–146, 2017.
9. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
10. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv preprint arXiv:2303.12712, 2023.
11. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. [A survey on evaluation of large language models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/). ACM Transactions on Intelligent Systems and Technology, 15(3):1–45, 2024.
12. Daoyuan Chen, Yilun Huang, Zhijian Ma, Hesen Chen, Xuchen Pan, Ce Ge, Dawei Gao, Yuexiang Xie, Zhaoyang Liu, Jinyang Gao, et al. Data-juicer: A one-stop data processing system for large language models. arXiv preprint arXiv:2309.02033, 2023a. — [arXiv:2309.02033](https://arxiv.org/abs/2309.02033) [find](https://scholar.google.com/scholar?q=Daoyuan+Chen%2C+Yilun+Huang%2C+Zhijian+Ma%2C+Hesen+Chen%2C+Xuchen+Pan%2C+Ce+Ge%2C+Dawei+Gao%2C+Yuexiang+Xie%2C+Zhaoyang+Liu%2C+Jinyang+Gao%2C+et+al%2E+Data%2Djuicer%3A+A+one%2Dstop+data+processing+system+for+large+language+model)
13. Huayu Chen, Guande He, Hang Su, and Jun Zhu. Noise contrastive alignment of language models with explicit rewards. arXiv preprint arXiv:2402.05369, 2024a. — [arXiv:2402.05369](https://arxiv.org/abs/2402.05369) [find](https://scholar.google.com/scholar?q=Huayu+Chen%2C+Guande+He%2C+Hang+Su%2C+and+Jun+Zhu%2E+Noise+contrastive+alignment+of+language+models+with+explicit+rewards%2E+arXiv+preprint+arXiv%3A2402%2E05369%2C+2024a%2E+%E2%80%94+%5BarXiv%3A2402%2E05369%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F24)
14. Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré. Skill-it! a data-driven skills framework for understanding and training language models. Advances in Neural Information Processing Systems, 36, 2024b. [find](https://scholar.google.com/scholar?q=Mayee+Chen%2C+Nicholas+Roberts%2C+Kush+Bhatia%2C+Jue+Wang%2C+Ce+Zhang%2C+Frederic+Sala%2C+and+Christopher+Re%CC%81%2E+Skill%2Dit%21+a+data%2Ddriven+skills+framework+for+understanding+and+training+language+models%2E+Advances+in+)
15. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2023b. ISSN 2835-8856. URL [https://openreview.net/](https://openreview.net/) forum?id=YfZ4ZPt8zd. [↗](https://openreview.net/)
16. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). Journal of Machine Learning Research, 24(240): 1–113, 2023.
17. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
18. Tri Dao. [Flashattention-2: Faster attention with better parallelism and work partitioning](https://md.research.elamur.ai/free/en/papers/dao-2023-flashattention-2-faster-attention-better/). arXiv preprint arXiv:2307.08691, 2023.
19. Databricks. [https://huggingface.co/datasets/databricks/](https://huggingface.co/datasets/databricks/) databricks-dolly-15k, 2023. [↗](https://huggingface.co/datasets/databricks/)
20. Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pp. 7480–7512. P… [find](https://scholar.google.com/scholar?q=Mostafa+Dehghani%2C+Josip+Djolonga%2C+Basil+Mustafa%2C+Piotr+Padlewski%2C+Jonathan+Heek%2C+Justin+Gilmer%2C+Andreas+Peter+Steiner%2C+Mathilde+Caron%2C+Robert+Geirhos%2C+Ibrahim+Alabdulmohsin%2C+et+al%2E+Scaling+vision+tran)
21. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805, 2018.
22. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. [Enhancing chat language models by scaling high-quality instructional conversations](https://md.research.elamur.ai/free/en/papers/ding-2023-enhancing-chat-language-models-scaling/). arXiv preprint arXiv:2305.14233, 2023.
23. Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, et al. What’s in my big data? arXiv preprint arXiv:2310.20707, 2023. [find](https://scholar.google.com/scholar?q=Yanai+Elazar%2C+Akshita+Bhagia%2C+Ian+Magnusson%2C+Abhilasha+Ravichander%2C+Dustin+Schwenk%2C+Alane+Suhr%2C+Pete+Walsh%2C+Dirk+Groeneveld%2C+Luca+Soldaini%2C+Sameer+Singh%2C+et+al%2E+What%E2%80%99s+in+my+big+data%3F+arXiv+preprint+a)
24. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+et+al%2E+The+pile%3A+An+800gb+dataset+of+diverse+text+for+language+)
25. Leo Gao, John Schulman, and Jacob Hilton. [Scaling laws for reward model overoptimization](https://md.research.elamur.ai/free/en/papers/gao-2022-scaling-laws-reward-model-overoptimization/). In International Conference on Machine Learning, pp. 10835–10866. PMLR, 2023.
26. Gemini-Team, Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint arXi…
27. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644, 2023. [find](https://scholar.google.com/scholar?q=Suriya+Gunasekar%2C+Yi+Zhang%2C+Jyoti+Aneja%2C+Caio+Ce%CC%81sar+Teodoro+Mendes%2C+Allie+Del+Giorno%2C+Sivakanth+Gopi%2C+Mojan+Javaheripi%2C+Piero+Kauffmann%2C+Gustavo+de+Rosa%2C+Olli+Saarikivi%2C+et+al%2E+Textbooks+are+all+you+)
28. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300, 2020.
29. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874, 2021.
30. Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, et al. [Scaling laws and interpretability of learning from repeated data](https://md.research.elamur.ai/free/en/papers/hernandez-2022-scaling-laws-interpretability-learning-repeated/). arXiv preprint arXiv:2205.10487, 2022.
31. Jiwoo Hong, Noah Lee, and James Thorne. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691, 2024. [find](https://scholar.google.com/scholar?q=Jiwoo+Hong%2C+Noah+Lee%2C+and+James+Thorne%2E+Orpo%3A+Monolithic+preference+optimization+without+reference+model%2E+arXiv+preprint+arXiv%3A2403%2E07691%2C+2024%2E)
32. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations, 2022. URL [https://openreview.net/forum?](https://openreview.net/forum?) id=nZeVKeeFYf9. [↗](https://openreview.net/forum?)
33. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. [C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models](https://md.research.elamur.ai/free/en/papers/huang-2023-c-eval-multi-level-multi/). Advances in Neural Information Processing Systems, 36, 2024.
34. Omid Jafari, Preeti Maurya, Parth Nagarkar, Khandker Mushfiqul Islam, and Chidambaram Cru-shev. A survey on locality sensitive hashing algorithms and their applications. arXiv preprint arXiv:2102.08942, 2021. — [arXiv:2102.08942](https://arxiv.org/abs/2102.08942) [find](https://scholar.google.com/scholar?q=Omid+Jafari%2C+Preeti+Maurya%2C+Parth+Nagarkar%2C+Khandker+Mushfiqul+Islam%2C+and+Chidambaram+Cru%2Dshev%2E+A+survey+on+locality+sensitive+hashing+algorithms+and+their+applications%2E+arXiv+preprint+arXiv%3A2102%2E0894)
35. He Junqing, Pan Kunhao, Dong Xiaoqun, Song Zhuoyang, Liu Yibo, Liang Yuxin, Wang Hao, Sun Qianguo, Zhang Songxin, Xie Zejian, et al. Never lost in the middle: Improving large language models via attention strengthening question answering. arXiv preprint arXiv:2311.09198, 2023. — [arXiv:2311.09198](https://arxiv.org/abs/2311.09198) [find](https://scholar.google.com/scholar?q=He+Junqing%2C+Pan+Kunhao%2C+Dong+Xiaoqun%2C+Song+Zhuoyang%2C+Liu+Yibo%2C+Liang+Yuxin%2C+Wang+Hao%2C+Sun+Qianguo%2C+Zhang+Songxin%2C+Xie+Zejian%2C+et+al%2E+Never+lost+in+the+middle%3A+Improving+large+language+models+via+atten)
36. Gregory Kamradt. [https://github.com/gkamradt/LLMTest_NeedleInAHaystack](https://github.com/gkamradt/LLMTest_NeedleInAHaystack), 2023. [↗](https://github.com/gkamradt/LLMTest_NeedleInAHaystack)
37. Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. [Reducing activation recomputation in large transformer models](https://md.research.elamur.ai/free/en/papers/korthikanti-2022-reducing-activation-recomputation-large-transformer/). Proceedings of Machine Learning and Systems, 5, 2023.
38. Ashutosh Kumar, Sagarika Singh, Shiv Vignesh Murty, and Swathy Ragupathy. The ethics of interaction: Mitigating security threats in llms. arXiv preprint arXiv:2401.12273, 2024. — [arXiv:2401.12273](https://arxiv.org/abs/2401.12273) [find](https://scholar.google.com/scholar?q=Ashutosh+Kumar%2C+Sagarika+Singh%2C+Shiv+Vignesh+Murty%2C+and+Swathy+Ragupathy%2E+The+ethics+of+interaction%3A+Mitigating+security+threats+in+llms%2E+arXiv+preprint+arXiv%3A2401%2E12273%2C+2024%2E+%E2%80%94+%5BarXiv%3A2401%2E12273%5D%28ht)
39. Alycia Lee, Brando Miranda, and Sanmi Koyejo. Beyond scale: the diversity coefficient as a data quality metric demonstrates llms are pre-trained on formally diverse data. arXiv preprint arXiv:2306.13840, 2023. — [arXiv:2306.13840](https://arxiv.org/abs/2306.13840) [find](https://scholar.google.com/scholar?q=Alycia+Lee%2C+Brando+Miranda%2C+and+Sanmi+Koyejo%2E+Beyond+scale%3A+the+diversity+coefficient+as+a+data+quality+metric+demonstrates+llms+are+pre%2Dtrained+on+formally+diverse+data%2E+arXiv+preprint+arXiv%3A2306%2E138)
40. Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. [Cmmlu: Measuring massive multitask language understanding in chinese](https://md.research.elamur.ai/free/en/papers/li-2023-cmmlu-measuring-massive-multitask-language/). arXiv preprint arXiv:2306.09212, 2023a.
41. Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. [Sequence parallelism: Long sequence training from system perspective](https://md.research.elamur.ai/free/en/papers/li-2023-sequence-parallelism-long-sequence-training/). arXiv preprint arXiv:2105.13120, 2021.
42. Ziniu Li, Tian Xu, Yushun Zhang, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effective, and efficient method for aligning large language models. arXiv preprint arXiv:2310.10505, 2023b. [find](https://scholar.google.com/scholar?q=Ziniu+Li%2C+Tian+Xu%2C+Yushun+Zhang%2C+Yang+Yu%2C+Ruoyu+Sun%2C+and+Zhi%2DQuan+Luo%2E+Remax%3A+A+simple%2C+effective%2C+and+efficient+method+for+aligning+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E10505%2C+2023b%2E)
43. Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and ”Teknium”. Openorca: An open dataset of gpt augmented flan reasoning traces. https://https://huggingface.co/Open-Orca/OpenOrca, 2023. [find](https://scholar.google.com/scholar?q=Wing+Lian%2C+Bleys+Goodson%2C+Eugene+Pentland%2C+Austin+Cook%2C+Chanvichet+Vong%2C+and+%E2%80%9DTeknium%E2%80%9D%2E+Openorca%3A+An+open+dataset+of+gpt+augmented+flan+reasoning+traces%2E+https%3A%2F%2Fhttps%3A%2F%2Fhuggingface%2Eco%2FOpen%2DOrca%2FOpenO)
44. Hao Liu, Matei Zaharia, and Pieter Abbeel. [Ring attention with blockwise transformers for near-infinite context](https://md.research.elamur.ai/free/en/papers/liu-2023-ring-attention-blockwise-transformers-near/). arXiv preprint arXiv:2310.01889, 2023.
45. Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint arXiv:2402.08268, 2024a. [find](https://scholar.google.com/scholar?q=Hao+Liu%2C+Wilson+Yan%2C+Matei+Zaharia%2C+and+Pieter+Abbeel%2E+World+model+on+million%2Dlength+video+and+language+with+ringattention%2E+arXiv+preprint+arXiv%3A2402%2E08268%2C+2024a%2E)
46. Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. [What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-makes-good-data-alignment-comprehensive/). In The Twelfth International Conference on Learning Representations, 2024b. URL [https://openreview.net/forum?id=BTKAeLqLMw](https://openreview.net/forum?id=BTKAeLqLMw). [↗](https://openreview.net/forum?id=BTKAeLqLMw)
47. Ilya Loshchilov and Frank Hutter. arXiv:1711.05101, 2017. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+arXiv%3A1711%2E05101%2C+2017%2E)
48. Decoupled weight decay regularization. arXiv preprint Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining llms at scale. arXiv preprint arXiv:2309.04564, 2023. [find](https://scholar.google.com/scholar?q=Decoupled+weight+decay+regularization%2E+arXiv+preprint+Max+Marion%2C+Ahmet+U%CC%88stu%CC%88n%2C+Luiza+Pozzobon%2C+Alex+Wang%2C+Marzieh+Fadaee%2C+and+Sara+Hooker%2E+When+less+is+more%3A+Investigating+data+pruning+for+pretraini)
49. Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. [Orca-math: Unlocking the potential of slms in grade school math](https://md.research.elamur.ai/free/en/papers/mitra-2024-orca-math-unlocking-potential-slms/), 2024.
50. Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. [Orca: Progressive learning from complex explanation traces of gpt-4](https://md.research.elamur.ai/free/en/papers/mukherjee-2023-orca-progressive-learning-complex-explanation/), 2023.
51. OpenAI. tiktoken: A fast bpe tokeniser for use with openai’s models. [https://github.com/](https://github.com/) openai/tiktoken, 2022. [↗](https://github.com/)
52. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
53. OpenCompass-Contributors. Opencompass: A universal evaluation platform for foundation models. GitHub repository, 2023. [find](https://scholar.google.com/scholar?q=OpenCompass%2DContributors%2E+Opencompass%3A+A+universal+evaluation+platform+for+foundation+models%2E+GitHub+repository%2C+2023%2E)
54. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in neural information processing systems, 35: 27730–27744, 2022.
55. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116… [find](https://scholar.google.com/scholar?q=Guilherme+Penedo%2C+Quentin+Malartic%2C+Daniel+Hesslow%2C+Ruxandra+Cojocaru%2C+Alessandro+Cappelli%2C+Hamza+Alobeidli%2C+Baptiste+Pannier%2C+Ebtesam+Almazrouei%2C+and+Julien+Launay%2E+The+refinedweb+dataset+for+falcon+)
56. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, and otahers. Improving language understanding by generative pre-training, 2018. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+Ilya+Sutskever%2C+and+otahers%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2C+2018%2E)
57. Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed H Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng. How to train data-efficient llms. arXiv preprint arXiv:2402.09668, 2024. [find](https://scholar.google.com/scholar?q=Noveen+Sachdeva%2C+Benjamin+Coleman%2C+Wang%2DCheng+Kang%2C+Jianmo+Ni%2C+Lichan+Hong%2C+Ed+H+Chi%2C+James+Caverlee%2C+Julian+McAuley%2C+and+Derek+Zhiyuan+Cheng%2E+How+to+train+data%2Defficient+llms%2E+arXiv+preprint+arXiv%3A24)
58. Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsahar, Niklas Muennighoff, Jason Phang, et al. What language model to train if you have one million gpu hours? arXiv preprint arXiv:2210.15424, 2022. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Thomas+Wang%2C+Daniel+Hesslow%2C+Lucile+Saulnier%2C+Stas+Bekman%2C+M+Saiful+Bari%2C+Stella+Biderman%2C+Hady+Elsahar%2C+Niklas+Muennighoff%2C+Jason+Phang%2C+et+al%2E+What+language+model+to+train+if+you+have)
59. Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2E+Glu+variants+improve+transformer%2E+arXiv+preprint+arXiv%3A2002%2E05202%2C+2020%2E)
60. Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Joel Hestness, Natalia Vassilieva, Daria Soboleva, and Eric Xing. [Slimpajama-dc: Understanding data combinations for llm training](https://md.research.elamur.ai/free/en/papers/shen-2023-slimpajama-dc-understanding-data-combinations/). arXiv preprint arXiv:2309.10818, 2023.
61. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2E+arXiv+preprin)
62. Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402.00159, 2024. [find](https://scholar.google.com/scholar?q=Luca+Soldaini%2C+Rodney+Kinney%2C+Akshita+Bhagia%2C+Dustin+Schwenk%2C+David+Atkinson%2C+Russell+Authur%2C+Ben+Bogin%2C+Khyathi+Chandu%2C+Jennifer+Dumas%2C+Yanai+Elazar%2C+et+al%2E+Dolma%3A+An+open+corpus+of+three+trillion+to)
63. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Neurocomputing, 568:127063, 2024.
64. Nishant Subramani, Sasha Luccioni, Jesse Dodge, and Margaret Mitchell. Detecting personal information in training corpora: an analysis. In Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023), pp. 208–220, 2023. [find](https://scholar.google.com/scholar?q=Nishant+Subramani%2C+Sasha+Luccioni%2C+Jesse+Dodge%2C+and+Margaret+Mitchell%2E+Detecting+personal+information+in+training+corpora%3A+an+analysis%2E+In+Proceedings+of+the+3rd+Workshop+on+Trustworthy+Natural+Langua)
65. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. [Challenging big-bench tasks and whether chain-of-thought can solve them](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/). arXiv preprint arXiv:2210.09261, 2022.
66. Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. [D4: Improving llm pretraining via document de-duplication and diversification](https://md.research.elamur.ai/free/en/papers/tirumala-2023-d4-improving-llm-pretraining-document/). Advances in Neural Information Processing Systems, 36, 2024.
67. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023a.
68. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023b.
69. Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint arXiv:2401.06080, 2024. [find](https://scholar.google.com/scholar?q=Binghai+Wang%2C+Rui+Zheng%2C+Lu+Chen%2C+Yan+Liu%2C+Shihan+Dou%2C+Caishuang+Huang%2C+Wei+Shen%2C+Senjie+Jin%2C+Enyu+Zhou%2C+Chenyu+Shi%2C+et+al%2E+Secrets+of+rlhf+in+large+language+models+part+ii%3A+Reward+modeling%2E+arXiv+pre)
70. Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lü, Rui Hu, et al. [Skywork: A more open bilingual foundation model](https://md.research.elamur.ai/free/en/papers/wei-2023-skywork-more-open-bilingual/). arXiv preprint arXiv:2310.19341, 2023.
71. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. [Ccnet: Extracting high quality monolingual datasets from web crawl data](https://md.research.elamur.ai/free/en/papers/wenzek-2020-ccnet-extracting-high-quality-monolingual/). arXiv preprint arXiv:1911.00359, 2019.
72. Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, et al. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint arXiv:2309.14322, 2023. [find](https://scholar.google.com/scholar?q=Mitchell+Wortsman%2C+Peter+J+Liu%2C+Lechao+Xiao%2C+Katie+Everett%2C+Alex+Alemi%2C+Ben+Adlam%2C+John+D+Co%2DReyes%2C+Izzeddin+Gur%2C+Abhishek+Kumar%2C+Roman+Novak%2C+et+al%2E+Small%2Dscale+proxies+for+large%2Dscale+transformer+tr)
73. Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. [Recursively summarizing books with human feedback](https://md.research.elamur.ai/free/en/papers/wu-2021-recursively-summarizing-books-human-feedback/). arXiv preprint arXiv:2109.10862, 2021.
74. Wei Wu, Bin Li, Ling Chen, Junbin Gao, and Chengqi Zhang. A review for weighted minhash algorithms. IEEE Transactions on Knowledge and Data Engineering, 34(6):2553–2573, 2020. [find](https://scholar.google.com/scholar?q=Wei+Wu%2C+Bin+Li%2C+Ling+Chen%2C+Junbin+Gao%2C+and+Chengqi+Zhang%2E+A+review+for+weighted+minhash+algorithms%2E+IEEE+Transactions+on+Knowledge+and+Data+Engineering%2C+34%286%29%3A2553%E2%80%932573%2C+2020%2E)
75. Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Sang+Michael+Xie%2C+Hieu+Pham%2C+Xuanyi+Dong%2C+Nan+Du%2C+Hanxiao+Liu%2C+Yifeng+Lu%2C+Percy+S+Liang%2C+Quoc+V+Le%2C+Tengyu+Ma%2C+and+Adams+Wei+Yu%2E+Doremi%3A+Optimizing+data+mixtures+speeds+up+language+model+pretraining%2E+)
76. Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. [Effective long-context scaling of foundation models](https://md.research.elamur.ai/free/en/papers/xiong-2023-effective-long-context-scaling-foundation/). arXiv preprint arXiv:2309.16039, 2023.
77. Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for llm alignment? a comprehensive study. arXiv preprint arXiv:2404.10719, 2024. — [arXiv:2404.10719](https://arxiv.org/abs/2404.10719) [find](https://scholar.google.com/scholar?q=Shusheng+Xu%2C+Wei+Fu%2C+Jiaxuan+Gao%2C+Wenjie+Ye%2C+Weilin+Liu%2C+Zhiyu+Mei%2C+Guangju+Wang%2C+Chao+Yu%2C+and+Yi+Wu%2E+Is+dpo+superior+to+ppo+for+llm+alignment%3F+a+comprehensive+study%2E+arXiv+preprint+arXiv%3A2404%2E10719%2C+)
78. Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. [Baichuan 2: Open large-scale language models](https://md.research.elamur.ai/free/en/papers/yang-2023-baichuan-2-open-large-scale/). arXiv preprint arXiv:2309.10305, 2023.
79. Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Shaochen Zhong, Bing Yin, and Xia Hu. [Harnessing the power of llms in practice: A survey on chatgpt and beyond](https://md.research.elamur.ai/free/en/papers/yang-2023-harnessing-power-llms-practice-survey/). ACM Transactions on Knowledge Discovery from Data, 18(6):1–32, 2024.
80. Jiasheng Ye, Peiju Liu, Tianxiang Sun, Yunhua Zhou, Jun Zhan, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint arXiv:2403.16952, 2024. [find](https://scholar.google.com/scholar?q=Jiasheng+Ye%2C+Peiju+Liu%2C+Tianxiang+Sun%2C+Yunhua+Zhou%2C+Jun+Zhan%2C+and+Xipeng+Qiu%2E+Data+mixing+laws%3A+Optimizing+data+mixtures+by+predicting+language+modeling+performance%2E+arXiv+preprint+arXiv%3A2403%2E16952%2C+2)
81. Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.[04652, 2024](https://md.research.elamur.ai/free/en/papers/ai-2024-yi-open-foundation-models-01/).
82. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. [Glm-130b: An open bilingual pre-trained model](https://md.research.elamur.ai/free/en/papers/zeng-2022-glm-130b-open-bilingual-pre/). arXiv preprint arXiv:2210.02414, 2022.
83. Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019. [find](https://scholar.google.com/scholar?q=Biao+Zhang+and+Rico+Sennrich%2E+Root+mean+square+layer+normalization%2E+Advances+in+Neural+Information+Processing+Systems%2C+32%2C+2019%2E)
84. Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. [Instruction tuning for large language models: A survey](https://md.research.elamur.ai/free/en/papers/zhang-2023-instruction-tuning-large-language-models/). arXiv preprint arXiv:2308.10792, 2023.
85. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. [A survey of large language models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/). arXiv preprint arXiv:2303.18223, 2023.
86. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations, 2023. [find](https://scholar.google.com/scholar?q=Chujie+Zheng%2C+Hao+Zhou%2C+Fandong+Meng%2C+Jie+Zhou%2C+and+Minlie+Huang%2E+Large+language+models+are+not+robust+multiple+choice+selectors%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2C+2)
87. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Advances in Neural Information Processing Systems, 36, 2024.
88. Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. [Agieval: A human-centric benchmark for evaluating foundation models](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/). arXiv preprint arXiv:2304.06364, 2023.
89. Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. Don’t make your llm an evaluation benchmark cheater. arXiv preprint arXiv:2311.01964, 2023. [find](https://scholar.google.com/scholar?q=Kun+Zhou%2C+Yutao+Zhu%2C+Zhipeng+Chen%2C+Wentong+Chen%2C+Wayne+Xin+Zhao%2C+Xu+Chen%2C+Yankai+Lin%2C+Ji%2DRong+Wen%2C+and+Jiawei+Han%2E+Don%E2%80%99t+make+your+llm+an+evaluation+benchmark+cheater%2E+arXiv+preprint+arXiv%3A2311%2E01964%2C)

*Updated 31 October 2026*