---
title: "SOLAR 10.7B — paper by Kim et al. (2023) — summary and ideas"
description: "The authors propose depth up-scaling (DUS), a simpler alternative that reuses a smaller pretrained… 64 references, cited by 68 papers here."
authors: ["Dahyun Kim", "Chanjun Park", "Sanghoon Kim", "Wonsung Lee", "Wonho Song", "Yunsu Kim", "Hyeonwoo Kim", "Yungi Kim", "Hyeonju Lee", "Jihoo Kim", "Changbae Ahn", "Seonghoon Yang", "Sukyung Lee", "Hyunbyung Park", "Gyoungjin Gim", "Mikyoung Cha", "Hwalsuk Lee", "Sunghun Kim"]
year: 2023
venue: "arXiv:2312.15166"
source: "https://arxiv.org/abs/2312.15166"
doi: "10.48550/arxiv.2312.15166"
arxiv: "2312.15166"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kim-2023-solar-10-7b-scaling-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kim-2023-solar-10-7b-scaling-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling

# SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling

**Authors:** Dahyun Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, [Yunsu Kim](https://md.research.elamur.ai/free/en/authors/yunsu-kim/), Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, Changbae Ahn, Seonghoon Yang, Sukyung Lee, Hyunbyung Park, Gyoungjin Gim, Mikyoung Cha, Hwalsuk Lee and Sunghun Kim · **Year:** 2023 · **Venue:** arXiv:2312.15166 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2312.15166) · [PDF](https://arxiv.org/pdf/2312.15166) · [DOI 10.48550/arxiv.2312.15166](https://doi.org/10.48550/arxiv.2312.15166) · [arXiv 2312.15166](https://arxiv.org/abs/2312.15166) · [OpenAlex](https://openalex.org/W4390306040)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [68](https://md.research.elamur.ai/free/en/papers/kim-2023-solar-10-7b-scaling-large/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Scaling up LLMs usually means training ever-larger networks or adopting mixture-of-experts designs, which complicate training and inference pipelines. The authors propose depth up-scaling (DUS), a simpler alternative that reuses a smaller pretrained model. They take a 32-layer Llama 2-style network initialized with Mistral 7B weights and make two copies. They remove the last 8 layers from one copy and the first 8 from the other, then stack the copies into a 48-layer model with 10.7B parameters. This splice initially hurts quality, so continued pretraining follows, and performance recovers quickly. The authors argue that dropping the middle layers reduces the mismatch where the two copies join.

The resulting SOLAR 10.7B is instruction-tuned and then preference-aligned with sDPO, mostly on open data plus synthetic math sets. Two differently tuned candidates are merged into SOLAR 10.7B-Instruct. On Open LLM Leaderboard tasks, this model beats much larger systems, including Mixtral-8x7B-Instruct. DUS adds no new modules, so it works with standard tooling such as HuggingFace. The weights are released under Apache 2.0.

## Key points
- **DUS configuration:** base n = 32 layers, m = 8 layers removed per copy, giving s = 2·(n−m) = 48 layers. The authors chose this depth to fit hardware limits and land between 7B and 13B parameters. They note that m = 8 has not been tuned for optimality.
- **Leaderboard scores (H6, the average of ARC, HellaSwag, MMLU, TruthfulQA, Winogrande, GSM8K):**
  - SOLAR 10.7B-Instruct: 74.20, ahead of Qwen 72B (73.60) and Mixtral 8x7B-Instruct-v0.1 (72.62).
  - Base SOLAR 10.7B: 66.04, compared with 60.97 for Mistral 7B.
- **Instruction tuning data:** Alpaca-GPT4, OpenOrca and Synth. Math-Instruct. The last set was built by rephrasing seed problems from the Math dataset in a MetaMath-like way, which avoids contamination with GSM8K.
- **Alignment data:** Orca DPO Pairs, Ultrafeedback Cleaned and Synth. Math-Alignment. In Synth. Math-Alignment, the rephrased answer serves as "chosen" and the original answer as "rejected" in each DPO tuple.
- **Model merging:** weight averaging and SLERP gave similar H6 results, suggesting the choice of merge method matters less than merging candidates with complementary strengths. "Merge v1" became the released Instruct model.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [UpStage](https://md.research.elamur.ai/free/en/companies/upstage/) — Company, Yongin-si, South Korea · 2 works · 655 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works

## Cite

Cite as: Dahyun Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, Changbae Ahn, Seonghoon Yang, Sukyung Lee, Hyunbyung Park, Gyoungjin Gim, Mikyoung Cha, Hwalsuk Lee and Sunghun Kim. 2023. SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling. arXiv:2312.15166.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{kim2023solar,
  title = {{SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling}},
  author = {Dahyun Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, Changbae Ahn, Seonghoon Yang, Sukyung Lee, Hyunbyung Park, Gyoungjin Gim, Mikyoung Cha, Hwalsuk Lee and Sunghun Kim},
  year = {2023},
  doi = {10.48550/arxiv.2312.15166},
  eprint = {2312.15166},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.15166}
}
```

- [← previous Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents](https://md.research.elamur.ai/free/en/papers/gunther-2023-jina-embeddings-2-8192-token/)
- [next → Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration](https://md.research.elamur.ai/free/en/papers/lyu-2023-macaw-llm-multi-modal-language/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

64 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ian L Alberts, Lorenzo Mercolli, Thomas Pyka, George Prenosil, Kuangyu Shi, Axel Rominger, and Ali Afshar-Oromieh. 2023. Large language models (llm) and chatgpt: what will the impact on nuclear medicine be? European journal of nuclear medicine and molecular imaging, 50(6):1549–1552. [find](https://scholar.google.com/scholar?q=Ian+L+Alberts%2C+Lorenzo+Mercolli%2C+Thomas+Pyka%2C+George+Prenosil%2C+Kuangyu+Shi%2C+Axel+Rominger%2C+and+Ali+Afshar%2DOromieh%2E+2023%2E+Large+language+models+%28llm%29+and+chatgpt%3A+what+will+the+impact+on+nuclear+medici)
2. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. [Palm 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/). arXiv preprint arXiv:2305.10403.
3. Aram Bahrini, Mohammadsadra Khamoshifar, Hossein Abbasimehr, Robert J Riggs, Maryam Esmaeili, Rastin Mastali Majdabadkohne, and Morteza Pase-hvar. 2023. Chatgpt: Applications, opportunities, and threats. In 2023 Systems and Information Engineering Design Symposium (SIEDS), pages 274–279. IEEE. [find](https://scholar.google.com/scholar?q=Aram+Bahrini%2C+Mohammadsadra+Khamoshifar%2C+Hossein+Abbasimehr%2C+Robert+J+Riggs%2C+Maryam+Esmaeili%2C+Rastin+Mastali+Majdabadkohne%2C+and+Morteza+Pase%2Dhvar%2E+2023%2E+Chatgpt%3A+Applications%2C+opportunities%2C+and+threa)
4. Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. 2023. Open llm leaderboard. [https://huggingface.co/spaces/](https://huggingface.co/spaces/) HuggingFaceH4/open_llm_leaderboard. [↗](https://huggingface.co/spaces/)
5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
6. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. [find](https://scholar.google.com/scholar?q=Peter+Clark%2C+Isaac+Cowhey%2C+Oren+Etzioni%2C+Tushar+Khot%2C+Ashish+Sabharwal%2C+Carissa+Schoenick%2C+and+Oyvind+Tafjord%2E+2018%2E+Think+you+have+solved+question+answering%3F+try+arc%2C+the+ai2+reasoning+challenge%2E+arX)
7. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
8. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377. [find](https://scholar.google.com/scholar?q=Ganqu+Cui%2C+Lifan+Yuan%2C+Ning+Ding%2C+Guanming+Yao%2C+Wei+Zhu%2C+Yuan+Ni%2C+Guotong+Xie%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2023%2E+Ultrafeedback%3A+Boosting+language+models+with+high%2Dquality+feedback%2E+arXiv+preprint+ar)
9. Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. 2023. [Investigating data contamination in modern benchmarks for large language models](https://md.research.elamur.ai/free/en/papers/deng-2024-investigating-data-contamination-modern-benchmarks/). arXiv preprint arXiv:2311.09783.
10. Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. 2023. [Raft: Reward ranked finetuning for generative foundation model alignment](https://md.research.elamur.ai/free/en/papers/dong-2023-raft-reward-ranked-finetuning-generative/). arXiv preprint arXiv:2304.06767.
11. Mohammad Fraiwan and Natheer Khasawneh. 2023. A review of chatgpt applications in education, marketing, software engineering, and healthcare: Benefits, drawbacks, and research directions. arXiv preprint arXiv:2305.00237. — [arXiv:2305.00237](https://arxiv.org/abs/2305.00237) [find](https://scholar.google.com/scholar?q=Mohammad+Fraiwan+and+Natheer+Khasawneh%2E+2023%2E+A+review+of+chatgpt+applications+in+education%2C+marketing%2C+software+engineering%2C+and+healthcare%3A+Benefits%2C+drawbacks%2C+and+research+directions%2E+arXiv+prepri)
12. Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. 2023. Megablocks: Efficient sparse training with mixture-of-experts. Proceedings of Machine Learning and Systems, 5. [find](https://scholar.google.com/scholar?q=Trevor+Gale%2C+Deepak+Narayanan%2C+Cliff+Young%2C+and+Matei+Zaharia%2E+2023%2E+Megablocks%3A+Efficient+sparse+training+with+mixture%2Dof%2Dexperts%2E+Proceedings+of+Machine+Learning+and+Systems%2C+5%2E)
13. Andrea Gesmundo and Kaitlin Maile. 2023. Composable function-preserving expansions for transformer architectures. arXiv preprint arXiv:2308.06103. — [arXiv:2308.06103](https://arxiv.org/abs/2308.06103) [find](https://scholar.google.com/scholar?q=Andrea+Gesmundo+and+Kaitlin+Maile%2E+2023%2E+Composable+function%2Dpreserving+expansions+for+transformer+architectures%2E+arXiv+preprint+arXiv%3A2308%2E06103%2E+%E2%80%94+%5BarXiv%3A2308%2E06103%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2308%2E06103)
14. Shahriar Golchin and Mihai Surdeanu. 2023. [Time travel in llms: Tracing data contamination in large language models](https://md.research.elamur.ai/free/en/papers/golchin-2023-time-travel-llms-tracing-data/). arXiv preprint arXiv:2308.08493.
15. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations.
16. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874.
17. Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. 2021. [Scaling laws for transfer](https://md.research.elamur.ai/free/en/papers/hernandez-2021-scaling-laws-transfer/). arXiv preprint arXiv:2102.01293.
18. Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, et al. 2023. Tutel: Adaptive mixture-of-experts at scale. Proceedings of Machine Learning and Systems, 5. [find](https://scholar.google.com/scholar?q=Changho+Hwang%2C+Wei+Cui%2C+Yifan+Xiong%2C+Ziyue+Yang%2C+Ze+Liu%2C+Han+Hu%2C+Zilong+Wang%2C+Rafael+Salas%2C+Jithin+Jose%2C+Prabhat+Ram%2C+et+al%2E+2023%2E+Tutel%3A+Adaptive+mixture%2Dof%2Dexperts+at+scale%2E+Proceedings+of+Machine+L)
19. Intel. 2023. Supervised fine-tuning and direct preference optimization on intel gaudi2. [find](https://scholar.google.com/scholar?q=Intel%2E+2023%2E+Supervised+fine%2Dtuning+and+direct+preference+optimization+on+intel+gaudi2%2E)
20. Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023. [Camels in a changing climate: Enhancing lm adaptation with tulu 2](https://md.research.elamur.ai/free/en/papers/ivison-2023-camels-changing-climate-enhancing-lm/).
21. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+2023%2E+Mist)
22. Jean Kaddour, Oscar Key, Piotr Nawrot, Pasquale Minervini, and Matt J Kusner. 2023. No train no gain: Revisiting efficient training algorithms for transformer-based language models. arXiv preprint arXiv:2307.06440. [find](https://scholar.google.com/scholar?q=Jean+Kaddour%2C+Oscar+Key%2C+Piotr+Nawrot%2C+Pasquale+Minervini%2C+and+Matt+J+Kusner%2E+2023%2E+No+train+no+gain%3A+Revisiting+efficient+training+algorithms+for+transformer%2Dbased+language+models%2E+arXiv+preprint+arX)
23. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. [find](https://scholar.google.com/scholar?q=Jared+Kaplan%2C+Sam+McCandlish%2C+Tom+Henighan%2C+Tom+B+Brown%2C+Benjamin+Chess%2C+Rewon+Child%2C+Scott+Gray%2C+Alec+Radford%2C+Jeffrey+Wu%2C+and+Dario+Amodei%2E+2020%2E)
24. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361.
25. Dahyun Kim, Yungi Kim, Wonho Song, Hyeonwoo Kim, Yunsu Kim, Sanghoon Kim, and Chanjun Park. 2024a. sdpo: Don’t use your data all at once. [find](https://scholar.google.com/scholar?q=Dahyun+Kim%2C+Yungi+Kim%2C+Wonho+Song%2C+Hyeonwoo+Kim%2C+Yunsu+Kim%2C+Sanghoon+Kim%2C+and+Chanjun+Park%2E+2024a%2E+sdpo%3A+Don%E2%80%99t+use+your+data+all+at+once%2E)
26. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). arXiv preprint arXiv:2305.18290.
27. Jihoo Kim, Wonho Song, Dahyun Kim, Yunsu Kim, Yungi Kim, and Chanjun Park. 2024b. Evalverse: Unified and accessible library for large language model evaluation. [find](https://scholar.google.com/scholar?q=Jihoo+Kim%2C+Wonho+Song%2C+Dahyun+Kim%2C+Yunsu+Kim%2C+Yungi+Kim%2C+and+Chanjun+Park%2E+2024b%2E+Evalverse%3A+Unified+and+accessible+library+for+large+language+model+evaluation%2E)
28. Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. [Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark](https://md.research.elamur.ai/free/en/papers/sainz-2023-nlp-evaluation-trouble-need-measure/). arXiv preprint arXiv:2310.18018.
29. Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. 2022. Sparse upcycling: Training mixture-of-experts from dense checkpoints. arXiv preprint arXiv:2212.05055. [find](https://scholar.google.com/scholar?q=Aran+Komatsuzaki%2C+Joan+Puigcerver%2C+James+Lee%2DThorp%2C+Carlos+Riquelme+Ruiz%2C+Basil+Mustafa%2C+Joshua+Ainslie%2C+Yi+Tay%2C+Mostafa+Dehghani%2C+and+Neil+Houlsby%2E+2022%2E+Sparse+upcycling%3A+Training+mixture%2Dof%2Dexperts)
30. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2021%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Communications+of+the+ACM%2C+64%289%29%3A99%E2%80%93106%2E)
31. Wing Lian. 2023. [https://huggingface.co/](https://huggingface.co/) winglian/omega-3b. [↗](https://huggingface.co/)
32. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. [Truthfulqa: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252.
33. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023. [The flan collection: Designing data and methods for effective instruction tuning](https://md.research.elamur.ai/free/en/papers/longpre-2023-flan-collection-designing-data-methods/). arXiv preprint arXiv:2301.13688.
34. Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. [Orca: Progressive learning from complex explanation traces of gpt-4](https://md.research.elamur.ai/free/en/papers/mukherjee-2023-orca-progressive-learning-complex-explanation/). arXiv preprint arXiv:2306.02707.
35. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
36. Yu Pan, Ye Yuan, Yichun Yin, Zenglin Xu, Lifeng Shang, Xin Jiang, and Qun Liu. 2023. Reusing pre-trained models by multi-linear operators for efficient training. arXiv preprint arXiv:2310.10699. — [arXiv:2310.10699](https://arxiv.org/abs/2310.10699) [find](https://scholar.google.com/scholar?q=Yu+Pan%2C+Ye+Yuan%2C+Yichun+Yin%2C+Zenglin+Xu%2C+Lifeng+Shang%2C+Xin+Jiang%2C+and+Qun+Liu%2E+2023%2E+Reusing+pre%2Dtrained+models+by+multi%2Dlinear+operators+for+efficient+training%2E+arXiv+preprint+arXiv%3A2310%2E10699%2E+%E2%80%94+%5Bar)
37. Hyunbyung Park, Sukyung Lee, Gyoungjin Gim, Yungi Kim, Dahyun Kim, and Chanjun Park. 2024. Dataverse: Open-source etl (extract, transform, load) pipeline for large language models. [find](https://scholar.google.com/scholar?q=Hyunbyung+Park%2C+Sukyung+Lee%2C+Gyoungjin+Gim%2C+Yungi+Kim%2C+Dahyun+Kim%2C+and+Chanjun+Park%2E+2024%2E+Dataverse%3A+Open%2Dsource+etl+%28extract%2C+transform%2C+load%29+pipeline+for+large+language+models%2E)
38. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. [Instruction tuning with gpt-4](https://md.research.elamur.ai/free/en/papers/peng-2023-instruction-tuning-gpt-4/). arXiv preprint arXiv:2304.03277.
39. Malik Sallam, Nesreen Salim, Muna Barakat, and Alaa Al-Tammemi. 2023. Chatgpt applications in medical, dental, pharmacy, and public health education: A descriptive study highlighting the advantages and limitations. Narra J, 3(1):e103–e103. [find](https://scholar.google.com/scholar?q=Malik+Sallam%2C+Nesreen+Salim%2C+Muna+Barakat%2C+and+Alaa+Al%2DTammemi%2E+2023%2E+Chatgpt+applications+in+medical%2C+dental%2C+pharmacy%2C+and+public+health+education%3A+A+descriptive+study+highlighting+the+advantages+an)
40. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2C+Azalia+Mirhoseini%2C+Krzysztof+Maziarz%2C+Andy+Davis%2C+Quoc+Le%2C+Geoffrey+Hinton%2C+and+Jeff+Dean%2E+2017%2E+Outrageously+large+neural+networks%3A+The+sparsely%2Dgated+mixture%2Dof%2Dexperts+layer%2E+arXiv+pr)
41. Tianxiao Shen, Myle Ott, Michael Auli, and Marc’Aurelio Ranzato. 2019. Mixture models for diverse machine translation: Tricks of the trade. In International conference on machine learning, pages 5719–5728. PMLR. [find](https://scholar.google.com/scholar?q=Tianxiao+Shen%2C+Myle+Ott%2C+Michael+Auli%2C+and+Marc%E2%80%99Aurelio+Ranzato%2E+2019%2E+Mixture+models+for+diverse+machine+translation%3A+Tricks+of+the+trade%2E+In+International+conference+on+machine+learning%2C+pages+5719%E2%80%93)
42. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2023. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789. [find](https://scholar.google.com/scholar?q=Weijia+Shi%2C+Anirudh+Ajith%2C+Mengzhou+Xia%2C+Yangsibo+Huang%2C+Daogao+Liu%2C+Terra+Blevins%2C+Danqi+Chen%2C+and+Luke+Zettlemoyer%2E+2023%2E+Detecting+pretraining+data+from+large+language+models%2E+arXiv+preprint+arXiv%3A)
43. Ken Shoemake. 1985. Animating rotation with quaternion curves. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques, pages 245–254. [find](https://scholar.google.com/scholar?q=Ken+Shoemake%2E+1985%2E+Animating+rotation+with+quaternion+curves%2E+In+Proceedings+of+the+12th+annual+conference+on+Computer+graphics+and+interactive+techniques%2C+pages+245%E2%80%93254%2E)
44. Mingxing Tan and Quoc Le. 2019. [Efficientnet: Rethinking model scaling for convolutional neural networks](https://md.research.elamur.ai/free/en/papers/tan-2019-efficientnet-rethinking-model-scaling-convolutional/). In International conference on machine learning, pages 6105–6114. PMLR.
45. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
46. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023%2E+Llama+2%3A+Open+foundation+and+fi)
47. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446. [find](https://scholar.google.com/scholar?q=Jack+W+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+et+al%2E+2021%2E+Scaling+language+models%3A+Methods%2C+a)
48. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. 2023. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944. [find](https://scholar.google.com/scholar?q=Lewis+Tunstall%2C+Edward+Beeching%2C+Nathan+Lambert%2C+Nazneen+Rajani%2C+Kashif+Rasul%2C+Younes+Belkada%2C+Shengyi+Huang%2C+Leandro+von+Werra%2C+Cl%C3%A9mentine+Fourrier%2C+Nathan+Habib%2C+et+al%2E+2023%2E+Zephyr%3A+Direct+distilla)
49. Peihao Wang, Rameswar Panda, Lucas Torroba Hennigen, Philip Greengard, Leonid Karlinsky, Rogerio Feris, David Daniel Cox, Zhangyang Wang, and Yoon Kim. 2023. Learning to grow pretrained models for efficient transformer training. arXiv preprint arXiv:2303.00980. — [arXiv:2303.00980](https://arxiv.org/abs/2303.00980) [find](https://scholar.google.com/scholar?q=Peihao+Wang%2C+Rameswar+Panda%2C+Lucas+Torroba+Hennigen%2C+Philip+Greengard%2C+Leonid+Karlinsky%2C+Rogerio+Feris%2C+David+Daniel+Cox%2C+Zhangyang+Wang%2C+and+Yoon+Kim%2E+2023%2E+Learning+to+grow+pretrained+models+for+eff)
50. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2022%2E+Self%2Dinstruct%3A+Aligning+language+model+with+self+generated+instructions%2E+arXiv+pre)
51. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Maarten+Bosma%2C+Vincent+Y+Zhao%2C+Kelvin+Guu%2C+Adams+Wei+Yu%2C+Brian+Lester%2C+Nan+Du%2C+Andrew+M+Dai%2C+and+Quoc+V+Le%2E+2021%2E+Finetuned+language+models+are+zero%2Dshot+learners%2E+arXiv+preprint+arXiv%3A2109)
52. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022a. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). arXiv preprint arXiv:2206.07682.
53. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022b. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems, 35:24824–24837.
54. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. [Huggingface’s transformers: State-ofthe-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). arXiv preprint arXiv:1910.03771.
55. Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. 2023. [Ties-merging: Resolving interference when merging models](https://md.research.elamur.ai/free/en/papers/yadav-2023-ties-merging-resolving-interference-merging/). In Thirty-seventh Conference on Neural Information Processing Systems.
56. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2023. [Large language models as optimizers](https://md.research.elamur.ai/free/en/papers/yang-2023-large-language-models-optimizers/). arXiv preprint arXiv:2309.03409.
57. Yiqun Yao, Zheng Zhang, Jing Li, and Yequan Wang. 2023. 2x faster language model pre-training via masked structural growth. arXiv preprint arXiv:2305.02869. — [arXiv:2305.02869](https://arxiv.org/abs/2305.02869) [find](https://scholar.google.com/scholar?q=Yiqun+Yao%2C+Zheng+Zhang%2C+Jing+Li%2C+and+Yequan+Wang%2E+2023%2E+2x+faster+language+model+pre%2Dtraining+via+masked+structural+growth%2E+arXiv+preprint+arXiv%3A2305%2E02869%2E+%E2%80%94+%5BarXiv%3A2305%2E02869%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)
58. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). arXiv preprint arXiv:2309.12284.
59. Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2023. [Rrhf: Rank responses to align language models with human feedback without tears](https://md.research.elamur.ai/free/en/papers/yuan-2023-rrhf-rank-responses-align-language/). arXiv preprint arXiv:2304.05302.
60. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Com)
61. Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. 2023. [Instruction tuning for large language models: A survey](https://md.research.elamur.ai/free/en/papers/zhang-2023-instruction-tuning-large-language-models/). arXiv preprint arXiv:2308.10792.
62. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. [A survey of large language models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/). arXiv preprint arXiv:2303.18223.
63. Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. 2023. Don’t make your llm an evaluation benchmark cheater. arXiv preprint arXiv:2311.01964. [find](https://scholar.google.com/scholar?q=Kun+Zhou%2C+Yutao+Zhu%2C+Zhipeng+Chen%2C+Wentong+Chen%2C+Wayne+Xin+Zhao%2C+Xu+Chen%2C+Yankai+Lin%2C+Ji%2DRong+Wen%2C+and+Jiawei+Han%2E+2023%2E+Don%E2%80%99t+make+your+llm+an+evaluation+benchmark+cheater%2E+arXiv+preprint+arXiv%3A2311%2E)
64. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593. [find](https://scholar.google.com/scholar?q=Daniel+M+Ziegler%2C+Nisan+Stiennon%2C+Jeffrey+Wu%2C+Tom+B+Brown%2C+Alec+Radford%2C+Dario+Amodei%2C+Paul+Christiano%2C+and+Geoffrey+Irving%2E+2019%2E+Fine%2Dtuning+language+models+from+human+preferences%2E+arXiv+preprint+ar)

*Updated 24 October 2026*