---
title: "TinyLlama: An Open-Source Small Language Model — Zhang 2024"
description: "Scaling laws such as Chinchilla's say parameters and data should grow together."
authors: ["Peiyuan Zhang", "Guangtao Zeng", "Tianduo Wang", "Wei Lu"]
year: 2024
venue: "arXiv:2401.02385"
source: "https://arxiv.org/abs/2401.02385"
doi: "10.48550/arxiv.2401.02385"
arxiv: "2401.02385"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2024-tinyllama-open-source-small-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2024-tinyllama-open-source-small-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / TinyLlama: An Open-Source Small Language Model

# TinyLlama: An Open-Source Small Language Model

**Authors:** [Peiyuan Zhang](https://md.research.elamur.ai/free/en/authors/peiyuan-zhang/), Guangtao Zeng, Tianduo Wang, [Wei Lu](https://md.research.elamur.ai/free/en/authors/wei-lu/) · **Year:** 2024 · **Venue:** arXiv:2401.02385 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2401.02385) · [PDF](https://arxiv.org/pdf/2401.02385) · [DOI 10.48550/arxiv.2401.02385](https://doi.org/10.48550/arxiv.2401.02385) · [arXiv 2401.02385](https://arxiv.org/abs/2401.02385) · [OpenAlex](https://openalex.org/W4390632065)

**Citations:** 63 citations (openalex-snapshot · 2026-10-14) · [286](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Scaling laws such as Chinchilla's say parameters and data should grow together. For deployment, though, inference cost matters more, and it was unclear how well a small model does when trained on far more data than those laws recommend. The authors test this with TinyLlama, a 1.1B-parameter decoder-only Transformer that reuses the Llama 2 architecture and tokenizer. They pretrain it on a ~950B-token mix of SlimPajama and StarCoder data for about three epochs, roughly 3T tokens in total. FlashAttention-2, FSDP, fused xFormers kernels and the lit-gpt framework make training cheaper. They then retrain from scratch as TinyLlama v1.1, which uses 2T tokens and three stages: general pretraining, domain-focused continual pretraining and a cooldown. This produces general, Math&Code and Chinese variants. On zero-shot commonsense benchmarks and InstructEval, TinyLlama scores higher than OPT-1.3B and Pythia-1.0B/1.4B, and each specialised variant gains clearly in its own domain. The code, intermediate checkpoints and data-processing details are all public, so the model is a cheap testbed for language-model research and on-device use.

## Key points
- Data: SlimPajama (a cleaned, deduplicated RedPajama with its GitHub part removed) plus StarCoder code data, sampled at about 7:3. Attention uses grouped-query attention with 32 query heads and 4 key-value groups.
- Efficiency: about 24,000 tokens/s per A100-40G GPU. Training on 300B tokens takes 3,456 GPU hours, against 4,830 for Pythia-1.0B and 7,920 for MPT-1.3B.
- v1.1 schedule: 1.5T tokens of SlimPajama only, then 350B tokens of domain-specific continual pretraining (Starcoder Python/Jupyter plus Proof Pile 2 for Math&Code, Skypile for Chinese), then a cooldown.
- Average commonsense score: 52.99 for v1.0 and 53.63 for v1.1, against 51.44 for OPT-1.3B and 51.33 for Pythia-1.4B. The Math&Code variant reaches 15.24 on HumanEval, compared with 4.27 for Pythia-1.4B.
- The Chinese variant averages 58.37 across xwinograd, xstorycloze, xnli and xcopa (Chinese only), against 49.86 for Pythia-1.4B. The authors link part of the gains from code-heavy training to Chinese text found inside the code corpora.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Mathematics & statistics](https://md.research.elamur.ai/free/en/shelves/mathematics/)

## Cited by

Cited by

286 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 280 not yet.

- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://md.research.elamur.ai/free/en/papers/cai-2024-medusa-simple-llm-inference-acceleration/)
- [Kuwain 1.5B: An Arabic SLM via Language Injection](https://md.research.elamur.ai/free/en/papers/hennara-2025-kuwain-1-5b-arabic-slm/)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://md.research.elamur.ai/free/en/papers/hu-2024-minicpm-unveiling-potential-small-language/)
- [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://md.research.elamur.ai/free/en/papers/li-2024-eagle-speculative-sampling-requires-rethinking/)
- [EuroLLM: Multilingual Language Models for Europe](https://md.research.elamur.ai/free/en/papers/martins-2024-eurollm-multilingual-language-models-europe/)
- [Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhang2024tinyllama,
  title = {{TinyLlama: An Open-Source Small Language Model}},
  author = {Zhang, Peiyuan and Zeng, Guangtao and Wang, Tianduo and Lu, Wei},
  year = {2024},
  doi = {10.48550/arxiv.2401.02385},
  eprint = {2401.02385},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.02385}
}
```

- [← previous Tibyan Corpus: Balanced and Comprehensive Error Coverage Corpus Using ChatGPT for Arabic Grammatical Error Correction](https://md.research.elamur.ai/free/en/papers/alrehili-2024-tibyan-corpus-balanced-comprehensive-error/)
- [next → To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning](https://md.research.elamur.ai/free/en/papers/sprague-2024-cot-cot-chain-thought-helps/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

47 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., and Sanghai, S. (2023). GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Ainslie%2C+J%2E%2C+Lee%2DThorp%2C+J%2E%2C+de+Jong%2C+M%2E%2C+Zemlyanskiy%2C+Y%2E%2C+Lebron%2C+F%2E%2C+and+Sanghai%2C+S%2E+%282023%29%2E+GQA%3A+Training+generalized+multi%2Dquery+transformer+models+from+multi%2Dhead+checkpoints%2E+In+Proceedings+of+EM) [our card](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/)
2. Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., Chu, E., Clark, J. H., Shafey, L. E., Huang, Y., Meier-Hellstern, K., Mishra, G., Moreira, E., Omernick, M., Robinson, K., Ruder, S., Tay, Y., Xiao, K., Xu, Y., Zhang, Y., Abrego, G… [find](https://scholar.google.com/scholar?q=Anil%2C+R%2E%2C+Dai%2C+A%2E+M%2E%2C+Firat%2C+O%2E%2C+Johnson%2C+M%2E%2C+Lepikhin%2C+D%2E%2C+Passos%2C+A%2E%2C+Shakeri%2C+S%2E%2C+Taropa%2C+E%2E%2C+Bailey%2C+P%2E%2C+Chen%2C+Z%2E%2C+Chu%2C+E%2E%2C+Clark%2C+J%2E+H%2E%2C+Shafey%2C+L%2E+E%2E%2C+Huang%2C+Y%2E%2C+Meier%2DHellstern%2C+K%2E%2C+Mishra%2C+G%2E%2C) [our card](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/)
3. Azerbayev, Z., Schoelkopf, H., Paster, K., Santos, M. D., McAleer, S., Jiang, A. Q., Deng, J., Biderman, S., and Welleck, S. (2023). Llemma: An open language model for mathematics. arXiv preprint arXiv:2310.10631. [find](https://scholar.google.com/scholar?q=Azerbayev%2C+Z%2E%2C+Schoelkopf%2C+H%2E%2C+Paster%2C+K%2E%2C+Santos%2C+M%2E+D%2E%2C+McAleer%2C+S%2E%2C+Jiang%2C+A%2E+Q%2E%2C+Deng%2C+J%2E%2C+Biderman%2C+S%2E%2C+and+Welleck%2C+S%2E+%282023%29%2E+Llemma%3A+An+open+language+model+for+mathematics%2E+arXiv+preprint+arXi) [our card](https://md.research.elamur.ai/free/en/papers/azerbayev-2023-llemma-open-language-model-mathematics/)
4. Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., Hui, B., Ji, L., Li, M., Lin, J., Lin, R., Liu, D., Liu, G., Lu, C., Lu, K., Ma, J., Men, R., Ren, X., Ren, X., Tan, C., Tan, S., Tu, J., Wang, P., Wang, S., Wang, W., Wu, S., Xu, B., Xu, J., Yang, A., Yang,… [find](https://scholar.google.com/scholar?q=Bai%2C+J%2E%2C+Bai%2C+S%2E%2C+Chu%2C+Y%2E%2C+Cui%2C+Z%2E%2C+Dang%2C+K%2E%2C+Deng%2C+X%2E%2C+Fan%2C+Y%2E%2C+Ge%2C+W%2E%2C+Han%2C+Y%2E%2C+Huang%2C+F%2E%2C+Hui%2C+B%2E%2C+Ji%2C+L%2E%2C+Li%2C+M%2E%2C+Lin%2C+J%2E%2C+Lin%2C+R%2E%2C+Liu%2C+D%2E%2C+Liu%2C+G%2E%2C+Lu%2C+C%2E%2C+Lu%2C+K%2E%2C+Ma%2C+J%2E%2C+Men%2C+R%2E%2C+Ren%2C+X%2E%2C+Ren%2C) [our card](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/)
5. Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., et al. (2024). Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954. [find](https://scholar.google.com/scholar?q=Bi%2C+X%2E%2C+Chen%2C+D%2E%2C+Chen%2C+G%2E%2C+Chen%2C+S%2E%2C+Dai%2C+D%2E%2C+Deng%2C+C%2E%2C+Ding%2C+H%2E%2C+Dong%2C+K%2E%2C+Du%2C+Q%2E%2C+Fu%2C+Z%2E%2C+et+al%2E+%282024%29%2E+Deepseek+llm%3A+Scaling+open%2Dsource+language+models+with+longtermism%2E+arXiv+preprint+arXiv%3A240)
6. Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. (2023). Pythia: A suite for analyzing large language models across training and scaling. In Proceedings of ICML. [find](https://scholar.google.com/scholar?q=Biderman%2C+S%2E%2C+Schoelkopf%2C+H%2E%2C+Anthony%2C+Q%2E+G%2E%2C+Bradley%2C+H%2E%2C+O%E2%80%99Brien%2C+K%2E%2C+Hallahan%2C+E%2E%2C+Khan%2C+M%2E+A%2E%2C+Purohit%2C+S%2E%2C+Prashanth%2C+U%2E+S%2E%2C+Raff%2C+E%2E%2C+et+al%2E+%282023%29%2E+Pythia%3A+A+suite+for+analyzing+large+language+)
7. Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. (2020). Piqa: Reasoning about physical commonsense in natural language. In Proceedings of AAAI. [find](https://scholar.google.com/scholar?q=Bisk%2C+Y%2E%2C+Zellers%2C+R%2E%2C+Gao%2C+J%2E%2C+Choi%2C+Y%2E%2C+et+al%2E+%282020%29%2E+Piqa%3A+Reasoning+about+physical+commonsense+in+natural+language%2E+In+Proceedings+of+AAAI%2E) [our card](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/)
8. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess… [find](https://scholar.google.com/scholar?q=Brown%2C+T%2E%2C+Mann%2C+B%2E%2C+Ryder%2C+N%2E%2C+Subbiah%2C+M%2E%2C+Kaplan%2C+J%2E+D%2E%2C+Dhariwal%2C+P%2E%2C+Neelakantan%2C+A%2E%2C+Shyam%2C+P%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+Agarwal%2C+S%2E%2C+Herbert%2DVoss%2C+A%2E%2C+Krueger%2C+G%2E%2C+Henighan%2C+T%2E%2C+Child%2C+R%2E%2C+Rames)
9. Chia, Y. K., Hong, P., Bing, L., and Poria, S. (2023). INSTRUCTEVAL: towards holistic evaluation of instruction-tuned large language models. CoRR, abs/2306.04757. [find](https://scholar.google.com/scholar?q=Chia%2C+Y%2E+K%2E%2C+Hong%2C+P%2E%2C+Bing%2C+L%2E%2C+and+Poria%2C+S%2E+%282023%29%2E+INSTRUCTEVAL%3A+towards+holistic+evaluation+of+instruction%2Dtuned+large+language+models%2E+CoRR%2C+abs%2F2306%2E04757%2E)
10. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022). Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. [find](https://scholar.google.com/scholar?q=Chowdhery%2C+A%2E%2C+Narang%2C+S%2E%2C+Devlin%2C+J%2E%2C+Bosma%2C+M%2E%2C+Mishra%2C+G%2E%2C+Roberts%2C+A%2E%2C+Barham%2C+P%2E%2C+Chung%2C+H%2E+W%2E%2C+Sutton%2C+C%2E%2C+Gehrmann%2C+S%2E%2C+et+al%2E+%282022%29%2E+Palm%3A+Scaling+language+modeling+with+pathways%2E+arXiv+prepr)
11. Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. (2019). BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of NAACL. [find](https://scholar.google.com/scholar?q=Clark%2C+C%2E%2C+Lee%2C+K%2E%2C+Chang%2C+M%2E%2DW%2E%2C+Kwiatkowski%2C+T%2E%2C+Collins%2C+M%2E%2C+and+Toutanova%2C+K%2E+%282019%29%2E+BoolQ%3A+Exploring+the+surprising+difficulty+of+natural+yes%2Fno+questions%2E+In+Proceedings+of+NAACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/)
12. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. (2018). Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. [find](https://scholar.google.com/scholar?q=Clark%2C+P%2E%2C+Cowhey%2C+I%2E%2C+Etzioni%2C+O%2E%2C+Khot%2C+T%2E%2C+Sabharwal%2C+A%2E%2C+Schoenick%2C+C%2E%2C+and+Tafjord%2C+O%2E+%282018%29%2E+Think+you+have+solved+question+answering%3F+try+arc%2C+the+ai2+reasoning+challenge%2E+arXiv+preprint+arXiv)
13. Conneau, A., Rinott, R., Lample, G., Williams, A., Bowman, S., Schwenk, H., and Stoyanov, V. (2018). XNLI: Evaluating cross-lingual sentence representations. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J., editors, Proceedings of EMNLP, pages 2475–2485, Brussels, Belgium. Association for… [find](https://scholar.google.com/scholar?q=Conneau%2C+A%2E%2C+Rinott%2C+R%2E%2C+Lample%2C+G%2E%2C+Williams%2C+A%2E%2C+Bowman%2C+S%2E%2C+Schwenk%2C+H%2E%2C+and+Stoyanov%2C+V%2E+%282018%29%2E+XNLI%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+In+Riloff%2C+E%2E%2C+Chiang%2C+D%2E%2C+Hockenmaier%2C+J%2E)
14. Dao, T. (2023). Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691. [find](https://scholar.google.com/scholar?q=Dao%2C+T%2E+%282023%29%2E+Flashattention%2D2%3A+Faster+attention+with+better+parallelism+and+work+partitioning%2E+arXiv+preprint+arXiv%3A2307%2E08691%2E)
15. Dauphin, Y. N., Fan, A., Auli, M., and Grangier, D. (2017). Language modeling with gated convolutional networks. In Proceedings of ICML. [find](https://scholar.google.com/scholar?q=Dauphin%2C+Y%2E+N%2E%2C+Fan%2C+A%2E%2C+Auli%2C+M%2E%2C+and+Grangier%2C+D%2E+%282017%29%2E+Language+modeling+with+gated+convolutional+networks%2E+In+Proceedings+of+ICML%2E) [our card](https://md.research.elamur.ai/free/en/papers/dauphin-2016-language-modeling-gated-convolutional-networks/)
16. Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., and Gardner, M. (2019). DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of NAACL. [find](https://scholar.google.com/scholar?q=Dua%2C+D%2E%2C+Wang%2C+Y%2E%2C+Dasigi%2C+P%2E%2C+Stanovsky%2C+G%2E%2C+Singh%2C+S%2E%2C+and+Gardner%2C+M%2E+%282019%29%2E+DROP%3A+A+reading+comprehension+benchmark+requiring+discrete+reasoning+over+paragraphs%2E+In+Proceedings+of+NAACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/)
17. Emelin, D. and Sennrich, R. (2021). Wino-X: Multilingual Winograd schemas for commonsense reasoning and coreference resolution. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t., editors, Proceedings of EMNLP, pages 8517–8532, Online and Punta Cana, Dominican Republic. Association for Comput… [find](https://scholar.google.com/scholar?q=Emelin%2C+D%2E+and+Sennrich%2C+R%2E+%282021%29%2E+Wino%2DX%3A+Multilingual+Winograd+schemas+for+commonsense+reasoning+and+coreference+resolution%2E+In+Moens%2C+M%2E%2DF%2E%2C+Huang%2C+X%2E%2C+Specia%2C+L%2E%2C+and+Yih%2C+S%2E+W%2E%2Dt%2E%2C+editors%2C+Proc) [our card](https://md.research.elamur.ai/free/en/papers/emelin-2021-wino-x-multilingual-winograd-schemas/)
18. Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac’h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. (2023). A fr… [find](https://scholar.google.com/scholar?q=Gao%2C+L%2E%2C+Tow%2C+J%2E%2C+Abbasi%2C+B%2E%2C+Biderman%2C+S%2E%2C+Black%2C+S%2E%2C+DiPofi%2C+A%2E%2C+Foster%2C+C%2E%2C+Golding%2C+L%2E%2C+Hsu%2C+J%2E%2C+Le+Noac%E2%80%99h%2C+A%2E%2C+Li%2C+H%2E%2C+McDonell%2C+K%2E%2C+Muennighoff%2C+N%2E%2C+Ociepa%2C+C%2E%2C+Phang%2C+J%2E%2C+Reynolds%2C+L%2E%2C+Schoelko)
19. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2021). Measuring massive multitask language understanding. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Hendrycks%2C+D%2E%2C+Burns%2C+C%2E%2C+Basart%2C+S%2E%2C+Zou%2C+A%2E%2C+Mazeika%2C+M%2E%2C+Song%2C+D%2E%2C+and+Steinhardt%2C+J%2E+%282021%29%2E+Measuring+massive+multitask+language+understanding%2E+In+Proceedings+of+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
20. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J. W., and Sifre, L.… [find](https://scholar.google.com/scholar?q=Hoffmann%2C+J%2E%2C+Borgeaud%2C+S%2E%2C+Mensch%2C+A%2E%2C+Buchatskaya%2C+E%2E%2C+Cai%2C+T%2E%2C+Rutherford%2C+E%2E%2C+de+las+Casas%2C+D%2E%2C+Hendricks%2C+L%2E+A%2E%2C+Welbl%2C+J%2E%2C+Clark%2C+A%2E%2C+Hennigan%2C+T%2E%2C+Noland%2C+E%2E%2C+Millican%2C+K%2E%2C+van+den+Driessche%2C+G)
21. Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Fang, Y., Huang, Y., Zhao, W., et al. (2024). Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395. [find](https://scholar.google.com/scholar?q=Hu%2C+S%2E%2C+Tu%2C+Y%2E%2C+Han%2C+X%2E%2C+He%2C+C%2E%2C+Cui%2C+G%2E%2C+Long%2C+X%2E%2C+Zheng%2C+Z%2E%2C+Fang%2C+Y%2E%2C+Huang%2C+Y%2E%2C+Zhao%2C+W%2E%2C+et+al%2E+%282024%29%2E+Minicpm%3A+Unveiling+the+potential+of+small+language+models+with+scalable+training+strategies) [our card](https://md.research.elamur.ai/free/en/papers/hu-2024-minicpm-unveiling-potential-small-language/)
22. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. [find](https://scholar.google.com/scholar?q=Kaplan%2C+J%2E%2C+McCandlish%2C+S%2E%2C+Henighan%2C+T%2E%2C+Brown%2C+T%2E+B%2E%2C+Chess%2C+B%2E%2C+Child%2C+R%2E%2C+Gray%2C+S%2E%2C+Radford%2C+A%2E%2C+Wu%2C+J%2E%2C+and+Amodei%2C+D%2E+%282020%29%2E+Scaling+laws+for+neural+language+models%2E+arXiv+preprint+arXiv%3A2001%2E0)
23. Lefaudeux, B., Massa, F., Liskovich, D., Xiong, W., Caggiano, V., Naren, S., Xu, M., Hu, J., Tintore, M., Zhang, S., Labatut, P., and Haziza, D. (2022). xformers: A modular and hackable transformer modelling library. [https://github.com/facebookresearch/xformers](https://github.com/facebookresearch/xformers). [↗](https://github.com/facebookresearch/xformers)
24. Li, R., allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., LI, J., Chim, J., Liu, Q., Zheltonozhskii, E., Zhuo, T. Y., Wang, T., Dehaene, O., Lamy-Poirier, J., Monteiro, J., Gontier, N., Yee, M.-H., Umapathi, L. K., Zhu, J., Lipkin, B., Oblokulov, M., Wang, Z., Murt… [find](https://scholar.google.com/scholar?q=Li%2C+R%2E%2C+allal%2C+L%2E+B%2E%2C+Zi%2C+Y%2E%2C+Muennighoff%2C+N%2E%2C+Kocetkov%2C+D%2E%2C+Mou%2C+C%2E%2C+Marone%2C+M%2E%2C+Akiki%2C+C%2E%2C+LI%2C+J%2E%2C+Chim%2C+J%2E%2C+Liu%2C+Q%2E%2C+Zheltonozhskii%2C+E%2E%2C+Zhuo%2C+T%2E+Y%2E%2C+Wang%2C+T%2E%2C+Dehaene%2C+O%2E%2C+Lamy%2DPoirier%2C+J%2E%2C+Montei)
25. Lightning-AI (2023). Lit-gpt. [find](https://scholar.google.com/scholar?q=Lightning%2DAI+%282023%29%2E+Lit%2Dgpt%2E)
26. Lin, X. V., Mihaylov, T., Artetxe, M., Wang, T., Chen, S., Simig, D., Ott, M., Goyal, N., Bhosale, S., Du, J., Pasunuru, R., Shleifer, S., Koura, P. S., Chaudhary, V., O’Horo, B., Wang, J., Zettlemoyer, L., Kozareva, Z., Diab, M., Stoyanov, V., and Li, X. (2022). Few-shot learning with multilingual… [find](https://scholar.google.com/scholar?q=Lin%2C+X%2E+V%2E%2C+Mihaylov%2C+T%2E%2C+Artetxe%2C+M%2E%2C+Wang%2C+T%2E%2C+Chen%2C+S%2E%2C+Simig%2C+D%2E%2C+Ott%2C+M%2E%2C+Goyal%2C+N%2E%2C+Bhosale%2C+S%2E%2C+Du%2C+J%2E%2C+Pasunuru%2C+R%2E%2C+Shleifer%2C+S%2E%2C+Koura%2C+P%2E+S%2E%2C+Chaudhary%2C+V%2E%2C+O%E2%80%99Horo%2C+B%2E%2C+Wang%2C+J%2E%2C+Zettlemoye) [our card](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/)
27. Loshchilov, I. and Hutter, F. (2019). Decoupled weight decay regularization. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Loshchilov%2C+I%2E+and+Hutter%2C+F%2E+%282019%29%2E+Decoupled+weight+decay+regularization%2E+In+Proceedings+of+ICLR%2E)
28. Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. (2018). Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Mihaylov%2C+T%2E%2C+Clark%2C+P%2E%2C+Khot%2C+T%2E%2C+and+Sabharwal%2C+A%2E+%282018%29%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+In+Proceedings+of+EMNLP%2E)
29. OpenAI (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774. [find](https://scholar.google.com/scholar?q=OpenAI+%282023%29%2E+Gpt%2D4+technical+report%2E+arXiv+preprint+arXiv%3A2303%2E08774%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
30. Ponti, E. M., Glavaš, G., Majewska, O., Liu, Q., Vulić, I., and Korhonen, A. (2020). XCOPA: A multilingual dataset for causal commonsense reasoning. In Webber, B., Cohn, T., He, Y., and Liu, Y., editors, Proceedings of EMNLP, pages 2362–2376, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ponti%2C+E%2E+M%2E%2C+Glava%C5%A1%2C+G%2E%2C+Majewska%2C+O%2E%2C+Liu%2C+Q%2E%2C+Vulic%CC%81%2C+I%2E%2C+and+Korhonen%2C+A%2E+%282020%29%2E+XCOPA%3A+A+multilingual+dataset+for+causal+commonsense+reasoning%2E+In+Webber%2C+B%2E%2C+Cohn%2C+T%2E%2C+He%2C+Y%2E%2C+and+Liu%2C+Y%2E%2C+edit) [our card](https://md.research.elamur.ai/free/en/papers/ponti-2020-xcopa-multilingual-dataset-causal-commonsense/)
31. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. (2021). Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. [find](https://scholar.google.com/scholar?q=Sakaguchi%2C+K%2E%2C+Bras%2C+R%2E+L%2E%2C+Bhagavatula%2C+C%2E%2C+and+Choi%2C+Y%2E+%282021%29%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Communications+of+the+ACM%2C+64%289%29%3A99%E2%80%93106%2E)
32. Shazeer, N. (2020). GLU variants improve transformer. CoRR, abs/2002.05202. [find](https://scholar.google.com/scholar?q=Shazeer%2C+N%2E+%282020%29%2E+GLU+variants+improve+transformer%2E+CoRR%2C+abs%2F2002%2E05202%2E)
33. Soboleva, D., Al-Khateeb, F., Myers, R., Steeves, J. R., Hestness, J., and Dey, N. (2023). SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. [find](https://scholar.google.com/scholar?q=Soboleva%2C+D%2E%2C+Al%2DKhateeb%2C+F%2E%2C+Myers%2C+R%2E%2C+Steeves%2C+J%2E+R%2E%2C+Hestness%2C+J%2E%2C+and+Dey%2C+N%2E+%282023%29%2E+SlimPajama%3A+A+627B+token+cleaned+and+deduplicated+version+of+RedPajama%2E)
34. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615. [find](https://scholar.google.com/scholar?q=Srivastava%2C+A%2E%2C+Rastogi%2C+A%2E%2C+Rao%2C+A%2E%2C+Shoeb%2C+A%2E+A%2E+M%2E%2C+Abid%2C+A%2E%2C+Fisch%2C+A%2E%2C+Brown%2C+A%2E+R%2E%2C+Santoro%2C+A%2E%2C+Gupta%2C+A%2E%2C+Garriga%2DAlonso%2C+A%2E%2C+et+al%2E+%282022%29%2E+Beyond+the+imitation+game%3A+Quantifying+and+extrapol) [our card](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
35. Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. (2021). Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864. [find](https://scholar.google.com/scholar?q=Su%2C+J%2E%2C+Lu%2C+Y%2E%2C+Pan%2C+S%2E%2C+Murtadha%2C+A%2E%2C+Wen%2C+B%2E%2C+and+Liu%2C+Y%2E+%282021%29%2E+Roformer%3A+Enhanced+transformer+with+rotary+position+embedding%2E+arXiv+preprint+arXiv%3A2104%2E09864%2E)
36. Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q., Chi, E., Zhou, D., and Wei, J. (2023). Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Findings of ACL. [find](https://scholar.google.com/scholar?q=Suzgun%2C+M%2E%2C+Scales%2C+N%2E%2C+Sch%C3%A4rli%2C+N%2E%2C+Gehrmann%2C+S%2E%2C+Tay%2C+Y%2E%2C+Chung%2C+H%2E+W%2E%2C+Chowdhery%2C+A%2E%2C+Le%2C+Q%2E%2C+Chi%2C+E%2E%2C+Zhou%2C+D%2E%2C+and+Wei%2C+J%2E+%282023%29%2E+Challenging+BIG%2Dbench+tasks+and+whether+chain%2Dof%2Dthought+can+sol) [our card](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/)
37. Thaddée, Y. T. (2023). Chinchilla’s death. [https://espadrine.github.io/blog/posts/chinchilla-s-death.html](https://espadrine.github.io/blog/posts/chinchilla-s-death.html). [↗](https://espadrine.github.io/blog/posts/chinchilla-s-death.html)
38. Together Computer (2023). Redpajama: an open dataset for training large language models. [find](https://scholar.google.com/scholar?q=Together+Computer+%282023%29%2E+Redpajama%3A+an+open+dataset+for+training+large+language+models%2E) [our card](https://md.research.elamur.ai/free/en/papers/weber-2024-redpajama-open-dataset-training-large/)
39. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023a). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. [find](https://scholar.google.com/scholar?q=Touvron%2C+H%2E%2C+Lavril%2C+T%2E%2C+Izacard%2C+G%2E%2C+Martinet%2C+X%2E%2C+Lachaux%2C+M%2E%2DA%2E%2C+Lacroix%2C+T%2E%2C+Rozi%C3%A8re%2C+B%2E%2C+Goyal%2C+N%2E%2C+Hambro%2C+E%2E%2C+Azhar%2C+F%2E%2C+et+al%2E+%282023a%29%2E+Llama%3A+Open+and+efficient+foundation+language+models%2E+ar)
40. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023b). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Touvron%2C+H%2E%2C+Martin%2C+L%2E%2C+Stone%2C+K%2E%2C+Albert%2C+P%2E%2C+Almahairi%2C+A%2E%2C+Babaei%2C+Y%2E%2C+Bashlykov%2C+N%2E%2C+Batra%2C+S%2E%2C+Bhargava%2C+P%2E%2C+Bhosale%2C+S%2E%2C+et+al%2E+%282023b%29%2E+Llama+2%3A+Open+foundation+and+fine%2Dtuned+chat+models%2E+arX)
41. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. In Proceedings of NeurIPS. [find](https://scholar.google.com/scholar?q=Vaswani%2C+A%2E%2C+Shazeer%2C+N%2E%2C+Parmar%2C+N%2E%2C+Uszkoreit%2C+J%2E%2C+Jones%2C+L%2E%2C+Gomez%2C+A%2E+N%2E%2C+Kaiser%2C+L%2E%2C+and+Polosukhin%2C+I%2E+%282017%29%2E+Attention+is+all+you+need%2E+In+Proceedings+of+NeurIPS%2E)
42. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. (2022). Chain of thought prompting elicits reasoning in large language models. In Proceedings of NeurIPS. [find](https://scholar.google.com/scholar?q=Wei%2C+J%2E%2C+Wang%2C+X%2E%2C+Schuurmans%2C+D%2E%2C+Bosma%2C+M%2E%2C+Chi%2C+E%2E%2C+Le%2C+Q%2E%2C+and+Zhou%2C+D%2E+%282022%29%2E+Chain+of+thought+prompting+elicits+reasoning+in+large+language+models%2E+In+Proceedings+of+NeurIPS%2E) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
43. Wei, T., Zhao, L., Zhang, L., Zhu, B., Wang, L., Yang, H., Li, B., Cheng, C., Lü, W., Hu, R., et al. (2023). Skywork: A more open bilingual foundation model. arXiv preprint arXiv:2310.19341. [find](https://scholar.google.com/scholar?q=Wei%2C+T%2E%2C+Zhao%2C+L%2E%2C+Zhang%2C+L%2E%2C+Zhu%2C+B%2E%2C+Wang%2C+L%2E%2C+Yang%2C+H%2E%2C+Li%2C+B%2E%2C+Cheng%2C+C%2E%2C+L%C3%BC%2C+W%2E%2C+Hu%2C+R%2E%2C+et+al%2E+%282023%29%2E+Skywork%3A+A+more+open+bilingual+foundation+model%2E+arXiv+preprint+arXiv%3A2310%2E19341%2E) [our card](https://md.research.elamur.ai/free/en/papers/wei-2023-skywork-more-open-bilingual/)
44. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. (2019). HellaSwag: Can a machine really finish your sentence? In Proceedings of the ACL. [find](https://scholar.google.com/scholar?q=Zellers%2C+R%2E%2C+Holtzman%2C+A%2E%2C+Bisk%2C+Y%2E%2C+Farhadi%2C+A%2E%2C+and+Choi%2C+Y%2E+%282019%29%2E+HellaSwag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+ACL%2E)
45. Zhang, B. and Sennrich, R. (2019). Root mean square layer normalization. In Proceedings of NeurIPS. [find](https://scholar.google.com/scholar?q=Zhang%2C+B%2E+and+Sennrich%2C+R%2E+%282019%29%2E+Root+mean+square+layer+normalization%2E+In+Proceedings+of+NeurIPS%2E)
46. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022). Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. [find](https://scholar.google.com/scholar?q=Zhang%2C+S%2E%2C+Roller%2C+S%2E%2C+Goyal%2C+N%2E%2C+Artetxe%2C+M%2E%2C+Chen%2C+M%2E%2C+Chen%2C+S%2E%2C+Dewan%2C+C%2E%2C+Diab%2C+M%2E%2C+Li%2C+X%2E%2C+Lin%2C+X%2E+V%2E%2C+et+al%2E+%282022%29%2E+Opt%3A+Open+pre%2Dtrained+transformer+language+models%2E+arXiv+preprint+arXiv%3A2205%2E)
47. Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Shen, L., Wang, Z., Wang, A., Li, Y., Su, T., Yang, Z., and Tang, J. (2023). Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discov… [find](https://scholar.google.com/scholar?q=Zheng%2C+Q%2E%2C+Xia%2C+X%2E%2C+Zou%2C+X%2E%2C+Dong%2C+Y%2E%2C+Wang%2C+S%2E%2C+Xue%2C+Y%2E%2C+Shen%2C+L%2E%2C+Wang%2C+Z%2E%2C+Wang%2C+A%2E%2C+Li%2C+Y%2E%2C+Su%2C+T%2E%2C+Yang%2C+Z%2E%2C+and+Tang%2C+J%2E+%282023%29%2E+Codegeex%3A+A+pre%2Dtrained+model+for+code+generation+with+multilingu) [our card](https://md.research.elamur.ai/free/en/papers/zheng-2023-codegeex-pre-trained-model-code/)