---
title: "Large Language Models: A Survey — Minaee 2024"
description: "Research on large language models moves so fast that students, researchers and engineers struggle to tell which recipes actually work."
authors: ["Shervin Minaee", "Tomas Mikolov", "Narjes Nikzad", "Meysam Chenaghlu", "Richard Socher", "Xavier Amatriain", "Jianfeng Gao"]
year: 2024
venue: "arXiv:2402.06196"
source: "https://arxiv.org/abs/2402.06196"
doi: "10.48550/arxiv.2402.06196"
arxiv: "2402.06196"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / Large Language Models: A Survey

# Large Language Models: A Survey

**Authors:** Shervin Minaee, [Tomas Mikolov](https://md.research.elamur.ai/free/en/authors/tomas-mikolov/), Narjes Nikzad, Meysam Chenaghlu, [Richard Socher](https://md.research.elamur.ai/free/en/authors/richard-socher/), Xavier Amatriain, [Jianfeng Gao](https://md.research.elamur.ai/free/en/authors/jianfeng-gao/) · **Year:** 2024 · **Venue:** arXiv:2402.06196 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.06196) · [PDF](https://arxiv.org/pdf/2402.06196) · [DOI 10.48550/arxiv.2402.06196](https://doi.org/10.48550/arxiv.2402.06196) · [arXiv 2402.06196](https://arxiv.org/abs/2402.06196) · [OpenAlex](https://openalex.org/W4391766565)

**Citations:** 231 citations (openalex-snapshot · 2026-10-14) · [236](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Research on large language models moves so fast that students, researchers and engineers struggle to tell which recipes actually work. This survey gives a structured map of the field. It traces language modeling through four stages: statistical n-gram models, neural language models, pre-trained models such as BERT, and today's LLMs with tens to hundreds of billions of parameters. Most attention goes to three major model families, GPT, LLaMA and PaLM, with notes on what each contributed and where each falls short. The authors then explain how LLMs are built and how they are extended with retrieval, external tools and agent frameworks. They also catalog the datasets used for pre-training, fine-tuning and evaluation, review common metrics, and compare published results of popular models on representative benchmarks. The paper closes with open problems, including non-transformer designs such as state space models. For a team starting work on Ukrainian NLP, it is a practical reference for the vocabulary, model landscape and tooling around modern LLMs.

## Key points
- The survey sorts LLM abilities into three groups. Basic abilities include comprehension, multilingual tasks and coding. Emergent ones are in-context learning, instruction following and multi-step reasoning such as chain-of-thought. Augmented ones cover tool use, planning and interacting with users.
- Early pre-trained models are split by architecture into encoder-only, decoder-only and encoder-decoder. On the encoder side it covers BERT, RoBERTa, ALBERT, DeBERTa and ELECTRA.
- It lists practical tooling:
  - Adaptation and training: LoRA, which freezes the base weights and trains low-rank matrices, and ColossalAI for distributed training.
  - Serving and deployment: vLLM, text-generation-inference, FastChat and OpenLLM.
  - Application frameworks: LangChain, Embedchain and AutoGen.
- Vector stores are also covered: Faiss, Milvus, Qdrant and Weaviate, plus LlamaIndex and Pinecone as further options for RAG.
- The arXiv ID is 2402.06196 (v3, March 2025). Co-authors include Tomas Mikolov, Richard Socher and Jianfeng Gao.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) · [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

## Cited by

Cited by

236 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 229 not yet.

- [Security and Privacy Challenges of Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/das-2024-security-privacy-challenges-large-language/)
- [ALHD: A Large-Scale and Multigenre Benchmark Dataset for Arabic LLM-Generated Text Detection](https://md.research.elamur.ai/free/en/papers/khairallah-2025-alhd-large-scale-multigenre-benchmark/)
- [Can Large Language Models Predict the Outcome of Judicial Decisions?](https://md.research.elamur.ai/free/en/papers/kmainasi-2025-large-language-models-predict-outcome/)
- [A Language Modeling Approach to Diacritic-Free Hebrew TTS](https://md.research.elamur.ai/free/en/papers/langstream-2024-roth-language-modeling-approach-diacritic-free/)
- [From Rosetta to Match-Up: A Paired Corpus of Linguistic Puzzles with Human and LLM Benchmarks](https://md.research.elamur.ai/free/en/papers/majmudar-2026-rosetta-match-up-paired-corpus/)
- [LLMs as Compiler for Arabic Programming Language](https://md.research.elamur.ai/free/en/papers/sibaee-2024-llms-compiler-arabic-programming-language/)
- [Multilingual coreference resolution as text generation](https://md.research.elamur.ai/free/en/papers/skachkova-2024-multilingual-coreference-resolution-as-text/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{minaee2024large,
  title = {{Large Language Models: A Survey}},
  author = {Minaee, Shervin and Mikolov, Tomas and Nikzad, Narjes and Chenaghlu, Meysam and Socher, Richard and Amatriain, Xavier and Gao, Jianfeng},
  year = {2024},
  doi = {10.48550/arxiv.2402.06196},
  eprint = {2402.06196},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.06196}
}
```

- [← previous Large Language Models for Mathematical Reasoning: Progresses and Challenges](https://md.research.elamur.ai/free/en/papers/ahn-2024-large-language-models-mathematical-reasoning/)
- [next → Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://md.research.elamur.ai/free/en/papers/brown-2024-large-language-monkeys-scaling-inference/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

51 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361, 2020. J. Hoffmann, S. B… [find](https://scholar.google.com/scholar?q=%5B1%5D+%5B2%5D+%5B3%5D+%5B4%5D+%5B5%5D+%5B6%5D+%5B7%5D+%5B8%5D+%5B9%5D+%5B10%5D+%5B11%5D+%5B12%5D+%5B13%5D+%5B14%5D+%5B15%5D+%5B16%5D+%5B17%5D+%5B18%5D+%5B19%5D+J%2E+Kaplan%2C+S%2E+McCandlish%2C+T%2E+Henighan%2C+T%2E+B%2E+Brown%2C+B%2E+Chess%2C+R%2E+Child%2C+S%2E+Gray%2C+A%2E+Radford%2C+J%2E+Wu%2C+and+D%2E+Amodei%2C)
2. Makuhari, 2010, pp. 1045–1048. A. Graves, “Generating sequences with recurrent neural networks,” arXiv preprint arXiv:1308.0850, 2013. P.-S. Huang, X. He, J. Gao, L. Deng, A. Acero, and L. Heck, “Learning deep structured semantic models for web search using clickthrough data,” in Proceedings of the… [find](https://scholar.google.com/scholar?q=Makuhari%2C+2010%2C+pp%2E+1045%E2%80%931048%2E+A%2E+Graves%2C+%E2%80%9CGenerating+sequences+with+recurrent+neural+networks%2C%E2%80%9D+arXiv+preprint+arXiv%3A1308%2E0850%2C+2013%2E+P%2E%2DS%2E+Huang%2C+X%2E+He%2C+J%2E+Gao%2C+L%2E+Deng%2C+A%2E+Acero%2C+and+L%2E+Heck%2C+%E2%80%9CLear)
3. Springer Nature, 2023, vol. 44. I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” Advances in neural information processing systems, vol. 27, 2014. K. Cho, B. Van Merriënboer, D. Bahdanau, and Y. Bengio, “On the properties of neural machine translation: E… [find](https://scholar.google.com/scholar?q=Springer+Nature%2C+2023%2C+vol%2E+44%2E+I%2E+Sutskever%2C+O%2E+Vinyals%2C+and+Q%2E+V%2E+Le%2C+%E2%80%9CSequence+to+sequence+learning+with+neural+networks%2C%E2%80%9D+Advances+in+neural+information+processing+systems%2C+vol%2E+27%2C+2014%2E+K%2E+Cho%2C+) [our card](https://md.research.elamur.ai/free/en/papers/cho-2014-properties-neural-machine-translation-encoder/)
4. Report,” [https://arxiv.org/pdf/2303](https://arxiv.org/pdf/2303). tmikolov. rnnlm. [Online]. Available: https://www.fit.vutbr.cz/∼ imikolov/rnnlm/ [43] [44] [45] [46] [47] [48] [49] [50] [51] [52] [53] [54] [55] [56] [57] [58] [59] [60] [61] S. Minaee, N. Kalchbrenner, E. Cambria, N. Nikzad, M. Chenaghlu, and J. Gao, “Deep lear… [↗](https://arxiv.org/pdf/2303)
5. OpenAI. (2022) Introducing chatgpt. [Online]. Available: [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt) H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.0928… [↗](https://openai.com/blog/chatgpt)
6. [64] [65] [66] [67] [68] [69] [70] [71] [72] [73] [74] [75] [76] [77] [78] [79] [80] [81] [82] els. [https://crfm.stanford](https://crfm.stanford). edu/2023/03/13/alpaca. html, vol. 3, no. 6, p. 7, 2023. T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” arXiv preprin… [↗](https://crfm.stanford)
7. [84] [85] [86] [87] [88] [89] [90] [91] [92] [93] [94] [95] [96] [97] [98] [99] [100] [101] O. Lieber, O. Sharir, B. Lenz, and Y. Shoham, “Jurassic-1: Technical details and evaluation,” White Paper. AI21 Labs, vol. 1, p. 9, 2021. N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y.… [find](https://scholar.google.com/scholar?q=%5B84%5D+%5B85%5D+%5B86%5D+%5B87%5D+%5B88%5D+%5B89%5D+%5B90%5D+%5B91%5D+%5B92%5D+%5B93%5D+%5B94%5D+%5B95%5D+%5B96%5D+%5B97%5D+%5B98%5D+%5B99%5D+%5B100%5D+%5B101%5D+O%2E+Lieber%2C+O%2E+Sharir%2C+B%2E+Lenz%2C+and+Y%2E+Shoham%2C+%E2%80%9CJurassic%2D1%3A+Technical+details+and+evaluation%2C%E2%80%9D+White+Paper%2E+A) [our card](https://md.research.elamur.ai/free/en/papers/du-2021-glam-efficient-scaling-language-models/)
8. PMLR, 2022, pp. 5547–5569. R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du et al., “Lamda: Language models for dialog applications,” arXiv preprint arXiv:2201.08239, 2022. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C.… [find](https://scholar.google.com/scholar?q=PMLR%2C+2022%2C+pp%2E+5547%E2%80%935569%2E+R%2E+Thoppilan%2C+D%2E+De+Freitas%2C+J%2E+Hall%2C+N%2E+Shazeer%2C+A%2E+Kulshreshtha%2C+H%2E%2DT%2E+Cheng%2C+A%2E+Jin%2C+T%2E+Bos%2C+L%2E+Baker%2C+Y%2E+Du+et+al%2E%2C+%E2%80%9CLamda%3A+Language+models+for+dialog+applications%2C%E2%80%9D+arX) [our card](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/)
9. PMLR, 2023, pp. 2397–2430. S. Mukherjee, A. Mitra, G. Jawahar, S. Agarwal, H. Palangi, and A. Awadallah, “Orca: Progressive learning from complex explanation traces of gpt-4,” arXiv preprint arXiv:2306.02707, 2023. R. Li, L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki,… [find](https://scholar.google.com/scholar?q=PMLR%2C+2023%2C+pp%2E+2397%E2%80%932430%2E+S%2E+Mukherjee%2C+A%2E+Mitra%2C+G%2E+Jawahar%2C+S%2E+Agarwal%2C+H%2E+Palangi%2C+and+A%2E+Awadallah%2C+%E2%80%9COrca%3A+Progressive+learning+from+complex+explanation+traces+of+gpt%2D4%2C%E2%80%9D+arXiv+preprint+arXiv%3A230) [our card](https://md.research.elamur.ai/free/en/papers/mukherjee-2023-orca-progressive-learning-complex-explanation/)
10. I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document transformer,” arXiv preprint arXiv:2004.05150, 2020. [find](https://scholar.google.com/scholar?q=I%2E+Beltagy%2C+M%2E+E%2E+Peters%2C+and+A%2E+Cohan%2C+%E2%80%9CLongformer%3A+The+long%2Ddocument+transformer%2C%E2%80%9D+arXiv+preprint+arXiv%3A2004%2E05150%2C+2020%2E)
11. [103] S. Iyer, X. V. Lin, R. Pasunuru, T. Mihaylov, D. Simig, P. Yu, K. Shuster, T. Wang, Q. Liu, P. S. Koura et al., “Opt-iml: Scaling language model instruction meta learning through the lens of generalization,” arXiv preprint arXiv:2212.12017, 2022. [find](https://scholar.google.com/scholar?q=%5B103%5D+S%2E+Iyer%2C+X%2E+V%2E+Lin%2C+R%2E+Pasunuru%2C+T%2E+Mihaylov%2C+D%2E+Simig%2C+P%2E+Yu%2C+K%2E+Shuster%2C+T%2E+Wang%2C+Q%2E+Liu%2C+P%2E+S%2E+Koura+et+al%2E%2C+%E2%80%9COpt%2Diml%3A+Scaling+language+model+instruction+meta+learning+through+the+lens+of+gen) [our card](https://md.research.elamur.ai/free/en/papers/iyer-2022-opt-iml-scaling-language-model/)
12. [104] Y. Hao, H. Song, L. Dong, S. Huang, Z. Chi, W. Wang, S. Ma, and F. Wei, “Language models are general-purpose interfaces,” arXiv preprint arXiv:2206.06336, 2022. — [arXiv:2206.06336](https://arxiv.org/abs/2206.06336) [find](https://scholar.google.com/scholar?q=%5B104%5D+Y%2E+Hao%2C+H%2E+Song%2C+L%2E+Dong%2C+S%2E+Huang%2C+Z%2E+Chi%2C+W%2E+Wang%2C+S%2E+Ma%2C+and+F%2E+Wei%2C+%E2%80%9CLanguage+models+are+general%2Dpurpose+interfaces%2C%E2%80%9D+arXiv+preprint+arXiv%3A2206%2E06336%2C+2022%2E+%E2%80%94+%5BarXiv%3A2206%2E06336%5D%28https%3A%2F%2Farxi)
13. [105] Z. Sun, Y. Shen, Q. Zhou, H. Zhang, Z. Chen, D. Cox, Y. Yang, and C. Gan, “Principle-driven self-alignment of language models from scratch with minimal human supervision,” arXiv preprint arXiv:2305.03047, 2023. [find](https://scholar.google.com/scholar?q=%5B105%5D+Z%2E+Sun%2C+Y%2E+Shen%2C+Q%2E+Zhou%2C+H%2E+Zhang%2C+Z%2E+Chen%2C+D%2E+Cox%2C+Y%2E+Yang%2C+and+C%2E+Gan%2C+%E2%80%9CPrinciple%2Ddriven+self%2Dalignment+of+language+models+from+scratch+with+minimal+human+supervision%2C%E2%80%9D+arXiv+preprint+arXiv%3A2) [our card](https://md.research.elamur.ai/free/en/papers/sun-2023-principle-driven-self-alignment-language/)
14. [106] W. E. team, “Palmyra-base Parameter Autoregressive Language Model,” [https://dev.writer.com](https://dev.writer.com), 2023. [↗](https://dev.writer.com)
15. [107] ——, “Camel-5b instructgpt,” [https://dev.writer.com](https://dev.writer.com), 2023. [↗](https://dev.writer.com)
16. [108] Yandex. Yalm. YaLM-100B [109] M. Team et al., “Introducing mpt-7b: a new standard for open-source, commercially usable llms,” 2023. [find](https://scholar.google.com/scholar?q=%5B108%5D+Yandex%2E+Yalm%2E+YaLM%2D100B+%5B109%5D+M%2E+Team+et+al%2E%2C+%E2%80%9CIntroducing+mpt%2D7b%3A+a+new+standard+for+open%2Dsource%2C+commercially+usable+llms%2C%E2%80%9D+2023%2E)
17. [110] A. Mitra, L. D. Corro, S. Mahajan, A. Codas, C. Simoes, S. Agarwal, X. Chen, A. Razdaibiedina, E. Jones, K. Aggarwal, H. Palangi, G. Zheng, C. Rosset, H. Khanpour, and A. Awadallah, “Orca 2: Teaching small language models how to reason,” 2023. [find](https://scholar.google.com/scholar?q=%5B110%5D+A%2E+Mitra%2C+L%2E+D%2E+Corro%2C+S%2E+Mahajan%2C+A%2E+Codas%2C+C%2E+Simoes%2C+S%2E+Agarwal%2C+X%2E+Chen%2C+A%2E+Razdaibiedina%2C+E%2E+Jones%2C+K%2E+Aggarwal%2C+H%2E+Palangi%2C+G%2E+Zheng%2C+C%2E+Rosset%2C+H%2E+Khanpour%2C+and+A%2E+Awadallah%2C+%E2%80%9COrca+2%3A+Tea) [our card](https://md.research.elamur.ai/free/en/papers/mitra-2023-orca-2-teaching-small-language/)
18. [111] L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig, “Pal: Program-aided language models,” in International Conference on Machine Learning. PMLR, 2023, pp. 10 764–10 799. [find](https://scholar.google.com/scholar?q=%5B111%5D+L%2E+Gao%2C+A%2E+Madaan%2C+S%2E+Zhou%2C+U%2E+Alon%2C+P%2E+Liu%2C+Y%2E+Yang%2C+J%2E+Callan%2C+and+G%2E+Neubig%2C+%E2%80%9CPal%3A+Program%2Daided+language+models%2C%E2%80%9D+in+International+Conference+on+Machine+Learning%2E+PMLR%2C+2023%2C+pp%2E+10+764%E2%80%9310+7)
19. [112] Anthropic. claude. [Online]. Available: [https://www.anthropic.com/](https://www.anthropic.com/) news/introducing-claude [113] E. Nijkamp, H. Hayashi, C. Xiong, S. Savarese, and Y. Zhou, “Codegen2: Lessons for training llms on programming and natural languages,” arXiv preprint arXiv:2305.02309, 2023. [↗](https://www.anthropic.com/)
20. [114] L. Tunstall, E. Beeching, N. Lambert, N. Rajani, K. Rasul, Y. Belkada, S. Huang, L. von Werra, C. Fourrier, N. Habib et al., “Zephyr: Direct distillation of lm alignment,” arXiv preprint arXiv:2310.16944, 2023. [find](https://scholar.google.com/scholar?q=%5B114%5D+L%2E+Tunstall%2C+E%2E+Beeching%2C+N%2E+Lambert%2C+N%2E+Rajani%2C+K%2E+Rasul%2C+Y%2E+Belkada%2C+S%2E+Huang%2C+L%2E+von+Werra%2C+C%2E+Fourrier%2C+N%2E+Habib+et+al%2E%2C+%E2%80%9CZephyr%3A+Direct+distillation+of+lm+alignment%2C%E2%80%9D+arXiv+preprint+arXiv%3A2)
21. [115] X. team. Grok. [Online]. Available: https://grok.x.ai/[116] J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou, “Qwen-vl: A frontier large vision-language model with versatile abilities,” arXiv preprint arXiv:2308.12966, 2023. [find](https://scholar.google.com/scholar?q=%5B115%5D+X%2E+team%2E+Grok%2E+%5BOnline%5D%2E+Available%3A+https%3A%2F%2Fgrok%2Ex%2Eai%2F%5B116%5D+J%2E+Bai%2C+S%2E+Bai%2C+S%2E+Yang%2C+S%2E+Wang%2C+S%2E+Tan%2C+P%2E+Wang%2C+J%2E+Lin%2C+C%2E+Zhou%2C+and+J%2E+Zhou%2C+%E2%80%9CQwen%2Dvl%3A+A+frontier+large+vision%2Dlanguage+model+with) [our card](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/)
22. [117] mixtral. mixtral. mixtral-of-experts/ [118] D. Wang, N. Raman, M. Sibue, Z. Ma, P. Babkin, S. Kaur, Y. Pei, A. Nourbakhsh, and X. Liu, “Docllm: A layout-aware generative language model for multimodal document understanding,” 2023. [find](https://scholar.google.com/scholar?q=%5B117%5D+mixtral%2E+mixtral%2E+mixtral%2Dof%2Dexperts%2F+%5B118%5D+D%2E+Wang%2C+N%2E+Raman%2C+M%2E+Sibue%2C+Z%2E+Ma%2C+P%2E+Babkin%2C+S%2E+Kaur%2C+Y%2E+Pei%2C+A%2E+Nourbakhsh%2C+and+X%2E+Liu%2C+%E2%80%9CDocllm%3A+A+layout%2Daware+generative+language+model+for+multi)
23. [119] D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. K. Li, F. Luo, Y. Xiong, and W. Liang, “Deepseek-coder: When the large language model meets programming – the rise of code intelligence,” 2024. [find](https://scholar.google.com/scholar?q=%5B119%5D+D%2E+Guo%2C+Q%2E+Zhu%2C+D%2E+Yang%2C+Z%2E+Xie%2C+K%2E+Dong%2C+W%2E+Zhang%2C+G%2E+Chen%2C+X%2E+Bi%2C+Y%2E+Wu%2C+Y%2E+K%2E+Li%2C+F%2E+Luo%2C+Y%2E+Xiong%2C+and+W%2E+Liang%2C+%E2%80%9CDeepseek%2Dcoder%3A+When+the+large+language+model+meets+programming+%E2%80%93+the+rise+o)
24. [120] F. Wan, X. Huang, D. Cai, X. Quan, W. Bi, and S. Shi, “Knowledge fusion of large language models,” 2024. [find](https://scholar.google.com/scholar?q=%5B120%5D+F%2E+Wan%2C+X%2E+Huang%2C+D%2E+Cai%2C+X%2E+Quan%2C+W%2E+Bi%2C+and+S%2E+Shi%2C+%E2%80%9CKnowledge+fusion+of+large+language+models%2C%E2%80%9D+2024%2E)
25. [121] P. Zhang, G. Zeng, T. Wang, and W. Lu, “Tinyllama: An open-source small language model,” 2024. [find](https://scholar.google.com/scholar?q=%5B121%5D+P%2E+Zhang%2C+G%2E+Zeng%2C+T%2E+Wang%2C+and+W%2E+Lu%2C+%E2%80%9CTinyllama%3A+An+open%2Dsource+small+language+model%2C%E2%80%9D+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2024-tinyllama-open-source-small-language/)
26. [122] C. Wu, Y. Gan, Y. Ge, Z. Lu, J. Wang, Y. Feng, P. Luo, and Y. Shan, “Llama pro: Progressive llama with block expansion,” 2024. [find](https://scholar.google.com/scholar?q=%5B122%5D+C%2E+Wu%2C+Y%2E+Gan%2C+Y%2E+Ge%2C+Z%2E+Lu%2C+J%2E+Wang%2C+Y%2E+Feng%2C+P%2E+Luo%2C+and+Y%2E+Shan%2C+%E2%80%9CLlama+pro%3A+Progressive+llama+with+block+expansion%2C%E2%80%9D+2024%2E)
27. [123] X. Amatriain, A. Sankar, J. Bing, P. K. Bodigutla, T. J. Hazen, and M. Kazi, “Transformer models: an introduction and catalog,” 2023. [find](https://scholar.google.com/scholar?q=%5B123%5D+X%2E+Amatriain%2C+A%2E+Sankar%2C+J%2E+Bing%2C+P%2E+K%2E+Bodigutla%2C+T%2E+J%2E+Hazen%2C+and+M%2E+Kazi%2C+%E2%80%9CTransformer+models%3A+an+introduction+and+catalog%2C%E2%80%9D+2023%2E)
28. [124] G. Penedo, Q. Malartic, D. Hesslow, R. Cojocaru, A. Cappelli, H. Alobeidli, B. Pannier, E. Almazrouei, and J. Launay, “The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only,” arXiv preprint arXiv:2306.01116, 2023. [find](https://scholar.google.com/scholar?q=%5B124%5D+G%2E+Penedo%2C+Q%2E+Malartic%2C+D%2E+Hesslow%2C+R%2E+Cojocaru%2C+A%2E+Cappelli%2C+H%2E+Alobeidli%2C+B%2E+Pannier%2C+E%2E+Almazrouei%2C+and+J%2E+Launay%2C+%E2%80%9CThe+refinedweb+dataset+for+falcon+llm%3A+outperforming+curated+corpora+with+w)
29. [125] D. Hernandez, T. Brown, T. Conerly, N. DasSarma, D. Drain, S. El-Showk, N. Elhage, Z. Hatfield-Dodds, T. Henighan, T. Hume et al., “Scaling laws and interpretability of learning from repeated data,” arXiv preprint arXiv:2205.10487, 2022. [find](https://scholar.google.com/scholar?q=%5B125%5D+D%2E+Hernandez%2C+T%2E+Brown%2C+T%2E+Conerly%2C+N%2E+DasSarma%2C+D%2E+Drain%2C+S%2E+El%2DShowk%2C+N%2E+Elhage%2C+Z%2E+Hatfield%2DDodds%2C+T%2E+Henighan%2C+T%2E+Hume+et+al%2E%2C+%E2%80%9CScaling+laws+and+interpretability+of+learning+from+repeated+da)
30. P. Shaw, J. Uszkoreit, and A. Vaswani, “Self-attention with relative position representations,” arXiv preprint arXiv:1803.02155, 2018. [find](https://scholar.google.com/scholar?q=P%2E+Shaw%2C+J%2E+Uszkoreit%2C+and+A%2E+Vaswani%2C+%E2%80%9CSelf%2Dattention+with+relative+position+representations%2C%E2%80%9D+arXiv+preprint+arXiv%3A1803%2E02155%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/shaw-2018-self-attention-relative-position-representations/)
31. J. Su, Y. Lu, S. Pan, B. Wen, and Y. Liu, “Roformer: En-[Online]. Available: [Online]. [https://github.com/yandex/Available:](https://github.com/yandex/Available:) https://mistral.ai/news/[128] [129] [130] [131] [132] [133] [134] [135] [136] [137] [138] [139] [140] [141] [142] [143] [144] [145] [146] [147] [148] hanced transformer with ro… [↗](https://github.com/yandex/Available:)
32. IEEE, 2020, pp. 1–16. B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, H. Cao, X. Cheng, M. Chung, M. Grella, K. K. GV et al., “Rwkv: Reinventing rnns for the transformer era,” arXiv preprint arXiv:2305.13048, 2023. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W… [find](https://scholar.google.com/scholar?q=IEEE%2C+2020%2C+pp%2E+1%E2%80%9316%2E+B%2E+Peng%2C+E%2E+Alcaide%2C+Q%2E+Anthony%2C+A%2E+Albalak%2C+S%2E+Arcadinho%2C+H%2E+Cao%2C+X%2E+Cheng%2C+M%2E+Chung%2C+M%2E+Grella%2C+K%2E+K%2E+GV+et+al%2E%2C+%E2%80%9CRwkv%3A+Reinventing+rnns+for+the+transformer+era%2C%E2%80%9D+arXiv+preprin) [our card](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/)
33. [150] [151] [152] [153] [154] [155] [156] [157] [158] [159] [160] [161] [162] [163] [164] USA: Association for Computational Linguistics, Jul. 2002, pp. 311– 318. [Online]. Available: https://aclanthology.org/P02-1040 B. Dhingra, M. Faruqui, A. Parikh, M.-W. Chang, D. Das, and W. Cohen, “Handling di… [↗](https://aclanthology.org/P02-1040) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
34. [167] [168] [169] [170] [171] [172] [173] [174] [175] [176] [177] [178] [179] [180] [181] [182] [183] [184] H. Wang, “Retrieval-augmented generation for large language models: A survey,” arXiv preprint arXiv:2312.10997, 2023. A. W. Services. (Year of publication, e.g., 2023) Question answering using… [find](https://scholar.google.com/scholar?q=%5B167%5D+%5B168%5D+%5B169%5D+%5B170%5D+%5B171%5D+%5B172%5D+%5B173%5D+%5B174%5D+%5B175%5D+%5B176%5D+%5B177%5D+%5B178%5D+%5B179%5D+%5B180%5D+%5B181%5D+%5B182%5D+%5B183%5D+%5B184%5D+H%2E+Wang%2C+%E2%80%9CRetrieval%2Daugmented+generation+for+large+language+models%3A+A+survey%2C%E2%80%9D+arXiv+preprin) [our card](https://md.research.elamur.ai/free/en/papers/gao-2023-retrieval-augmented-generation-large-language/)
35. [186] G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy, “RACE: Large-scale ReAding comprehension dataset from examinations,” in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, M. Palmer, R. Hwa, and S. Riedel, Eds. Copenhagen, Denmark: Association for Computationa… [find](https://scholar.google.com/scholar?q=%5B186%5D+G%2E+Lai%2C+Q%2E+Xie%2C+H%2E+Liu%2C+Y%2E+Yang%2C+and+E%2E+Hovy%2C+%E2%80%9CRACE%3A+Large%2Dscale+ReAding+comprehension+dataset+from+examinations%2C%E2%80%9D+in+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Language+) [our card](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/)
36. [187] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “SQuAD: 100,000+ questions for machine comprehension of text,” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, J. Su, K. Duh, and X. Carreras, Eds. Austin, Texas: Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=%5B187%5D+P%2E+Rajpurkar%2C+J%2E+Zhang%2C+K%2E+Lopyrev%2C+and+P%2E+Liang%2C+%E2%80%9CSQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2C%E2%80%9D+in+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Pr) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
37. [188] C. Clark, K. Lee, M. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova, “Boolq: Exploring the surprising difficulty of natural yes/no questions,” CoRR, vol. abs/1905.10044, 2019. [Online]. Available: http://arxiv.org/abs/1905.10044 [↗](http://arxiv.org/abs/1905.10044) [our card](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/)
38. [189] D. Khashabi, S. Chaturvedi, M. Roth, S. Upadhyay, and D. Roth, “Looking beyond the surface:a challenge set for reading comprehension over multiple sentences,” in Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL), 2018. [find](https://scholar.google.com/scholar?q=%5B189%5D+D%2E+Khashabi%2C+S%2E+Chaturvedi%2C+M%2E+Roth%2C+S%2E+Upadhyay%2C+and+D%2E+Roth%2C+%E2%80%9CLooking+beyond+the+surface%3Aa+challenge+set+for+reading+comprehension+over+multiple+sentences%2C%E2%80%9D+in+Proceedings+of+North+American+Ch) [our card](https://md.research.elamur.ai/free/en/papers/khashabi-2018-looking-beyond-surface-challenge-set/)
39. [190] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, “Training verifiers to solve math word problems,” CoRR, vol. abs/2110.14168, 2021. [Online]. Available: https://arxiv.org/abs/2110.14168 [↗](https://arxiv.org/abs/2110.14168) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
40. [191] D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt, “Measuring mathematical problem solving with the MATH dataset,” CoRR, vol. abs/2103.03874, 2021. [Online]. Available: https://arxiv.org/abs/2103.03874 [↗](https://arxiv.org/abs/2103.03874) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/)
41. [192] R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi, “Hellaswag: Can a machine really finish your sentence?” 2019. [find](https://scholar.google.com/scholar?q=%5B192%5D+R%2E+Zellers%2C+A%2E+Holtzman%2C+Y%2E+Bisk%2C+A%2E+Farhadi%2C+and+Y%2E+Choi%2C+%E2%80%9CHellaswag%3A+Can+a+machine+really+finish+your+sentence%3F%E2%80%9D+2019%2E)
42. [193] P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try arc, the AI2 reasoning challenge,” CoRR, vol. abs/1803.05457, 2018. [Online]. Available: http://arxiv.org/abs/1803.05457 [↗](http://arxiv.org/abs/1803.05457)
43. [194] Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi, “PIQA: reasoning about physical commonsense in natural language,” CoRR, vol. abs/1911.11641, 2019. [Online]. Available: http://arxiv.org/abs/ 1911.11641 [↗](http://arxiv.org/abs/) [our card](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/)
44. [195] M. Sap, H. Rashkin, D. Chen, R. L. Bras, and Y. Choi, “Socialiqa: Commonsense reasoning about social interactions,” CoRR, vol. abs/1904.09728, 2019. [Online]. Available: http://arxiv.org/abs/1904. 09728 [↗](http://arxiv.org/abs/1904) [our card](https://md.research.elamur.ai/free/en/papers/sap-2019-socialiqa-commonsense-reasoning-about-social/)
45. [196] T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal, “Can a suit of armor conduct electricity? A new dataset for open book question answering,” CoRR, vol. abs/1809.02789, 2018. [Online]. Available: http://arxiv.org/abs/1809.02789 [↗](http://arxiv.org/abs/1809.02789)
46. [197] S. Lin, J. Hilton, and O. Evans, “Truthfulqa: Measuring how models mimic human falsehoods,” arXiv preprint arXiv:2109.07958, 2021. [find](https://scholar.google.com/scholar?q=%5B197%5D+S%2E+Lin%2C+J%2E+Hilton%2C+and+O%2E+Evans%2C+%E2%80%9CTruthfulqa%3A+Measuring+how+models+mimic+human+falsehoods%2C%E2%80%9D+arXiv+preprint+arXiv%3A2109%2E07958%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/)
47. [198] Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning, “Hotpotqa: A dataset for diverse, explainable multi-hop question answering,” CoRR, vol. abs/1809.09600, 2018. [Online]. Available: http://arxiv.org/abs/1809.09600 [↗](http://arxiv.org/abs/1809.09600) [our card](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/)
48. [199] Y. Zhuang, Y. Yu, K. Wang, H. Sun, and C. Zhang, “Toolqa: A dataset for llm question answering with external tools,” arXiv preprint arXiv:2306.13304, 2023. [find](https://scholar.google.com/scholar?q=%5B199%5D+Y%2E+Zhuang%2C+Y%2E+Yu%2C+K%2E+Wang%2C+H%2E+Sun%2C+and+C%2E+Zhang%2C+%E2%80%9CToolqa%3A+A+dataset+for+llm+question+answering+with+external+tools%2C%E2%80%9D+arXiv+preprint+arXiv%3A2306%2E13304%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhuang-2023-toolqa-dataset-llm-question-answering/)
49. [200] D. Chen, J. Bolton, and C. D. Manning, “A thorough examination of the cnn/daily mail reading comprehension task,” in Association for Computational Linguistics (ACL), 2016. [find](https://scholar.google.com/scholar?q=%5B200%5D+D%2E+Chen%2C+J%2E+Bolton%2C+and+C%2E+D%2E+Manning%2C+%E2%80%9CA+thorough+examination+of+the+cnn%2Fdaily+mail+reading+comprehension+task%2C%E2%80%9D+in+Association+for+Computational+Linguistics+%28ACL%29%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2016-thorough-examination-cnn-daily-mail/)
50. [201] R. Nallapati, B. Zhou, C. Gulcehre, B. Xiang et al., “Abstractive text summarization using sequence-to-sequence rnns and beyond,” arXiv preprint arXiv:1602.06023, 2016. [find](https://scholar.google.com/scholar?q=%5B201%5D+R%2E+Nallapati%2C+B%2E+Zhou%2C+C%2E+Gulcehre%2C+B%2E+Xiang+et+al%2E%2C+%E2%80%9CAbstractive+text+summarization+using+sequence%2Dto%2Dsequence+rnns+and+beyond%2C%E2%80%9D+arXiv+preprint+arXiv%3A1602%2E06023%2C+2016%2E)
51. Y. Bai and D. Z. Wang, “More than reading comprehension: A survey [203] [204] [205] [206] [207] [208] [209] [210] [211] [212] [213] [214] [215] [216] [217] [218] [219] [220] [221] [222] [223] [224] on datasets and metrics of textual question answering,” arXiv preprint arXiv:2109.12264, 2021. H.-Y. H… [find](https://scholar.google.com/scholar?q=Y%2E+Bai+and+D%2E+Z%2E+Wang%2C+%E2%80%9CMore+than+reading+comprehension%3A+A+survey+%5B203%5D+%5B204%5D+%5B205%5D+%5B206%5D+%5B207%5D+%5B208%5D+%5B209%5D+%5B210%5D+%5B211%5D+%5B212%5D+%5B213%5D+%5B214%5D+%5B215%5D+%5B216%5D+%5B217%5D+%5B218%5D+%5B219%5D+%5B220%5D+%5B221%5D+%5B222%5D+%5B223%5D+%5B224%5D+on) [our card](https://md.research.elamur.ai/free/en/papers/li-2023-halueval-large-scale-hallucination-evaluation/)