---
title: "Large language models — methods, history and key papers"
description: "A large language model (LLM) is a neural network, usually a transformer with billions of parameters, trained on very large text collections to…"
canonical: "https://research.elamur.ai/free/en/concepts/large-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/concepts/large-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [concepts](https://md.research.elamur.ai/free/en/concepts/) / Large language models

# 🏔️ Large language models

Field · 43 key papers · [all concepts](https://md.research.elamur.ai/free/en/concepts/)

Part of: [🗨️ Natural language processing](https://md.research.elamur.ai/free/en/concepts/natural-language-processing/) · [📎 Retrieval-augmented generation](https://md.research.elamur.ai/free/en/concepts/retrieval-augmented-generation/) · [💭 Chatbots](https://md.research.elamur.ai/free/en/concepts/chatbot/) · [🦾 Artificial intelligence](https://md.research.elamur.ai/free/en/concepts/artificial-intelligence/)

A large language model (LLM) is a neural network, usually a transformer with billions of parameters, trained on very large text collections to predict the next token. That single objective lets it generate text, answer questions, write code and follow instructions given in plain language.

Also: LLM, LLMs, large-scale language model

About 16 minutes to read · every paper named here has its own page in this library

**On this page**

1. [What it is](#a-what-it-is)
2. [How it works](#a-how-it-works)
3. [History](#a-history)
4. [Methods](#a-methods)
5. [Datasets and benchmarks](#a-datasets-and-benchmarks)
6. [People and institutions](#a-people-and-institutions)
7. [Open questions](#a-open-questions)
8. [The library in figures](#a-in-figures)

## What it is

A [language model](https://md.research.elamur.ai/free/en/concepts/language-model/) assigns probabilities to sequences of words. Given the beginning of a sentence, it estimates which word, or piece of a word, is likely to come next. A large language model applies the same idea at enormous scale. It has billions or hundreds of billions of adjustable parameters. It trains on a large share of the public written web, plus books and code. The resulting system can be prompted in ordinary language to translate, summarize, explain, argue, write programs or hold a conversation. Most users meet one through a [chatbot](https://md.research.elamur.ai/free/en/concepts/chatbot/) interface.

"Large" has no fixed threshold, and it has moved over time. GPT-3 had 175 billion parameters, roughly ten times any earlier dense language model, while GPT-2 had 1.5 billion ([Brown et al. 2020](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)). Some overviews limit their scope to pretrained models of about 10 billion parameters and above ([Naveed et al. 2023](https://md.research.elamur.ai/free/en/papers/naveed-2023-comprehensive-overview-large-language-models/)). A more useful criterion looks at behaviour and use. One major survey separates LLMs from earlier pretrained models on three grounds ([Zhao et al. 2023](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)):

- they show abilities such as in-context learning;
- people use them mainly through prompting rather than task-specific fine-tuning;
- building them blurs the line between research and engineering.

That survey places LLMs as the fourth stage of language modelling. Statistical n-gram models came first, then neural models, then pretrained models such as ELMo and BERT ([Zhao et al. 2023](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)).

The Stanford report on "foundation models" gave the field much of its vocabulary ([Bommasani et al. 2021](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)). In this view, one model is pretrained with self-supervision on broad data and then adapted to many tasks. The report stresses two consequences. The first is emergence: capabilities such as in-context learning arise from scale rather than explicit design. A narrower, threshold-based sense of the word is discussed under History. The second is homogenization: many applications rest on a few base models, so their strengths and their flaws spread together ([Bommasani et al. 2021](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)). For [natural language processing](https://md.research.elamur.ai/free/en/concepts/natural-language-processing/) and deep learning, the practical result is that one pretrained network now underlies tasks that once each needed a separately trained system ([Bommasani et al. 2021](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)).

The shelves its papers stand on Artificial Intelligence: 1458 of the 1,524 papers here.

**The numbers**

| Shelf | Papers |
| --- | --- |
| [Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) | 1,458 papers |
| [Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) | 1,369 papers |
| [Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) | 380 papers |
| [Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) | 153 papers |
| [Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) | 97 papers |
| [Health Informatics](https://md.research.elamur.ai/free/en/shelves/medicine/health-informatics/) | 38 papers |
| [Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) | 34 papers |
| [Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) | 34 papers |
| [Political Science and International Relations](https://md.research.elamur.ai/free/en/shelves/social-sciences/political-science-and-international-relations/) | 25 papers |
| [Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) | 22 papers |
| [Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) | 19 papers |
| [Social Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/social-psychology/) | 16 papers |

## How it works

### Next-token prediction

Text is first cut into tokens by [subword tokenization](https://md.research.elamur.ai/free/en/concepts/subword-tokenization/). Common tokenizers include byte-pair encoding (BPE), WordPiece and unigram models ([Naveed et al. 2023](https://md.research.elamur.ai/free/en/papers/naveed-2023-comprehensive-overview-large-language-models/)). LLaMA, for example, uses BPE as implemented in SentencePiece. It splits numbers into single digits and falls back to bytes for unknown characters ([Touvron et al. 2023](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/)). The model then learns to predict each token from the tokens before it. The standard training loss is the average negative log-likelihood of the corpus:

```
L(θ) = − (1/T) Σ_{t=1..T} log p_θ(x_t | x_1, …, x_{t−1})
```

Here x_1 … x_T is a training sequence of T tokens, and θ stands for all the parameters of the network. The term p_θ(x_t | x_1, …, x_{t−1}) is the probability the model assigns to the true next token given everything before it. Minimising L(θ) is the same as minimising cross-entropy between the model's predictions and the data. The scaling-law literature uses this same quantity to measure quality ([Kaplan et al. 2020](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/)). Because the labels come from the text itself, no human annotation is needed. This self-supervision is what makes web-scale training possible ([Bommasani et al. 2021](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)).

At generation time, the model samples one token, appends it to the input and repeats. This autoregressive loop is simple. Bubeck and colleagues argue that it also causes some weaknesses, such as poor planning, because the model cannot revise what it has already written ([Bubeck et al. 2023](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/)).

### The decoder-only transformer

Almost all current LLMs are built on the [transformer](https://md.research.elamur.ai/free/en/concepts/transformer/) of Vaswani et al. (2017). Its core operation, self-attention, is usually written as:

```
Attention(Q, K, V) = softmax(Q Kᵀ / √d_k) V
```

Q, K and V are matrices of query, key and value vectors, computed from the token representations by learned projections. d_k is the dimension of the keys, and softmax turns each row of scores into weights that sum to one. In a decoder-only model, a causal mask stops each position from attending to later positions, so the network can only use the past. Surveys group pretrained models into three families ([Pan et al. 2023](https://md.research.elamur.ai/free/en/papers/pan-2023-unifying-large-language-models-knowledge/); [Chua; & Li 2024](https://md.research.elamur.ai/free/en/papers/fan-2024-survey-rag-meeting-llms-towards/)):

- encoder-only, such as BERT;
- encoder-decoder, such as T5;
- decoder-only, such as GPT, PaLM and LLaMA.

The decoder-only family dominates generative LLMs. PaLM is a dense decoder-only transformer ([Chowdhery et al. 2022](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)). LLaMA adds now-standard refinements ([Touvron et al. 2023](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/)):

- RMSNorm, which rescales activations by their root mean square, applied before each sub-layer;
- the SwiGLU activation, a gated form of the feed-forward nonlinearity;
- rotary positional embeddings.

### Pretraining on web-scale text

Pretraining corpora mix filtered web crawls with higher-quality sources. About two-thirds of LLaMA's corpus is CommonCrawl, cleaned with CCNet, a pipeline that deduplicates web pages and filters them by language and quality. It adds C4, GitHub, Wikipedia, books, arXiv and Stack Exchange, for roughly 1.4 trillion tokens ([Touvron et al. 2023](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/)). PaLM trained on 780 billion tokens ([Chowdhery et al. 2022](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)). GPT-4 used public internet data and licensed third-party data ([OpenAI et al. 2023](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)).

How to spend a training budget is governed by [scaling laws](https://md.research.elamur.ai/free/en/concepts/scaling-laws/). Kaplan and colleagues found that test loss falls as a power law in each of three quantities, as long as the other two are not the bottleneck ([Kaplan et al. 2020](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/)):

```
L(N) ∝ N^(−α_N),   L(D) ∝ D^(−α_D),   L(C) ∝ C^(−α_C)
```

N is the number of non-embedding parameters, D the number of training tokens and C the training compute. The α are fitted exponents. In their experiments they were about 0.076 for parameters and 0.095 for data. For compute allocated optimally, the exponent was about 0.050 ([Kaplan et al. 2020](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/)). Network shape, meaning depth, width and number of heads, mattered little. Their compute rule favoured very large models trained for relatively few tokens ([Kaplan et al. 2020](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/)).

DeepMind's Chinchilla study revised that rule. Using over 400 training runs, it concluded that parameters and tokens should grow in equal proportion ([Hoffmann et al. 2022](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)). Chinchilla has 70 billion parameters and was trained on 1.4 trillion tokens. It used the same compute as the 280-billion-parameter Gopher and beat it ([Hoffmann et al. 2022](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)). LLaMA went further. It deliberately trained past the Chinchilla optimum, because inference cost dominates once a model is deployed ([Touvron et al. 2023](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/)).

### From base model to assistant

A pretrained model continues text. It does not, by default, follow requests. Two further stages turn it into an assistant.

The first is supervised instruction tuning, also called supervised fine-tuning (SFT). In InstructGPT, 40 hired labelers wrote demonstration answers to real prompts, and GPT-3 was fine-tuned on them ([Ouyang et al. 2022](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)).

The second is reinforcement learning from human feedback (RLHF). Labelers rank several model outputs, and a reward model learns those preferences. The policy is then optimised against the reward model with proximal policy optimisation (PPO), a standard reinforcement-learning algorithm ([Ouyang et al. 2022](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)). Raters preferred the outputs of a 1.3-billion-parameter InstructGPT over those of the 175-billion-parameter GPT-3 ([Ouyang et al. 2022](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)). Bai and colleagues applied the same recipe to dialogue assistants meant to be both helpful and harmless. They found that, for large models, alignment training cost little or nothing on standard benchmarks (Bai et al. 2022).

More recently, reinforcement learning with verifiable rewards has been used to teach reasoning. DeepSeek-R1-Zero was trained with no supervised warm-up. Its algorithm was group relative policy optimisation (GRPO), which rates each sampled answer against the others drawn for the same question instead of relying on a critic model. Rule-based rewards checked the final answers. The journal version of the paper tracks pass@1 on the 2024 American Invitational Mathematics Examination (AIME) during training. Pass@1 is the share of problems solved by a single sampled answer. It rose from 15.6% to 77.9% ([DeepSeek-AI et al. 2025](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/)).

## History

### Pretraining before "large"

The pretrain-then-adapt paradigm took hold in 2018. Radford et al. (2018) introduced GPT, a generatively pretrained transformer. Devlin et al. (2018) introduced BERT, trained with masked language modelling. The foundation-models report dates the current era to about 2018–2019, starting with BERT. It contrasts BERT's masked objective with GPT's next-word prediction ([Bommasani et al. 2021](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)). Three enablers came together at that point ([Bommasani et al. 2021](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)):

- GPU throughput and memory grew roughly tenfold in four years;
- transformers parallelised well;
- self-supervised objectives could exploit unlabelled text.

### GPT-3 and in-context learning

GPT-3 marked the shift to prompting. The model was evaluated with no weight updates. The task description and a few examples were simply placed in the prompt, a mechanism its authors named in-context learning ([Brown et al. 2020](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)). Few-shot performance grew faster with model size than zero-shot performance. In some settings it approached fine-tuned systems ([Brown et al. 2020](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)). This made [few-shot learning](https://md.research.elamur.ai/free/en/concepts/few-shot-learning/) through prompts a central research object.

### PaLM, Chinchilla and emergent abilities

In 2022 Google trained PaLM, a 540-billion-parameter model, on 6,144 TPU v4 chips. It reached 46.2% model FLOPs utilisation, the share of the chips' peak arithmetic rate spent on the model's own computation ([Chowdhery et al. 2022](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)). It set new few-shot results on 28 of 29 widely used English benchmarks. On some BIG-bench tasks, quality jumped only at the largest size ([Chowdhery et al. 2022](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)).

Such jumps were collected under the label "emergent abilities". These are skills that small models lack and larger models suddenly show, so they cannot be extrapolated from small-scale trends ([Wei et al. 2022](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/)). For example, GPT-3's accuracy on multi-digit arithmetic stayed near zero until about 2·10²² training FLOPs, around 13 billion parameters ([Wei et al. 2022](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/)). Chinchilla's revision of the scaling rule, covered under pretraining above, came out the same year ([Hoffmann et al. 2022](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)).

### ChatGPT and GPT-4

ChatGPT, released at the end of 2022, made the instruction-tuned and RLHF-trained assistant a mass-market product. It descends from the InstructGPT line of work ([Ouyang et al. 2022](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)). Research output followed. After ChatGPT's release, arXiv papers mentioning "large language model" in the title or abstract went from 0.40 to 8.58 per day ([Zhao et al. 2023](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)). Early studies showed ChatGPT near the roughly 60% pass mark on the US medical licensing exam ([Kung et al. 2023](https://md.research.elamur.ai/free/en/papers/kung-2023-performance-chatgpt-usmle-potential-ai/)). Other work showed it outperforming crowd-workers on text annotation ([Gilardi et al. 2023](https://md.research.elamur.ai/free/en/papers/gilardi-2023-chatgpt-outperforms-crowd-workers-text/)).

GPT-4 (2023) accepts images as well as text. Its report withholds architecture, size and compute details ([OpenAI et al. 2023](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)). It scored about the 90th percentile on a simulated bar exam. Its final loss was forecast in advance from runs using up to 10,000 times less compute ([OpenAI et al. 2023](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)). This does not contradict emergence. Aggregate loss can fall smoothly and predictably even when particular abilities appear abruptly ([OpenAI et al. 2023](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/); [Wei et al. 2022](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/)). A Microsoft team with early access argued that GPT-4 showed "sparks" of general intelligence. Their study was purely observational ([Bubeck et al. 2023](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/)).

### Open models

Meta's LLaMA (2023) showed that competitive models could be trained only on public data. LLaMA-13B outperformed GPT-3 on most benchmarks, and the weights were shared with researchers ([Touvron et al. 2023](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/)). Llama 2 added fine-tuned chat models (Touvron et al. 2023). The Llama 3 family followed (Grattafiori et al. 2024), along with specialised offshoots such as Code Llama ([Rozière et al. 2024](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)). Openly released reasoning models such as DeepSeek-R1, together with smaller models distilled from it, extended the trend ([DeepSeek-AI et al. 2025](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/)).

| Model | Year | Parameters | Note |
| --- | --- | --- | --- |
| GPT-3 | 2020 | 175B | in-context learning ([Brown et al. 2020](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)) |
| InstructGPT | 2022 | 1.3B, 6B, 175B | supervised fine-tuning plus RLHF ([Ouyang et al. 2022](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)) |
| PaLM | 2022 | 540B | Pathways training ([Chowdhery et al. 2022](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)) |
| Chinchilla | 2022 | 70B | compute-optimal data ([Hoffmann et al. 2022](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)) |
| LLaMA | 2023 | 7B–65B | public data, weights for researchers ([Touvron et al. 2023](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/)) |
| GPT-4 | 2023 | undisclosed | multimodal input ([OpenAI et al. 2023](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)) |

Landmark papers on this subject: the 12 most cited, in time The 12 most cited papers on this subject span 2021–2025; the dot grows with the citations.

**The numbers**

| Paper | Citations |
| --- | --- |
| [2021 — LoRA: Low-Rank Adaptation of Large Language Models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/) | 8,473 |
| [2021 — Training Verifiers to Solve Math Word Problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/) | 3,680 |
| [2022 — Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/) | 11,308 |
| [2022 — Large Language Models Encode Clinical Knowledge](https://md.research.elamur.ai/free/en/papers/singhal-2022-large-language-models-encode-clinical/) | 4,477 |
| [2022 — Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/) | 3,795 |
| [2023 — GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/) | 12,732 |
| [2023 — LLaMA: Open and Efficient Foundation Language Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/) | 9,472 |
| [2023 — ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education](https://md.research.elamur.ai/free/en/papers/kasneci-2023-chatgpt-good-opportunities-challenges-large/) | 7,436 |
| [2023 — Efficient Memory Management for Large Language Model Serving with PagedAttention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/) | 4,453 |
| [2023 — Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models](https://md.research.elamur.ai/free/en/papers/kung-2023-performance-chatgpt-usmle-potential-ai/) | 4,168 |
| [2023 — A Survey on Evaluation of Large Language Models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/) | 3,641 |
| [2025 — DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/) | 4,593 |

Papers on this subject by year 1,524 dated papers from 2007 to 2026; the busiest year is 2023 with 483, and 100% of them appeared in 2018 or later.

**The numbers**

| Year | Papers |
| --- | --- |
| 2026 | 370 |
| 2025 | 189 |
| 2024 | 394 |
| 2023 | 483 |
| 2022 | 68 |
| 2021 | 16 |
| 2020 | 2 |
| 2019 | 1 |
| 2007 | 1 |

## Methods

### Prompting and reasoning

Prompt engineering became a method in its own right. Reynolds and McDonell argued that few-shot examples mostly help GPT-3 locate a task it already knows rather than learn a new one. In their translation case study, well-written zero-shot prompts matched or beat few-shot prompts ([Reynolds & McDonell 2021](https://md.research.elamur.ai/free/en/papers/reynolds-2021-prompt-programming-large-language-models/)).

Chain-of-thought prompting adds worked intermediate steps to each example. With PaLM 540B it raised accuracy on GSM8K, a set of grade-school maths word problems, from about 18% to about 57%. The benefit appeared only above roughly 100 billion parameters ([Wei et al. 2022](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)).

Kojima and colleagues showed that a single trigger, "Let's think step by step", works with no examples at all. It raised InstructGPT's accuracy on MultiArith, a set of multi-step arithmetic word problems, from 17.7% to 78.7% ([Kojima et al. 2022](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/)).

Least-to-most prompting breaks a problem into easier subquestions and answers them in sequence. It generalised to problems harder than its examples. On every split of SCAN, a benchmark that maps simple commands to action sequences, it reached at least 99% ([al. 2022](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/)).

Tree of Thoughts extends the idea to deliberate search over alternative reasoning paths (Yao et al. 2023). These lines of work sit within the wider study of [reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/).

### Efficient adaptation and serving

Full fine-tuning of a 175-billion-parameter model is costly. LoRA freezes the pretrained weights and learns a low-rank update in selected layers. For GPT-3 it cut trainable parameters by about 10,000 times and GPU memory by about three times, with no added inference latency ([Hu et al. 2021](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/)). QLoRA combines this kind of adapter with a quantised base model so that fine-tuning fits on far smaller hardware (Dettmers et al. 2023). Overviews also list pruning, quantization, distillation and context-length interpolation as efficiency tools ([Naveed et al. 2023](https://md.research.elamur.ai/free/en/papers/naveed-2023-comprehensive-overview-large-language-models/)). More on this is under [efficient models](https://md.research.elamur.ai/free/en/topics/efficient-models/).

Serving is its own bottleneck. Each request's key-value cache grows as tokens are generated, and its final length is not known in advance. Existing systems therefore reserved space for the maximum possible length. Only 20.4–38.2% of their cache memory ended up storing actual token states ([Kwon et al. 2023](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/)). PagedAttention borrows paged virtual memory from operating systems and allocates fixed-size blocks only when needed. The resulting vLLM system delivers 2–4 times the throughput of earlier systems at the same latency ([Kwon et al. 2023](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/)). Hardware concerns are covered further under [GPU and parallel computing](https://md.research.elamur.ai/free/en/topics/gpu-parallel-computing/).

### Retrieval, tools and agents

[Retrieval-augmented generation](https://md.research.elamur.ai/free/en/concepts/retrieval-augmented-generation/) lets a model consult an external document store before answering. It addresses stale knowledge and untraceable claims. A survey sorts these systems into Naive, Advanced and Modular RAG ([Gao et al. 2023](https://md.research.elamur.ai/free/en/papers/gao-2023-retrieval-augmented-generation-large-language/)). Another groups retrieval-augmented LLMs by architecture, training strategy and application ([Chua; & Li 2024](https://md.research.elamur.ai/free/en/papers/fan-2024-survey-rag-meeting-llms-towards/)). Combining LLMs with a [knowledge graph](https://md.research.elamur.ai/free/en/concepts/knowledge-graph/) is a related programme, with knowledge flowing in either direction ([Pan et al. 2023](https://md.research.elamur.ai/free/en/papers/pan-2023-unifying-large-language-models-knowledge/)).

LLMs also serve as the controller of autonomous agents. Surveys describe such agents in terms of modules for profile, memory, planning and action ([Wang et al. 2023](https://md.research.elamur.ai/free/en/papers/wang-2023-survey-large-language-model-based/)), or brain, perception and action ([Xi et al. 2023](https://md.research.elamur.ai/free/en/papers/xi-2023-rise-potential-large-language/)). One concrete case is Coscientist, a GPT-4-based agent that planned and ran chemistry experiments through web search, code execution and laboratory hardware ([Boiko et al. 2023](https://md.research.elamur.ai/free/en/papers/boiko-2023-autonomous-chemical-research-large-language/)).

### Code and multimodality

Training on source code produced Codex. It was fine-tuned on public GitHub code, starting from GPT models of up to 12 billion parameters ([Chen et al. 2021](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)). Code Llama continued Llama 2 pretraining on 500 billion tokens of code-heavy data ([Rozière et al. 2024](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)). See also [code and programming](https://md.research.elamur.ai/free/en/topics/code-and-programming/).

Among [multimodal models](https://md.research.elamur.ai/free/en/topics/multimodal/), several systems connect a vision encoder to a language model:

- BLIP-2 links a frozen image encoder to a frozen LLM through a small Querying Transformer (Q-Former). Its learned query vectors pick out the visual features relevant to text ([Li et al. 2023](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/)).
- InstructBLIP feeds the instruction itself into that connector ([Dai et al. 2023](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/)).
- LLaVA trains on image-grounded instruction data written by a text-only GPT-4 ([Liu et al. 2023](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/)). Its successor LLaVA-1.5 showed that a simple MLP connector, a small multi-layer perceptron, suffices ([Liu et al. 2023](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/)).
- MiniGPT-4 trains only a single linear layer between BLIP-2's visual stack and Vicuna ([Zhu et al. 2023](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/)).

A survey maps these designs as encoder, connector and LLM ([Yin et al. 2023](https://md.research.elamur.ai/free/en/papers/yin-2023-survey-multimodal-large-language-models/)).

## Datasets and benchmarks

[Evaluation and benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) for LLMs have become a field of their own. A survey organises the field around three questions: what abilities to test, on which benchmarks, and with which protocols. It finds reasoning and robustness to perturbed inputs to be recurring weak points ([Chang et al. 2023](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/)). The main benchmark types are:

| Type | Examples | What is scored |
| --- | --- | --- |
| Knowledge and exams | MMLU (57 subjects), professional exams, MultiMedQA | multiple-choice accuracy ([OpenAI et al. 2023](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/); [Singhal et al. 2022](https://md.research.elamur.ai/free/en/papers/singhal-2022-large-language-models-encode-clinical/)) |
| Reasoning | GSM8K, MultiArith, BIG-bench tasks | final-answer accuracy ([Wei et al. 2022](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/); [Kojima et al. 2022](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/)) |
| Code | HumanEval, MBPP | unit tests passed (pass@k) ([Chen et al. 2021](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/); [Austin et al. 2021](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/)) |
| Open-ended chat | MT-Bench, Chatbot Arena | judgements by humans or a strong LLM (Zheng et al. 2023) |

Code evaluation shows why the scoring method matters. Text-overlap metrics such as [BLEU](https://md.research.elamur.ai/free/en/concepts/bleu/) often gave wrong programs higher scores than correct ones. HumanEval instead runs unit tests on 164 hand-written Python problems. Codex-12B solved 28.8% of them with a single sample ([Chen et al. 2021](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)). MBPP contributed 974 entry-level programming tasks ([Austin et al. 2021](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/)).

In medicine, MultiMedQA combined existing question-answering sets with a new collection of consumer health queries. Flan-PaLM reached 67.6% on MedQA, but physicians found serious problems with its long-form answers ([Singhal et al. 2022](https://md.research.elamur.ai/free/en/papers/singhal-2022-large-language-models-encode-clinical/)).

Two trends stand out. First, models are increasingly used as judges. G-Eval scores generated text with GPT-4 (Liu et al. 2023), and MT-Bench and Chatbot Arena study when an LLM judge agrees with people (Zheng et al. 2023). Second, test-set contamination is a constant worry:

- GPT-3's authors flagged it ([Brown et al. 2020](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/));
- GPT-4's report removed contaminated exam questions ([OpenAI et al. 2023](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/));
- the evaluation survey calls for dynamic tests that resist leakage ([Chang et al. 2023](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/)).

Where the papers on this subject were published Language Resources and Evaluation Conference (LREC) carries the most, 160 of 1524.

**The numbers**

| Venue | Papers |
| --- | --- |
| [Language Resources and Evaluation Conference (LREC)](https://md.research.elamur.ai/free/en/venues/lrec/) | 160 papers |
| [Findings of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/findings/) | 122 papers |
| [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/) | 115 papers |
| [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) | 83 papers |
| [Conference on Neural Information Processing Systems (NeurIPS)](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) | 73 papers |
| [Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)](https://md.research.elamur.ai/free/en/venues/naacl/) | 22 papers |
| [Annual Conference of the European Association for Machine Translation (EAMT)](https://md.research.elamur.ai/free/en/venues/eamt/) | 20 papers |
| [Journal of Chinese Information Processing (中文信息学报)](https://md.research.elamur.ai/free/en/venues/journal-of-chinese-information-processing/) | 20 papers |
| [Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)](https://md.research.elamur.ai/free/en/venues/sigdial/) | 17 papers |
| [Conference of the European Chapter of the Association for Computational Linguistics (EACL)](https://md.research.elamur.ai/free/en/venues/eacl/) | 17 papers |

Citations of the papers on this subject, by when they were published Papers published in 2020–2024 gather 94% of the 362,297 citations of the papers on this subject.

**The numbers**

| Published | Papers and their citations |
| --- | --- |
| 2025–2029 | 559 papers, 20,555 citations |
| 2020–2024 | 963 papers, 340,343 citations |
| 2015–2019 | 1 papers, 693 citations |
| 2005–2009 | 1 papers, 706 citations |

## People and institutions

A handful of research programmes shaped the methods above. [Jason Wei](https://md.research.elamur.ai/free/en/authors/jason-wei/) led the work on chain-of-thought prompting ([Wei et al. 2022](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)) and on emergent abilities ([Wei et al. 2022](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/)). He also co-authored the clinical-knowledge study ([Singhal et al. 2022](https://md.research.elamur.ai/free/en/papers/singhal-2022-large-language-models-encode-clinical/)). [Denny Zhou](https://md.research.elamur.ai/free/en/authors/denny-zhou/), at Google, co-authored chain-of-thought and emergence and led least-to-most prompting ([al. 2022](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/)). [Quoc Le](https://md.research.elamur.ai/free/en/authors/quoc-v-le/) co-authored chain-of-thought and the study of program synthesis with large models ([Austin et al. 2021](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/)). [Colin Raffel](https://md.research.elamur.ai/free/en/authors/colin-raffel/), at the University of Toronto, the Vector Institute and Hugging Face, co-authored the emergent-abilities paper ([Wei et al. 2022](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/)).

Efficiency and open infrastructure have their own figures. [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/), at the University of Washington and Meta, co-authored QLoRA (Dettmers et al. 2023). [Ion Stoica](https://md.research.elamur.ai/free/en/authors/ion-stoica/), at the University of California, Berkeley, belongs to the group behind PagedAttention and vLLM ([Kwon et al. 2023](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/)). The same group produced the Chatbot Arena work (Zheng et al. 2023).

Surveys have organised the field's knowledge. [Yue Zhang](https://md.research.elamur.ai/free/en/authors/yue-zhang/), at Westlake University, co-authored the survey on evaluating LLMs ([Chang et al. 2023](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/)). [Ji-Rong Wen](https://md.research.elamur.ai/free/en/authors/ji-rong-wen/), at Renmin University of China, belongs to the group behind a widely used LLM survey ([Zhao et al. 2023](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)).

Authors with the most papers on this subject Jason Wei leads with 14 papers; the 12 authors shown wrote 142 of the papers on this subject between them, counting each paper once per author.

**The numbers**

| Author | Papers here · their citations |
| --- | --- |
| [Jason Wei](https://md.research.elamur.ai/free/en/authors/jason-wei/) | 14 papers · 23,971 citations |
| [Fei Huang](https://md.research.elamur.ai/free/en/authors/fei-huang/) | 14 papers · 3,764 citations |
| [Jingren Zhou](https://md.research.elamur.ai/free/en/authors/jingren-zhou/) | 14 papers · 2,829 citations |
| [Denny Zhou](https://md.research.elamur.ai/free/en/authors/denny-zhou/) | 13 papers · 20,911 citations |
| [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/) | 12 papers · 4,805 citations |
| [Furu Wei](https://md.research.elamur.ai/free/en/authors/furu-wei/) | 12 papers · 2,389 citations |
| [Yejin Choi](https://md.research.elamur.ai/free/en/authors/yejin-choi/) | 12 papers · 1,511 citations |
| [Firoj Alam](https://md.research.elamur.ai/free/en/authors/firoj-alam/) | 11 papers · 231 citations |
| [Quoc Le](https://md.research.elamur.ai/free/en/authors/quoc-v-le/) | 10 papers · 18,251 citations |
| [Colin Raffel](https://md.research.elamur.ai/free/en/authors/colin-raffel/) | 10 papers · 6,746 citations |
| [Bowen Yu](https://md.research.elamur.ai/free/en/authors/bowen-yu/) | 10 papers · 5,279 citations |
| [Lidong Bing](https://md.research.elamur.ai/free/en/authors/lidong-bing/) | 10 papers · 2,161 citations |

Industrial laboratories built most frontier models: OpenAI (GPT-3, InstructGPT, GPT-4), Google (PaLM), DeepMind (Chinchilla), Meta (LLaMA) and DeepSeek (DeepSeek-R1). Academia supplies much of the analysis, evaluation and open tooling. Active universities include:

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/)
- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/)
- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/)
- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/)
- [Zhejiang University](https://md.research.elamur.ai/free/en/universities/zhejiang-university/)
- the [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/)
- the [University of California, Berkeley](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/)

They work alongside the [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/), [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/), [Google](https://md.research.elamur.ai/free/en/companies/google/) and [Alibaba Group](https://md.research.elamur.ai/free/en/companies/alibaba-group/).

Where the authors on this subject work now Carnegie Mellon University is the current home of authors of 124 papers on this subject; counted are the authors with a page here, by their latest institution.

**The numbers**

| Institution | Papers |
| --- | --- |
| [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) | 124 papers |
| [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/) | 122 papers |
| [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) | 122 papers |
| [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) | 112 papers |
| [Zhejiang University](https://md.research.elamur.ai/free/en/universities/zhejiang-university/) | 101 papers |
| [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) | 96 papers |
| [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) | 90 papers |
| [Google](https://md.research.elamur.ai/free/en/companies/google/) | 86 papers |
| [Alibaba Group](https://md.research.elamur.ai/free/en/companies/alibaba-group/) | 84 papers |
| [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) | 82 papers |

Countries of the authors on this subject United States comes first: authors working there now wrote 755 papers on this subject.

**The numbers**

| Country | Papers |
| --- | --- |
| [United States](https://md.research.elamur.ai/free/en/countries/united-states/) | 755 papers |
| [China](https://md.research.elamur.ai/free/en/countries/china/) | 611 papers |
| [United Kingdom](https://md.research.elamur.ai/free/en/countries/united-kingdom/) | 206 papers |
| Hong Kong | 129 papers |
| [Singapore](https://md.research.elamur.ai/free/en/countries/singapore/) | 92 papers |
| [Canada](https://md.research.elamur.ai/free/en/countries/canada/) | 90 papers |
| [Germany](https://md.research.elamur.ai/free/en/countries/germany/) | 81 papers |
| [Switzerland](https://md.research.elamur.ai/free/en/countries/switzerland/) | 61 papers |
| [Australia](https://md.research.elamur.ai/free/en/countries/australia/) | 57 papers |
| [Japan](https://md.research.elamur.ai/free/en/countries/japan/) | 57 papers |

## Open questions

### Hallucination and factuality

LLMs produce fluent statements that are false or unsupported. One survey divides the problem into two kinds. Factuality hallucinations conflict with verifiable facts. Faithfulness hallucinations diverge from the input or from the model's own earlier text. The survey traces causes to data, training and decoding ([Huang et al. 2023](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/)). Another proposes three types instead: input-conflicting, context-conflicting and fact-conflicting ([Zhang et al. 2023](https://md.research.elamur.ai/free/en/papers/zhang-2023-siren-s-song-ai-ocean/)).

The stakes are concrete. One cited study found leading models hallucinating in 69% to 88% of answers to specific legal queries ([Chua; & Li 2024](https://md.research.elamur.ai/free/en/papers/fan-2024-survey-rag-meeting-llms-towards/)). RLHF reduced fabrication on closed-domain tasks from 41% to 21% of cases for InstructGPT, which still leaves many errors ([Ouyang et al. 2022](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)). Retrieval helps, but retrieval-augmented systems still hallucinate ([Huang et al. 2023](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/)).

### Bias, safety and security

Models trained on unfiltered web text absorb social bias and can amplify it. A survey classifies bias metrics by whether they inspect embeddings, token probabilities or generated text. It sorts mitigations by pipeline stage, from pre-processing to post-processing ([Gallegos et al. 2023](https://md.research.elamur.ai/free/en/papers/gallegos-2023-bias-fairness-large-language-models/)). RLHF did not noticeably reduce InstructGPT's bias scores on two tests ([Ouyang et al. 2022](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)). Winogender probes gendered pronoun resolution, and CrowS-Pairs compares paired stereotyped and less-stereotyped sentences.

On security, a review of 281 publications separates defensive uses, offensive uses and the models' own vulnerabilities. Those vulnerabilities include data poisoning, extraction of training data and prompt injection ([Yao et al. 2023](https://md.research.elamur.ai/free/en/papers/yao-2023-survey-large-language-model-llm/)). Medical and educational deployment raise their own concerns. In medicine, models can give potentially harmful clinical answers ([Singhal et al. 2022](https://md.research.elamur.ai/free/en/papers/singhal-2022-large-language-models-encode-clinical/)). In education, the concerns are the need for AI literacy, fact-checking and fair access ([Kasneci et al. 2023](https://md.research.elamur.ai/free/en/papers/kasneci-2023-chatgpt-good-opportunities-challenges-large/)). These issues are gathered under [bias, fairness and safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/).

### Do they reason, and why do abilities emerge?

Whether generated chains of thought reflect real reasoning remains unsettled. The chain-of-thought authors note that the steps can be wrong even when the answer is right ([Wei et al. 2022](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)). The emergence paper documents sharp thresholds but does not explain them ([Wei et al. 2022](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/)). The prompt-programming critique, described under Prompting and reasoning, adds a further doubt. Some apparent in-context learning may be task location rather than learning ([Reynolds & McDonell 2021](https://md.research.elamur.ai/free/en/papers/reynolds-2021-prompt-programming-large-language-models/)). Explaining what models represent internally is the agenda of [interpretability](https://md.research.elamur.ai/free/en/topics/interpretability/).

### Cost, openness and coverage

Training remains expensive. LLaMA-65B took about 21 days on 2,048 A100 GPUs ([Touvron et al. 2023](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/)), and frontier reports increasingly withhold such details ([OpenAI et al. 2023](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)). Three lines of work lower the bill without removing it:

- compute-optimal training ([Hoffmann et al. 2022](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/));
- parameter-efficient tuning ([Hu et al. 2021](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/));
- better serving ([Kwon et al. 2023](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/)).

Finally, much LLM research centres on English, so speakers of other languages may benefit less ([Kasneci et al. 2023](https://md.research.elamur.ai/free/en/papers/kasneci-2023-chatgpt-good-opportunities-challenges-large/)). That gap links LLM research to work on [multilingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) and [low-resource languages](https://md.research.elamur.ai/free/en/topics/low-resource-languages/).

## The library in figures

Neighbouring subjects: papers shared Natural language processing shares the most: 1304 of the 1,524 papers here (86%) stand on both.

**The numbers**

| Shelf | Papers shared |
| --- | --- |
| [Natural language processing](https://md.research.elamur.ai/free/en/concepts/natural-language-processing/) | 1304 shared · 86% |
| [Few-shot learning](https://md.research.elamur.ai/free/en/concepts/few-shot-learning/) | 211 shared · 14% |
| [Question answering](https://md.research.elamur.ai/free/en/concepts/question-answering/) | 54 shared · 4% |
| [Retrieval-augmented generation](https://md.research.elamur.ai/free/en/concepts/retrieval-augmented-generation/) | 54 shared · 4% |
| [Chatbots](https://md.research.elamur.ai/free/en/concepts/chatbot/) | 46 shared · 3% |
| [Machine learning](https://md.research.elamur.ai/free/en/concepts/machine-learning/) | 43 shared · 3% |
| [Text summarization](https://md.research.elamur.ai/free/en/concepts/text-summarization/) | 32 shared · 2% |
| [Morphology](https://md.research.elamur.ai/free/en/concepts/morphology/) | 29 shared · 2% |
| [Natural language understanding](https://md.research.elamur.ai/free/en/concepts/natural-language-understanding/) | 29 shared · 2% |
| [Knowledge graph](https://md.research.elamur.ai/free/en/concepts/knowledge-graph/) | 20 shared · 1% |

## Try it: Mova’s tokens against GPT-2’s

Type a sentence and see how two tokenizers cut it. [Mova](https://md.research.elamur.ai/free/en/projects/mova/)’s English tokenizer follows the Universal Dependencies conventions: words, punctuation and clitics (`did` + `n’t`), abbreviations kept whole (`U.S.`). GPT-2’s byte-pair encoding cuts by frequency: common words stay whole, a leading space is part of the token, rare words fall into pieces. Both run in your browser; the text is not sent anywhere.

Six of the examples, token by token (Mova’s tokenizer, then GPT-2’s BPE; ␣ marks a space that belongs to the token):

Mova’s tokenizer: a table-driven automaton with dictionaries learned from the English Web Treebank and the ESL spoken treebank of Universal Dependencies (CC BY-SA 4.0). GPT-2: OpenAI’s vocabulary (© 2019 OpenAI, Modified MIT License), its first 12,000 merges of 50,000 — a word the full vocabulary keeps whole can split here.

## Key papers

The works this page cites, the most cited first; citations as counted by OpenAlex plus the papers of this library that cite the work, read 2026-10-24.

- [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/) — Tom B. Brown, Benjamin Mann, Nick Ryder et al. · 2020 · 13,703 citations
- [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/) — OpenAI, Josh Achiam, Steven Adler et al. · 2023 · 12,732 citations
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/) — Jason Wei, Xuezhi Wang, Dale Schuurmans et al. · 2022 · 11,308 citations
- [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/) — Long Ouyang, Jeff Wu, Xu Jiang et al. · 2022 · 10,005 citations
- [LLaMA: Open and Efficient Foundation Language Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/) — Hugo Touvron, Thibaut Lavril, Gautier Izacard et al. · 2023 · 9,472 citations
- [LoRA: Low-Rank Adaptation of Large Language Models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/) — Edward J. Hu, Yelong Shen, Phillip Wallis et al. · 2021 · 8,473 citations
- [ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education](https://md.research.elamur.ai/free/en/papers/kasneci-2023-chatgpt-good-opportunities-challenges-large/) — Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann et al. · 2023 · 7,436 citations
- [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/) — Aakanksha Chowdhery, Sharan Narang, Jacob Devlin et al. · 2022 · 4,680 citations
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/) — DeepSeek-AI, Daya Guo, Dejian Yang et al. · 2025 · 4,593 citations
- [Large Language Models Encode Clinical Knowledge](https://md.research.elamur.ai/free/en/papers/singhal-2022-large-language-models-encode-clinical/) — Karan Singhal, Shekoofeh Azizi, Tao Tu et al. · 2022 · 4,477 citations
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/) — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang et al. · 2023 · 4,453 citations
- [Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models](https://md.research.elamur.ai/free/en/papers/kung-2023-performance-chatgpt-usmle-potential-ai/) — Tiffany H. Kung, Morgan Cheatham, Arielle Medenilla et al. · 2023 · 4,168 citations
- [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/) — Takeshi Kojima, Shixiang Shane Gu, Machel Reid et al. · 2022 · 3,795 citations
- [A Survey on Evaluation of Large Language Models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/) — Yupeng Chang, Xu Wang, Jindong Wang et al. · 2023 · 3,641 citations
- [Scaling Laws for Neural Language Models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/) — Jared Kaplan, Sam McCandlish, Tom Henighan et al. · 2020 · 3,295 citations
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/) — Lei Huang, Weijiang Yu, Weitao Ma et al. · 2023 · 3,222 citations
- [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/) — Rishi Bommasani, Drew A. Hudson, Ehsan Adeli et al. · 2021 · 3,056 citations
- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/) — Wayne Xin Zhao, Kun Zhou, Junyi Li et al. · 2023 · 2,863 citations
- [Improved Baselines with Visual Instruction Tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/) — Haotian Liu, Chunyuan Li, Yuheng Li et al. · 2023 · 2,558 citations
- [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/) — Mark Chen, Jerry Tworek, Heewoo Jun et al. · 2021 · 2,548 citations
- [Visual Instruction Tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/) — Haotian Liu, Chunyuan Li, Qingyang Wu et al. · 2023 · 2,491 citations
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/) — Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan et al. · 2023 · 2,400 citations
- [A Survey on Large Language Model based Autonomous Agents](https://md.research.elamur.ai/free/en/papers/wang-2023-survey-large-language-model-based/) — Lei Wang, Chen Ma, Xueyang Feng et al. · 2023 · 2,267 citations
- [Emergent Abilities of Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/) — Jason Wei, Yi Tay, Rishi Bommasani et al. · 2022 · 2,229 citations
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/) — Junnan Li, Dongxu Li, Silvio Savarese et al. · 2023 · 2,122 citations
- [Training Compute-Optimal Large Language Models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/) — Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch et al. · 2022 · 1,950 citations
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/gao-2023-retrieval-augmented-generation-large-language/) — Yunfan Gao, Yun Xiong, Xinyu Gao et al. · 2023 · 1,648 citations
- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://md.research.elamur.ai/free/en/papers/pan-2023-unifying-large-language-models-knowledge/) — Shirui Pan, Linhao Luo, Yufei Wang et al. · 2023 · 1,549 citations
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://md.research.elamur.ai/free/en/papers/gilardi-2023-chatgpt-outperforms-crowd-workers-text/) — Fabrizio Gilardi, Meysam Alizadeh, Maël Kubli · 2023 · 1,533 citations
- [Code Llama: Open Foundation Models for Code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/) — Baptiste Rozière, Jonas Gehring, Fabian Gloeckle et al. · 2024 · 1,399 citations
- [A Comprehensive Overview of Large Language Models](https://md.research.elamur.ai/free/en/papers/naveed-2023-comprehensive-overview-large-language-models/) — Humza Naveed, Asad Ullah Khan, Shi Qiu et al. · 2023 · 1,286 citations
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://md.research.elamur.ai/free/en/papers/xi-2023-rise-potential-large-language/) — Zhiheng Xi, Wenxiang Chen, Xin Guo et al. · 2023 · 1,264 citations
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://md.research.elamur.ai/free/en/papers/yao-2023-survey-large-language-model-llm/) — Yifan Yao, Jinhao Duan, Kaidi Xu et al. · 2023 · 1,182 citations
- [Least-to-most prompting enables complex reasoning in large language models](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/) — 2022 · 1,148 citations
- [Program Synthesis with Large Language Models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/) — Jacob Austin, Augustus Odena, Maxwell Nye et al. · 2021 · 1,139 citations
- [Autonomous chemical research with large language models](https://md.research.elamur.ai/free/en/papers/boiko-2023-autonomous-chemical-research-large-language/) — Daniil A. Boiko, Robert MacKnight, Ben Kline et al. · 2023 · 1,123 citations
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://md.research.elamur.ai/free/en/papers/reynolds-2021-prompt-programming-large-language-models/) — Laria Reynolds, Kyle McDonell · 2021 · 1,115 citations
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/) — Deyao Zhu, Jun Chen, Xiaoqian Shen et al. · 2023 · 1,106 citations
- [Bias and Fairness in Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/gallegos-2023-bias-fairness-large-language-models/) — Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow et al. · 2023 · 1,074 citations
- [A Survey on Multimodal Large Language Models](https://md.research.elamur.ai/free/en/papers/yin-2023-survey-multimodal-large-language-models/) — Shukang Yin, Chaoyou Fu, Sirui Zhao et al. · 2023 · 1,062 citations
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/) — Wenliang Dai, Junnan Li, Dongxu Li et al. · 2023 · 1,051 citations
- [A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models](https://md.research.elamur.ai/free/en/papers/fan-2024-survey-rag-meeting-llms-towards/) — Wenqi Fan; Yujuan Ding; Liangbo Ning; Shijie Wang; Hengyun Li; Dawei Yin; Tat-Seng Chua;, Qing Li · 2024 · 1,006 citations
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2023-siren-s-song-ai-ocean/) — Yue Zhang, Yafu Li, Leyang Cui et al. · 2023 · 966 citations

## Related concepts

- [Language model](https://md.research.elamur.ai/free/en/concepts/language-model/) — A language model is a probability distribution over sequences of tokens (words, subwords or characters). It is usually factored into…
- [Transformer and attention](https://md.research.elamur.ai/free/en/concepts/transformer/) — The Transformer is a neural network architecture that represents a sequence by letting each element gather information from all others…
- [Scaling laws](https://md.research.elamur.ai/free/en/concepts/scaling-laws/) — Scaling laws are empirical relationships, usually power laws, that describe how a neural network's performance, often its test loss…

## More papers

Topics: [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/)

Shelves: [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/)

Updated 24 October 2026

- [← previous concept Language model](https://md.research.elamur.ai/free/en/concepts/language-model/)
- [next concept → Latent semantic indexing](https://md.research.elamur.ai/free/en/concepts/latent-semantic-indexing/)
- [↑ back to concepts](https://md.research.elamur.ai/free/en/concepts/)