---
title: "Language models & pretraining — papers"
description: "Language models & pretraining: 1039 papers of the Elamur Research library, the most cited first."
canonical: "https://research.elamur.ai/free/en/topics/language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/topics/language-models/)

[home](https://md.research.elamur.ai/free/en/) / [topics](https://md.research.elamur.ai/free/en/topics/) / Language models & pretraining

# 🧠 Language models & pretraining

1039 papers · page 1 of 42 · the most cited first · [all topics](https://md.research.elamur.ai/free/en/topics/)

Models that learn a language from raw text, and how they are pretrained, tuned and prompted.

💡 Explained: [Language model](https://md.research.elamur.ai/free/en/concepts/language-model/) · [Scaling laws](https://md.research.elamur.ai/free/en/concepts/scaling-laws/) · [Subword tokenization](https://md.research.elamur.ai/free/en/concepts/subword-tokenization/) · [Transformer and attention](https://md.research.elamur.ai/free/en/concepts/transformer/) · [Word embeddings](https://md.research.elamur.ai/free/en/concepts/word-embeddings/)

A survey · about 18 minutes to read · every paper named here has its own page in this library

**In this article**

1. [What it is](#a-what-it-is)
2. [History and milestones](#a-history-and-milestones)
3. [Main problems and tasks](#a-main-problems-and-tasks)
4. [Methods and schools](#a-methods-and-schools)
5. [Datasets, resources and benchmarks](#a-datasets-resources-and-benchmarks)
6. [Landmark papers in this library](#a-landmark-papers-in-this-library)
7. [People](#a-people)
8. [Institutions and places](#a-institutions-and-places)
9. [Open questions](#a-open-questions)
10. [Neighbouring topics](#a-neighbouring-topics)
11. [Key works in this library](#a-key-works-in-this-library)

A language model assigns probabilities to sequences of words, or more generally of tokens, and so can predict what comes next in a text. For most of the history of computational linguistics this was a modest component: a statistical module that helped a speech recogniser or a translation decoder prefer fluent output over garbled output. Over roughly the last decade it became the centre of the field. Neural networks trained to predict missing or upcoming tokens in very large unlabelled corpora turned out to learn representations that transfer to almost every language task, and, once scaled far enough, to perform many tasks directly from an instruction or a handful of examples.

This topic gathers the papers in this library on how such models are built, pretrained, adapted and used: from n-gram smoothing and fast lookup structures, through the encoder models of the BERT family and the encoder–decoder and decoder-only systems that followed, to prompting, instruction tuning, reinforcement learning for reasoning, retrieval augmentation and tool use. It also covers the large body of work that studies these models from the outside — what they know, where they fail, how to evaluate them, and what risks they carry.

The collection is heavily weighted towards recent years, reflecting how the field itself grew. Older statistical work is present but sparse; the bulk of the papers date from the period after pretrained Transformers became the default starting point for natural language processing.

## What it is

At its core, language modelling is density estimation over text. A model is given a context and asked to estimate the distribution of a token: the next one (an autoregressive or causal model), a masked one in the middle of a sentence (a masked or denoising model), or a span to be regenerated (as in many encoder–decoder objectives). The quality of the estimate is usually summarised by perplexity on held-out text, though in modern practice models are judged far more by what they can do downstream.

**Pretraining** is the practice of first training a model on such a self-supervised objective over large, generic corpora, and only then adapting it to a particular purpose. Adaptation can take several forms:

- **Fine-tuning** all weights on labelled data for a task.
- **Parameter-efficient tuning**, where only a small set of added or selected parameters is trained while the backbone is frozen.
- **Prompting**, where the task is expressed as text and the frozen model completes it, possibly after seeing a few demonstrations (in-context learning).
- **Instruction tuning and preference-based alignment**, which train a general model to follow requests and to prefer responses that people, or automatic checkers, judge better.

The topic overlaps strongly with several neighbours. [Lexical semantics & word embeddings](https://md.research.elamur.ai/free/en/topics/lexical-semantics/) studies static word vectors, which are in effect the input layer of earlier neural language models. [Text generation](https://md.research.elamur.ai/free/en/topics/text-generation/) and [Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) use language models as engines. [Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) extends pretraining to images and video. [Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) and [Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) increasingly treat large language models as their main object of study. What distinguishes this topic is its focus on the model itself: its objective, architecture, training data, scale and the methods that turn a raw pretrained network into a useful system.

## History and milestones

Landmark papers on this topic: the 12 most cited, in time The 12 most cited papers on this topic span 2019–2025; the dot grows with the citations.

**The numbers**

| Paper | Citations |
| --- | --- |
| [2019 — BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/) | 23,422 |
| [2019 — Momentum Contrast for Unsupervised Visual Representation Learning](https://md.research.elamur.ai/free/en/papers/he-2019-momentum-contrast-unsupervised-visual-representation/) | 13,143 |
| [2019 — BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://md.research.elamur.ai/free/en/papers/lee-2019-biobert-pre-trained-biomedical-language/) | 8,281 |
| [2020 — Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/) | 10,782 |
| [2021 — LoRA: Low-Rank Adaptation of Large Language Models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/) | 5,972 |
| [2021 — Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/) | 5,375 |
| [2022 — Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/) | 11,278 |
| [2022 — Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/) | 5,770 |
| [2023 — GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/) | 12,694 |
| [2023 — ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education](https://md.research.elamur.ai/free/en/papers/kasneci-2023-chatgpt-good-opportunities-challenges-large/) | 7,429 |
| [2023 — LLaMA: Open and Efficient Foundation Language Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/) | 5,609 |
| [2025 — DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/deepseek-2025-ai-deepseek-r1-incentivizing-reasoning-capability/) | 4,593 |

### Statistical language models

The statistical era, roughly from the 1980s to the early 2010s, was dominated by n-gram models: the probability of a word was estimated from counts of the preceding one, two or more words. The central technical problem was sparsity, since most possible word sequences never occur in any training corpus. Smoothing methods redistributed probability mass to unseen events, and [Chen and Goodman's empirical study of smoothing](https://md.research.elamur.ai/free/en/papers/chen-1996-empirical-study-smoothing-techniques-language/) became the standard comparison of these methods, establishing the strength of modified Kneser–Ney smoothing. Bayesian views followed, including [a hierarchical Pitman–Yor language model](https://md.research.elamur.ai/free/en/papers/teh-2006-hierarchical-bayesian-language-model-based/), which reinterpreted Kneser–Ney-style smoothing as approximate inference in a nonparametric model.

Language models also spread beyond speech and translation. [Ponte and Croft's language modelling approach to retrieval](https://md.research.elamur.ai/free/en/papers/ponte-1998-language-modeling-approach-information/) proposed ranking documents by how likely each document's own model was to generate the query, founding a whole line of retrieval research. Noisy-channel spelling correction, illustrated by [an improved noisy-channel error model for spelling](https://md.research.elamur.ai/free/en/papers/brill-2000-improved-error-model-noisy-channel/), and readability estimation, as in [reading level assessment with SVMs and language models](https://md.research.elamur.ai/free/en/papers/schwarm-2005-reading-level-assessment-support-vector/), used language model scores as features.

As machine translation systems grew, engineering mattered as much as modelling. [KenLM](https://md.research.elamur.ai/free/en/papers/heafield-2011-kenlm-faster-smaller-language-model/) and [scalable modified Kneser–Ney estimation](https://md.research.elamur.ai/free/en/papers/heafield-2013-scalable-modified-kneser-ney-language/) made it practical to build and query very large n-gram models, while [intelligent selection of training data](https://md.research.elamur.ai/free/en/papers/moore-2010-intelligent-selection-language-model-training/) addressed which data such models should be trained on — a question that returned with force in the era of web-scale pretraining. [Forest Rescoring](https://md.research.elamur.ai/free/en/papers/huang-2007-forest-rescoring-faster-decoding-integrated/) dealt with integrating the language model efficiently into syntax-based translation decoding.

### Neural language models

Neural language models, proposed by Bengio and colleagues in the early 2000s and given new life by recurrent networks around 2010, replaced counts with learned continuous representations. A shared benchmark helped: [the One Billion Word Benchmark](https://md.research.elamur.ai/free/en/papers/chelba-2013-one-billion-word-benchmark-measuring/) gave the community a large common test bed. Architecture work then explored alternatives to word-level LSTMs. [Character-Aware Neural Language Models](https://md.research.elamur.ai/free/en/papers/kim-2015-character-aware-neural-language-models/) built word representations from characters, which helps morphologically rich languages; [Language Modeling with Gated Convolutional Networks](https://md.research.elamur.ai/free/en/papers/dauphin-2016-language-modeling-gated-convolutional-networks/) showed that non-recurrent, parallelisable networks with gated linear units could compete with recurrent models; [work on reusing the output embedding](https://md.research.elamur.ai/free/en/papers/press-2017-output-embedding-improve-language-models/) introduced the now-standard tying of input and output embeddings; and [Regularizing and Optimizing LSTM Language Models](https://md.research.elamur.ai/free/en/papers/merity-2017-regularizing-optimizing-lstm-language-models/) set strong LSTM baselines through careful regularisation.

### From features to pretraining

The idea that a language model could supply features for other tasks matured around 2017–2018. [TagLM-style semi-supervised sequence tagging](https://md.research.elamur.ai/free/en/papers/peters-2017-semi-supervised-sequence-tagging-bidirectional/) added pretrained language model states to a sequence tagger, a direct precursor of ELMo. In 2018 the Transformer-based GPT and BERT showed that a single pretrained network, fine-tuned with a thin task head, could set new results across standard benchmarks. The following two years saw a rapid sequence of refinements to the pretraining objective — permutation, span, unified and replaced-token objectives — together with domain-specific and language-specific variants.

### Scale, prompting and alignment

GPT-3 in 2020 demonstrated that a sufficiently large autoregressive model could perform tasks from a prompt with a few examples and no gradient updates. Research on prompting, in-context learning and scaling followed. From 2022 onward, chain-of-thought prompting, instruction tuning, reinforcement learning from human feedback and the release of ChatGPT turned large language models into general-purpose assistants and moved them into almost every corner of NLP. The most recent phase, represented here by work on reinforcement learning with verifiable rewards and on test-time compute, concentrates on reasoning.

Papers on this topic by year 1,039 dated papers from 1996 to 2026; the busiest year is 2023 with 304, and 97% of them appeared in 2018 or later.

**The numbers**

| Year | Papers |
| --- | --- |
| 2026 | 182 |
| 2025 | 75 |
| 2024 | 189 |
| 2023 | 304 |
| 2022 | 99 |
| 2021 | 71 |
| 2020 | 49 |
| 2019 | 39 |
| 2018 | 5 |
| 2017 | 6 |
| 2016 | 1 |
| 2015 | 1 |
| 2014 | 3 |
| 2013 | 3 |
| 2011 | 1 |
| 2010 | 1 |
| 2009 | 1 |
| 2007 | 1 |
| 2006 | 1 |
| 2005 | 1 |
| 2003 | 1 |
| 2001 | 2 |
| 2000 | 1 |
| 1998 | 1 |
| 1996 | 1 |

## Main problems and tasks

Work in this topic clusters around a handful of recurring questions.

**Choosing a pretraining objective.** Should the model predict the next token, reconstruct masked tokens, regenerate spans or detect replaced tokens? Each objective trades off understanding against generation, and several papers here try to unify them.

**Adapting to domains and languages.** General models trained on Wikipedia and books underperform on biomedical, scientific, legal or social-media text and on languages other than English. Continued pretraining, pretraining from scratch on in-domain data and custom vocabularies are the main remedies.

**Eliciting and measuring knowledge.** Language models absorb factual and commonsense knowledge. How much, how reliably, and how to read it out are the subjects of probing studies.

**Making models do tasks without task-specific training.** Prompt design, automatic prompt search, demonstration selection and calibration all aim to make few-shot and zero-shot use dependable.

**Reasoning.** Multi-step arithmetic, logic and planning were long weak points. Prompting strategies, decoding strategies and reinforcement learning target them.

**Grounding and tools.** Retrieval augmentation, tool calls and agent loops connect a model to external information and actions.

**Reliability and safety.** Hallucination, toxicity, social bias, memorisation of training data, jailbreaks and adversarial attacks are studied both as phenomena and as targets for mitigation.

**Efficiency.** Training and serving large models is costly; compression, pruning, parameter-efficient tuning and fast inference engines are active areas, overlapping with [Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/).

## Methods and schools

### Architectures

Three families of Transformer models dominate. **Encoder-only** models (BERT, RoBERTa, ELECTRA, DeBERTa) read the whole input bidirectionally and are strong at classification, tagging and extraction. **Encoder–decoder** models (T5, BART and summarisation-oriented systems such as [PEGASUS](https://md.research.elamur.ai/free/en/papers/zhang-2019-pegasus-pre-training-extracted-gap/)) map an input to an output sequence. **Decoder-only** models (the GPT line, PaLM, LLaMA and their descendants) generate left to right and have become the default for large general-purpose systems. Several papers here try to combine the strengths of these families in one network, notably [UniLM](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/), [GLM](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/) and [MASS](https://md.research.elamur.ai/free/en/papers/song-2019-mass-masked-sequence-sequence-pre/). Architectural work on efficiency includes mixture-of-experts scaling, as in [GLaM](https://md.research.elamur.ai/free/en/papers/du-2021-glam-efficient-scaling-language-models/).

### Pretraining objectives

The masked language modelling of BERT was quickly questioned and extended. [XLNet](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/) used permutations of the factorisation order; [SpanBERT](https://md.research.elamur.ai/free/en/papers/joshi-2019-spanbert-improving-pre-training-representing/) masked contiguous spans and predicted them from their boundaries; [MPNet](https://md.research.elamur.ai/free/en/papers/song-2020-mpnet-masked-permuted-pre-training/) combined masking and permutation; [ERNIE](https://md.research.elamur.ai/free/en/papers/sun-2019-ernie-enhanced-representation-through-knowledge/) and [ERNIE 2.0](https://md.research.elamur.ai/free/en/papers/sun-2019-ernie-2-0-continual/) masked entities and phrases and introduced continual multi-task pretraining; [DeBERTaV3](https://md.research.elamur.ai/free/en/papers/he-2021-debertav3-improving-deberta-using-electra/) adopted ELECTRA's replaced-token detection; [LinkBERT](https://md.research.elamur.ai/free/en/papers/yasunaga-2022-linkbert-pretraining-language-models-document/) used hyperlinks between documents as an additional signal. A recurring finding is that the objective can matter as much as the amount of data or parameters.

### Domain and language specialisation

Domain models form a school of their own. Biomedicine produced [BioBERT](https://md.research.elamur.ai/free/en/papers/lee-2019-biobert-pre-trained-biomedical-language/) and [PubMedBERT](https://md.research.elamur.ai/free/en/papers/gu-2020-domain-specific-language-model-pretraining/); science produced [SciBERT](https://md.research.elamur.ai/free/en/papers/beltagy-2019-scibert-pretrained-language-model-scientific/) and [Galactica](https://md.research.elamur.ai/free/en/papers/taylor-2022-galactica-large-language-model-science/); finance produced [BloombergGPT](https://md.research.elamur.ai/free/en/papers/wu-2023-bloomberggpt-large-language-model-finance/) and [FinGPT](https://md.research.elamur.ai/free/en/papers/yang-2023-fingpt-open-source-financial-large/); social media produced [BERTweet](https://md.research.elamur.ai/free/en/papers/nguyen-2020-bertweet-pre-trained-language-model/). Language-specific models include [CamemBERT](https://md.research.elamur.ai/free/en/papers/martin-2019-camembert-tasty-french-language-model/) for French, [PhoBERT](https://md.research.elamur.ai/free/en/papers/nguyen-2020-phobert-pre-trained-language-models/) for Vietnamese, Chinese whole-word-masking models in [the Chinese whole-word-masking study](https://md.research.elamur.ai/free/en/papers/cui-2019-pre-training-whole-word-masking/), and [IndicNLPSuite](https://md.research.elamur.ai/free/en/papers/kakwani-2020-indicnlpsuite-monolingual-corpora-evaluation-benchmarks/) for Indian languages. The main debate is between continuing to train a general model and training from scratch on in-domain text; the PubMedBERT study argued that, where in-domain text is plentiful, the latter wins.

### Prompting and in-context learning

Prompting began as a probing tool and became an adaptation method. Cloze-style prompts were used to read out facts, then automated in [AutoPrompt](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/) and replaced by continuous prompts in [GPT Understands, Too](https://md.research.elamur.ai/free/en/papers/liu-2021-gpt-understands-too/). For mid-sized models, prompt-based fine-tuning showed strong few-shot results in [LM-BFF](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/) and [It's Not Just Size That Matters](https://md.research.elamur.ai/free/en/papers/schick-2021-it-s-not-just-size/). For large models, the questions became which demonstrations to choose ([retrieving similar examples for GPT-3](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/), [learned prompt retrieval](https://md.research.elamur.ai/free/en/papers/rubin-2021-learning-retrieve-prompts-context-learning/)), how to correct biases in the output distribution ([Calibrate Before Use](https://md.research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/)), and what in-context learning actually relies on ([Rethinking the Role of Demonstrations](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/)). A sceptical line, including [true few-shot learning](https://md.research.elamur.ai/free/en/papers/perez-2021-true-few-shot-learning-language/), warned that tuning prompts on held-out data overstates few-shot ability. Surveys such as [Liu et al.'s prompting survey](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/) and [A Survey on In-context Learning](https://md.research.elamur.ai/free/en/papers/dong-2023-survey-context-learning/) organise this literature.

### Reasoning methods

Chain-of-thought prompting asks the model to write intermediate steps before answering. Variants include zero-shot triggers, sampling several chains and voting, decomposing problems into easier subproblems ([Least-to-most prompting](https://md.research.elamur.ai/free/en/papers/al-2022-least-most-prompting-enables-complex/)), automatically constructing demonstrations ([Automatic Chain of Thought Prompting](https://md.research.elamur.ai/free/en/papers/zhang-2022-automatic-chain-thought-prompting-large/)) and scratchpads ([Show your work](https://md.research.elamur.ai/free/en/papers/al-2021-show-your-work-scratchpads-intermediate/)). Training-based approaches distil reasoning into smaller models ([Distilling Step-by-Step!](https://md.research.elamur.ai/free/en/papers/hsieh-2023-distilling-step-step-outperforming-larger/), [specialising smaller models for multi-step reasoning](https://md.research.elamur.ai/free/en/papers/fu-2023-specializing-smaller-language-models-towards/)), bootstrap from the model's own outputs ([Large Language Models Can Self-Improve](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/)), or use process supervision ([Math-Shepherd](https://md.research.elamur.ai/free/en/papers/wang-2024-math-shepherd-verify-reinforce-llms/)). Reinforcement learning against checkable answers, as in DeepSeekMath, DeepSeek-R1 and [DAPO](https://md.research.elamur.ai/free/en/papers/yu-2025-dapo-open-source-llm-reinforcement/), and allocating more computation at inference time ([Scaling LLM Test-Time Compute Optimally](https://md.research.elamur.ai/free/en/papers/snell-2024-scaling-llm-test-time-compute/)) define the current frontier. A persistent debate concerns whether written reasoning reflects what the model actually computes; a study of [unfaithful chain-of-thought explanations](https://md.research.elamur.ai/free/en/papers/turpin-2023-language-models-don-t-always/) gives evidence that it often does not.

### Instruction tuning and alignment

Instruction tuning trains a model on many tasks phrased as instructions. Relevant papers here include [The Flan Collection](https://md.research.elamur.ai/free/en/papers/longpre-2023-flan-collection-designing-data-methods/), [Instruction Tuning with GPT-4](https://md.research.elamur.ai/free/en/papers/peng-2023-instruction-tuning-gpt-4/), [WizardLM](https://md.research.elamur.ai/free/en/papers/xu-2023-wizardlm-empowering-large-pre-trained/), [How Far Can Camels Go?](https://md.research.elamur.ai/free/en/papers/wang-2023-far-camels-go-exploring-state/) and the crowd-sourced [OpenAssistant Conversations](https://md.research.elamur.ai/free/en/papers/kopf-2023-openassistant-conversations-democratizing-large-language/). Feedback-based training is represented by [fine-grained human feedback](https://md.research.elamur.ai/free/en/papers/wu-2023-fine-grained-human-feedback-gives/), and the survey [Zhang et al.'s instruction-tuning survey](https://md.research.elamur.ai/free/en/papers/zhang-2023-instruction-tuning-large-language-models/) maps the area.

### Parameter-efficient adaptation

As models grew, full fine-tuning became expensive to store and run. [A Nature Machine Intelligence study of delta-tuning](https://md.research.elamur.ai/free/en/papers/ding-2023-parameter-efficient-fine-tuning-large/) frames adapters, LoRA, prefix and prompt tuning as one family; [work on intrinsic dimensionality](https://md.research.elamur.ai/free/en/papers/aghajanyan-2021-intrinsic-dimensionality-explains-effectiveness-language/) offers a theoretical account of why so few parameters suffice; [a comparison of few-shot parameter-efficient tuning with in-context learning](https://md.research.elamur.ai/free/en/papers/liu-2022-few-shot-parameter-efficient-fine/) compares the approach directly with prompting; [LLM-Adapters](https://md.research.elamur.ai/free/en/papers/hu-2023-llm-adapters-adapter-family-parameter/) and [LlamaFactory](https://md.research.elamur.ai/free/en/papers/zheng-2024-llamafactory-unified-efficient-fine-tuning/) package the methods for practical use.

### Retrieval, tools and agents

Retrieval augmentation gives a model access to an external corpus at inference time. Surveys ([Gao et al.'s RAG survey](https://md.research.elamur.ai/free/en/papers/gao-2023-retrieval-augmented-generation-large-language/), [A Survey on RAG Meeting LLMs](https://md.research.elamur.ai/free/en/papers/fan-2024-survey-rag-meeting-llms-towards/)) and methods ([REPLUG](https://md.research.elamur.ai/free/en/papers/shi-2023-replug-retrieval-augmented-black-box/), [query rewriting for retrieval](https://md.research.elamur.ai/free/en/papers/ma-2023-query-rewriting-retrieval-augmented-large/), [RAFT](https://md.research.elamur.ai/free/en/papers/zhang-2024-raft-adapting-language-model-domain/)) define the area. Tool use and agents extend the idea from documents to actions: [Toolformer](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/), [ToolLLM](https://md.research.elamur.ai/free/en/papers/qin-2023-toolllm-facilitating-large-language-models/), [Gorilla](https://md.research.elamur.ai/free/en/papers/patil-2023-gorilla-large-language-model-connected/), [ReAct](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/) and [Search-R1](https://md.research.elamur.ai/free/en/papers/jin-2025-search-r1-training-llms-reason/).

## Datasets, resources and benchmarks

Pretraining corpora are mostly web-derived. Well-known examples include BooksCorpus and English Wikipedia (used for BERT), Common Crawl and its cleaned derivatives such as C4 and OSCAR, and curated mixtures used for later open models. Earlier, the Penn Treebank and the [One Billion Word Benchmark](https://md.research.elamur.ai/free/en/papers/chelba-2013-one-billion-word-benchmark-measuring/) were the standard perplexity test sets, joined later by WikiText-103.

Evaluation has moved from perplexity and GLUE/SuperGLUE towards broad capability suites. Papers here introduce or describe several widely used ones:

| Resource | What it tests |
| --- | --- |
| [BIG-bench](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/) | A large collaborative set of hard tasks for extrapolating capability with scale |
| [HumanEval](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/) | Functional correctness of generated Python code, judged by unit tests |
| [LAMA](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) | Factual and commonsense knowledge via cloze queries |
| [RealToxicityPrompts](https://md.research.elamur.ai/free/en/papers/gehman-2020-realtoxicityprompts-evaluating-neural-toxic-degeneration/) | Toxic continuations of naturally occurring prompts |
| [StereoSet](https://md.research.elamur.ai/free/en/papers/nadeem-2021-stereoset-measuring-stereotypical-bias-pretrained/) and [CrowS-Pairs](https://md.research.elamur.ai/free/en/papers/nangia-2020-crows-pairs-challenge-dataset-measuring/) | Stereotypical social bias in masked models |
| [AGIEval](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/) and [C-Eval](https://md.research.elamur.ai/free/en/papers/huang-2023-c-eval-multi-level-multi/) | Human exam questions in English and Chinese |
| [SWE-bench](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/) and [LiveCodeBench](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/) | Real software issues and contamination-resistant coding problems |
| [HaluEval](https://md.research.elamur.ai/free/en/papers/li-2023-halueval-large-scale-hallucination-evaluation/) | Recognition of hallucinated content |
| [Instruction-Following Evaluation](https://md.research.elamur.ai/free/en/papers/zhou-2023-instruction-following-evaluation-large-language/) | Compliance with verifiable formatting instructions |

Conversation logs such as [LMSYS-Chat-1M](https://md.research.elamur.ai/free/en/papers/zheng-2023-lmsys-chat-1m-large-scale/) document how people actually use deployed models. For the broader landscape of benchmarks, see [Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) and the survey [Chang et al.'s evaluation survey](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/).

## Landmark papers in this library

Citations of the papers on this topic, by when they were published Papers published in 2020–2024 gather 74% of the 446,030 citations of the papers on this topic.

**The numbers**

| Published | Papers and their citations |
| --- | --- |
| 2025–2029 | 257 papers, 10,325 citations |
| 2020–2024 | 712 papers, 331,834 citations |
| 2015–2019 | 52 papers, 90,451 citations |
| 2010–2014 | 8 papers, 6,446 citations |
| 2005–2009 | 4 papers, 1,526 citations |
| 2000–2004 | 4 papers, 2,372 citations |
| 1995–1999 | 2 papers, 3,076 citations |

### GPT-4 Technical Report (2023)

[The GPT-4 report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/) describes a large multimodal model that takes text and images as input. It is notable both for its results on professional and academic exams and for what it leaves out: architecture, size and training data are withheld. Its emphasis on forecasting final loss from much smaller training runs made predictable scaling an explicit engineering goal. The later [GPT-4o System Card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/) applied a similar disclosure style to a model that also handles audio, focusing on red-teaming and risk evaluation.

### Chain-of-Thought Prompting (2022)

[Chain-of-thought prompting](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/) showed that including worked reasoning in few-shot demonstrations sharply improves large models on arithmetic, commonsense and symbolic tasks, with the benefit appearing only beyond a certain scale. It became the starting point for an entire subfield of reasoning methods.

### Large Language Models Are Zero-Shot Reasoners (2022)

[This paper](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/) found that a single generic trigger phrase asking the model to reason step by step recovers much of the chain-of-thought gain without any demonstrations, setting a strong and simple baseline.

### Self-Consistency (2022)

[Self-consistency](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/) replaces greedy decoding of a single reasoning chain with sampling many chains and taking a majority vote over final answers. It needs no extra training and became a standard companion to chain-of-thought prompting.

### DeepSeek-R1 (2025)

[DeepSeek-R1](https://md.research.elamur.ai/free/en/papers/deepseek-2025-ai-deepseek-r1-incentivizing-reasoning-capability/) trained reasoning behaviour with reinforcement learning against rule-checked rewards, skipping supervised fine-tuning entirely in its first variant. Long reasoning traces, self-checking and reflection emerged from this process. The approach built on the GRPO algorithm introduced in [DeepSeekMath](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/), which also showed how carefully filtered web mathematics can lift a small open model.

### XLNet (2019)

[XLNet](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/) proposed permutation language modelling, keeping an autoregressive objective while letting each token learn from context on both sides, and borrowed recurrence and relative positions from Transformer-XL. It was one of the most discussed post-BERT objectives.

### BioBERT (2019) and SciBERT (2019)

[BioBERT](https://md.research.elamur.ai/free/en/papers/lee-2019-biobert-pre-trained-biomedical-language/) continued BERT's pretraining on biomedical literature; [SciBERT](https://md.research.elamur.ai/free/en/papers/beltagy-2019-scibert-pretrained-language-model-scientific/) pretrained on scientific papers with its own vocabulary. Together they established domain-adaptive pretraining as standard practice and remain common baselines in biomedical and scientific text mining.

### Language Models as Knowledge Bases? (2019)

[This study](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) introduced the LAMA probe and showed that pretrained masked models, without fine-tuning, recall a surprising amount of relational knowledge. It launched the probing literature that followed, including [LPAQA](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/), [Negated and Misprimed Probes](https://md.research.elamur.ai/free/en/papers/kassner-2020-negated-misprimed-probes-pretrained-language/) and [a study of long-tail knowledge](https://md.research.elamur.ai/free/en/papers/kandpal-2022-large-language-models-struggle-learn/).

### Pre-train, Prompt, and Predict (2021)

[This survey](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/) gave prompting a common vocabulary and placed it in a sequence of paradigms in NLP, from feature engineering through architecture engineering and pretrain-then-fine-tune to prompt engineering.

### On the Opportunities and Risks of Foundation Models (2021)

[The Stanford report](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/) coined the term "foundation models" and argued that two forces — emergence of abilities with scale and homogenisation around a few base models — define the period. It shaped both research agendas and policy discussion.

### Emergent Abilities of Large Language Models (2022)

[This paper](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/) collected cases where task performance stays near chance until models pass a certain scale and then rises sharply. The claim was influential and contested; later analyses, including BIG-bench's own, argued that some apparent jumps depend on the choice of metric.

### Lost in the Middle (2023)

[Lost in the Middle](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/) showed that models with long context windows are most accurate when the relevant evidence is placed at the beginning or end of the input, and least accurate when it is buried in the middle — a result with direct consequences for retrieval-augmented systems.

### Evaluating Large Language Models Trained on Code (2021)

[The Codex paper](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/) introduced HumanEval and the pass@k metric, making execution-based testing the norm for code generation. [Code Llama](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/) later provided strong open-weight code models, and [EvalPlus](https://md.research.elamur.ai/free/en/papers/liu-2023-your-code-generated-chatgpt-really/) strengthened the tests themselves.

### ReAct (2022) and Toolformer (2023)

[ReAct](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/) interleaved free-text reasoning with actions in an environment, giving an early template for language model agents. [Toolformer](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/) let a model teach itself where API calls help, keeping only calls that make the following text easier to predict.

### Vision–language pretraining

Several of the most cited works here carry pretraining into vision. [LXMERT](https://md.research.elamur.ai/free/en/papers/tan-2019-lxmert-learning-cross-modality-encoder/) and [ViLBERT](https://md.research.elamur.ai/free/en/papers/lu-2019-vilbert-pretraining-task-agnostic-visiolinguistic/) were early cross-modal Transformers; [Flamingo](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/), [BLIP-2](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/) and [Visual Instruction Tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/) connected frozen vision encoders to large language models. Self-supervised vision methods such as [Momentum Contrast](https://md.research.elamur.ai/free/en/papers/he-2019-momentum-contrast-unsupervised-visual-representation/) were explicitly motivated by the success of unsupervised pretraining in language.

## People

Authors with the most papers on this topic Furu Wei leads with 16 papers; the 12 authors shown wrote 126 of the papers on this topic between them, counting each paper once per author.

**The numbers**

| Author | Papers here · their citations |
| --- | --- |
| [Furu Wei](https://md.research.elamur.ai/free/en/authors/furu-wei/) | 16 papers · 9,405 citations |
| [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/) | 14 papers · 9,302 citations |
| [Denny Zhou](https://md.research.elamur.ai/free/en/authors/denny-zhou/) | 13 papers · 20,468 citations |
| [Weizhu Chen](https://md.research.elamur.ai/free/en/authors/weizhu-chen/) | 11 papers · 11,047 citations |
| [Jianfeng Gao](https://md.research.elamur.ai/free/en/authors/jianfeng-gao/) | 11 papers · 7,998 citations |
| [Jason Wei](https://md.research.elamur.ai/free/en/authors/jason-wei/) | 10 papers · 18,611 citations |
| [Nan Duan](https://md.research.elamur.ai/free/en/authors/nan-duan/) | 10 papers · 3,820 citations |
| [Quoc Le](https://md.research.elamur.ai/free/en/authors/quoc-v-le/) | 9 papers · 19,786 citations |
| [Xuezhi Wang](https://md.research.elamur.ai/free/en/authors/xuezhi-wang/) | 8 papers · 18,391 citations |
| [Li Dong](https://md.research.elamur.ai/free/en/authors/li-dong/) | 8 papers · 4,851 citations |
| [Eric Wallace](https://md.research.elamur.ai/free/en/authors/eric-wallace/) | 8 papers · 4,679 citations |
| [Lei Li](https://md.research.elamur.ai/free/en/authors/lei-li/) | 8 papers · 3,281 citations |

[Furu Wei](https://md.research.elamur.ai/free/en/authors/furu-wei/), of Microsoft Research, is the most prolific author on this topic. His papers here range across unified pretraining ([UniLM](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/), [UniLMv2](https://md.research.elamur.ai/free/en/papers/bao-2020-unilmv2-pseudo-masked-language-models/)), document understanding ([LayoutLM](https://md.research.elamur.ai/free/en/papers/xu-2019-layoutlm-pre-training-text-layout/)) and vision–language models ([VL-BERT](https://md.research.elamur.ai/free/en/papers/su-2019-vl-bert-pre-training-generic/), [Oscar](https://md.research.elamur.ai/free/en/papers/li-2020-oscar-object-semantics-aligned-pre/)). His frequent co-authors [Li Dong](https://md.research.elamur.ai/free/en/authors/li-dong/) and [Ming Zhou](https://md.research.elamur.ai/free/en/authors/ming-zhou/) appear on much of the same work; Ming Zhou also co-authored [HIBERT](https://md.research.elamur.ai/free/en/papers/zhang-2019-hibert-document-level-pre-training/) and [UniXcoder](https://md.research.elamur.ai/free/en/papers/guo-2022-unixcoder-unified-cross-modal-pre/).

[Denny Zhou](https://md.research.elamur.ai/free/en/authors/denny-zhou/) (Google) and [Jason Wei](https://md.research.elamur.ai/free/en/authors/jason-wei/) (now at OpenAI) are central to the reasoning literature: chain-of-thought prompting, self-consistency, the emergent-abilities analysis and the Flan instruction-tuning collection. [Xuezhi Wang](https://md.research.elamur.ai/free/en/authors/xuezhi-wang/) led the self-consistency work and contributed to multilingual chain-of-thought. [Quoc Le](https://md.research.elamur.ai/free/en/authors/quoc-v-le/) links this line with earlier pretraining research as a co-author of both XLNet and chain-of-thought prompting.

[Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/) (University of Washington and Meta) appears on SpanBERT, Toolformer, [Mask-Predict](https://md.research.elamur.ai/free/en/papers/ghazvininejad-2019-mask-predict-parallel-decoding-conditional/) and the analysis of what makes in-context learning work. [Hannaneh Hajishirzi](https://md.research.elamur.ai/free/en/authors/hannaneh-hajishirzi/) (University of Washington and the Allen Institute for AI) works on in-context learning, [noisy channel prompting](https://md.research.elamur.ai/free/en/papers/min-2021-noisy-channel-language-model-prompting/) and open instruction tuning. [Danqi Chen](https://md.research.elamur.ai/free/en/authors/danqi-chen/), whose affiliations include Princeton University, contributed to SpanBERT, few-shot prompt-based fine-tuning, [citation-generating models](https://md.research.elamur.ai/free/en/papers/gao-2023-enabling-large-language-models-generate/) and [context compression](https://md.research.elamur.ai/free/en/papers/chevalier-2023-adapting-language-models-compress-contexts/).

[Jianfeng Gao](https://md.research.elamur.ai/free/en/authors/jianfeng-gao/) (Microsoft) spans pretraining objectives, DeBERTaV3, [DialoGPT](https://md.research.elamur.ai/free/en/papers/zhang-2019-dialogpt-large-scale-generative-pre/) and vision–language pretraining; [Weizhu Chen](https://md.research.elamur.ai/free/en/authors/weizhu-chen/) and [Pengcheng He](https://md.research.elamur.ai/free/en/authors/pengcheng-he/) are his co-authors on DeBERTaV3, and Chen also studied demonstration selection for GPT-3. [Nan Duan](https://md.research.elamur.ai/free/en/authors/nan-duan/) (Microsoft Research Asia) works on code pretraining and retrieval-augmented generation.

[Eric Wallace](https://md.research.elamur.ai/free/en/authors/eric-wallace/) (now at OpenAI) is associated with AutoPrompt, calibration of few-shot prompting, [extraction of memorised training data](https://md.research.elamur.ai/free/en/papers/carlini-2020-extracting-training-data-large-language/) and long-tail knowledge. [Samuel R. Bowman](https://md.research.elamur.ai/free/en/authors/samuel-r-bowman/) and [Ethan Perez](https://md.research.elamur.ai/free/en/authors/ethan-perez/) (Anthropic) appear on bias measurement, red-teaming ([automated red-teaming](https://md.research.elamur.ai/free/en/papers/perez-2022-red-teaming-language-models-language/), [a large human red-teaming study](https://md.research.elamur.ai/free/en/papers/ganguli-2022-red-teaming-language-models-reduce/)) and the faithfulness of chain-of-thought explanations.

[Percy Liang](https://md.research.elamur.ai/free/en/authors/percy-liang/) (Stanford University) contributed to long-context analysis, LinkBERT, [FlexGen](https://md.research.elamur.ai/free/en/papers/sheng-2023-flexgen-high-throughput-generative-inference/) and the study of [whose opinions language models reflect](https://md.research.elamur.ai/free/en/papers/santurkar-2023-whose-opinions-language-models-reflect/). [Yejin Choi](https://md.research.elamur.ai/free/en/authors/yejin-choi/) (Stanford) and [Noah A. Smith](https://md.research.elamur.ai/free/en/authors/noah-a-smith/) (Allen Institute) co-authored RealToxicityPrompts; Choi also worked on [symbolic knowledge distillation](https://md.research.elamur.ai/free/en/papers/west-2022-symbolic-knowledge-distillation-general-language/) and [maieutic prompting](https://md.research.elamur.ai/free/en/papers/jung-2022-maieutic-prompting-logically-consistent-reasoning/). [Hinrich Schütze](https://md.research.elamur.ai/free/en/authors/hinrich-schutze/) (LMU Munich) contributed pattern-based few-shot learning with small models, [dataset generation with pretrained models](https://md.research.elamur.ai/free/en/papers/schick-2021-generating-datasets-pretrained-language-models/) and [multilingual knowledge probing](https://md.research.elamur.ai/free/en/papers/kassner-2021-multilingual-lama-investigating-knowledge-multilingual/).

Among authors of the older statistical work, [Kenneth Heafield](https://md.research.elamur.ai/free/en/authors/kenneth-heafield/) built the KenLM toolkit and its scalable estimator, and [Yoon Kim](https://md.research.elamur.ai/free/en/authors/yoon-kim/) co-authored the character-aware neural model and an early study of [temporal language change with neural language models](https://md.research.elamur.ai/free/en/papers/kim-2014-temporal-analysis-language-through-neural/).

## Institutions and places

Where the authors on this topic work now Carnegie Mellon University is the current home of authors of 88 papers on this topic; counted are the authors with a page here, by their latest institution.

**The numbers**

| Institution | Papers |
| --- | --- |
| [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) | 88 papers |
| [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) | 84 papers |
| [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/) | 80 papers |
| [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) | 80 papers |
| [Microsoft (United States)](https://md.research.elamur.ai/free/en/companies/microsoft/) | 79 papers |
| [Google (United States)](https://md.research.elamur.ai/free/en/companies/google/) | 76 papers |
| [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) | 73 papers |
| [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) | 67 papers |
| [University of Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/universities/university-of-chinese-academy-of-sciences/) | 58 papers |
| [Zhejiang University](https://md.research.elamur.ai/free/en/universities/zhejiang-university/) | 56 papers |

Among the current homes of authors on this topic, Carnegie Mellon University, Tsinghua University, Microsoft, the Chinese Academy of Sciences, Stanford University, the University of Washington, Peking University and Google lead. The list mixes universities with industrial laboratories — Microsoft Research Asia, Alibaba, Google DeepMind, OpenAI and the Allen Institute for AI all appear prominently — which reflects how much of large-scale pretraining depends on computing resources concentrated in companies.

Countries of the authors on this topic United States comes first: authors working there now wrote 586 papers on this topic.

**The numbers**

| Country | Papers |
| --- | --- |
| United States | 586 papers |
| China | 392 papers |
| United Kingdom | 210 papers |
| Hong Kong | 90 papers |
| Canada | 81 papers |
| Singapore | 70 papers |
| Germany | 67 papers |
| India | 50 papers |
| France | 47 papers |
| Israel | 46 papers |

Geographically the field is dominated by the United States and China, followed at a distance by the United Kingdom, Hong Kong, Canada, Singapore and Germany.

Where the papers on this topic were published Proceedings of the Fifteenth Language Resources and Evaluation Conference carries the most, 89 of 1039.

**The numbers**

| Venue | Papers |
| --- | --- |
| Proceedings of the Fifteenth Language Resources and Evaluation Conference | 89 papers |
| Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) | 19 papers |
| Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing | 15 papers |
| Findings of the Association for Computational Linguistics: EMNLP 2023 | 14 papers |
| Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) | 13 papers |
| Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) | 13 papers |
| Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) | 11 papers |
| Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics | 10 papers |
| Findings of the Association for Computational Linguistics: ACL 2024 | 9 papers |
| Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (*SEM 2026) | 9 papers |

Publication is concentrated in the main computational linguistics venues — the ACL annual meetings, EMNLP, NAACL and their Findings volumes, and TACL — alongside the language resources conference LREC. A large share of the most cited work first appeared as arXiv preprints, often with later conference or journal versions.

## Open questions

**What do these models know, and how reliably?** Probing studies show substantial stored knowledge, but also sensitivity to phrasing, weakness on rare facts and confident errors. Hallucination remains a central problem; surveys such as [Huang et al.'s hallucination survey](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/) and [Siren's Song in the AI Ocean](https://md.research.elamur.ai/free/en/papers/zhang-2023-siren-s-song-ai-ocean/) classify its causes, and [work on snowballing hallucinations](https://md.research.elamur.ai/free/en/papers/zhang-2023-language-model-hallucinations-snowball/) shows how early errors propagate.

**Do models reason?** Gains from chain-of-thought and reinforcement learning are large, yet explanations can be unfaithful, models are distracted by [irrelevant context](https://md.research.elamur.ai/free/en/papers/shi-2023-large-language-models-easily-distracted/), and some improvements may make outputs more reliable rather than adding new abilities.

**How should models be evaluated?** Benchmark contamination, saturation and the gap between benchmark scores and real use are recurring concerns, motivating contamination-resistant and continuously refreshed tests.

**Safety and security.** [universal adversarial suffixes](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/), [Jailbroken](https://md.research.elamur.ai/free/en/papers/wei-2023-jailbroken-llm-safety-training-fail/) and [query-efficient black-box jailbreaking](https://md.research.elamur.ai/free/en/papers/chao-2023-jailbreaking-black-box-large-language/) show that safety training can be circumvented. Bias persists, as documented in [studies of anti-Muslim bias](https://md.research.elamur.ai/free/en/papers/abid-2021-persistent-anti-muslim-bias-large/) and [a survey of bias and fairness](https://md.research.elamur.ai/free/en/papers/gallegos-2023-bias-fairness-large-language-models/). Questions of release policy, raised early in [OpenAI's report on release strategies](https://md.research.elamur.ai/free/en/papers/solaiman-2019-release-strategies-social-impacts-language/), remain unsettled.

**Interpretability.** Why particular behaviours arise is still poorly understood. Work from [attention analysis](https://md.research.elamur.ai/free/en/papers/vig-2019-analyzing-structure-attention-transformer-language/) to [inference-time intervention](https://md.research.elamur.ai/free/en/papers/li-2023-inference-time-intervention-eliciting-truthful/) and [language-specific neurons](https://md.research.elamur.ai/free/en/papers/tang-2024-language-specific-neurons-key-multilingual/) tries to open the black box; [a survey of explainability methods](https://md.research.elamur.ai/free/en/papers/zhao-2023-explainability-large-language-models-survey/) summarises methods.

**Languages beyond English.** Models remain strongest in English. Tokenisers that fragment other languages ([How Good is Your Tokenizer?](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/)), uneven training data and English-centred benchmarks all disadvantage other languages, a concern shared with [Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) and [Low-resource languages](https://md.research.elamur.ai/free/en/topics/low-resource-languages/).

**Language models as models of people.** Studies such as [Out of One, Many](https://md.research.elamur.ai/free/en/papers/argyle-2022-out-one-many-using-language/), [theory-of-mind evaluations](https://md.research.elamur.ai/free/en/papers/kosinski-2023-evaluating-large-language-models-theory/) and [What BERT Is Not](https://md.research.elamur.ai/free/en/papers/ettinger-2020-what-bert-is-not-lessons/) ask how far model behaviour resembles human cognition and whether models can stand in for human participants — questions that connect to [Psycholinguistics & language acquisition](https://md.research.elamur.ai/free/en/topics/psycholinguistics/).

## Neighbouring topics

Neighbouring topics: papers shared Reasoning shares the most: 99 of the 1039 papers here (10%) stand on both.

**The numbers**

| Topic | Papers shared |
| --- | --- |
| [Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) | 99 shared · 10% |
| [Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) | 95 shared · 9% |
| [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) | 89 shared · 9% |
| [Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) | 75 shared · 7% |
| [Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) | 62 shared · 6% |
| [Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) | 59 shared · 6% |
| [Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) | 30 shared · 3% |
| [Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) | 24 shared · 2% |
| [Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) | 20 shared · 2% |
| [Psycholinguistics & language acquisition](https://md.research.elamur.ai/free/en/topics/psycholinguistics/) | 20 shared · 2% |

The closest neighbour is [Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/), since much recent reasoning research is research on prompting and training large language models. [Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) follows closely, as most new benchmarks are built to test such models. [Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) shares the many multimodal pretraining papers. [Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) covers toxicity, stereotypes, red-teaming and jailbreaks. [Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/), [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) and [Hebrew & Aramaic](https://md.research.elamur.ai/free/en/topics/hebrew-aramaic/) cover language-specific models and evaluations.

Applications connect the topic to [Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) (retrieval augmentation and query-likelihood ranking), [Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/), [Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/), [Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) and [Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/), where language modelling ideas now extend to audio, as in [AudioLM](https://md.research.elamur.ai/free/en/papers/borsos-2022-audiolm-language-modeling-approach-audio/). Technical foundations link it to [Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/), [Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/), [Interpretability](https://md.research.elamur.ai/free/en/topics/interpretability/) and [Morphology](https://md.research.elamur.ai/free/en/topics/morphology/). Smaller overlaps exist with [Idioms & multiword expressions](https://md.research.elamur.ai/free/en/topics/multiword-expressions/) and [Semantic parsing & meaning representation](https://md.research.elamur.ai/free/en/topics/semantic-parsing/).

The shelves this topic stands on Linguistics & languages: 320 of the 1039 papers here.

**The numbers**

| Shelf | Papers |
| --- | --- |
| [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) | 320 papers |
| [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) | 309 papers |
| [Other](https://md.research.elamur.ai/free/en/shelves/other/) | 287 papers |
| [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/) | 110 papers |
| [Mathematics & statistics](https://md.research.elamur.ai/free/en/shelves/mathematics/) | 81 papers |
| [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) | 32 papers |
| [Computing & GPUs](https://md.research.elamur.ai/free/en/shelves/computing/) | 26 papers |
| [Education](https://md.research.elamur.ai/free/en/shelves/education/) | 3 papers |
| [History & humanities](https://md.research.elamur.ai/free/en/shelves/history/) | 3 papers |
| [Law](https://md.research.elamur.ai/free/en/shelves/law/) | 3 papers |
| [Life sciences & medicine](https://md.research.elamur.ai/free/en/shelves/life-sciences/) | 1 paper |
| [Society & ethics](https://md.research.elamur.ai/free/en/shelves/society/) | 1 paper |

The topic draws on several shelves: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/), [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/), [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/), [Mathematics & statistics](https://md.research.elamur.ai/free/en/shelves/mathematics/), [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) and [Computing & GPUs](https://md.research.elamur.ai/free/en/shelves/computing/).

## Key works in this library

- [Chen and Goodman's empirical study of smoothing](https://md.research.elamur.ai/free/en/papers/chen-1996-empirical-study-smoothing-techniques-language/) — the reference comparison of n-gram smoothing.
- [Ponte and Croft's language modelling approach to retrieval](https://md.research.elamur.ai/free/en/papers/ponte-1998-language-modeling-approach-information/) — query likelihood as a ranking principle.
- [KenLM](https://md.research.elamur.ai/free/en/papers/heafield-2011-kenlm-faster-smaller-language-model/) — efficient n-gram storage for translation decoders.
- [Character-Aware Neural Language Models](https://md.research.elamur.ai/free/en/papers/kim-2015-character-aware-neural-language-models/) — words built from characters with convolutions.
- [Language Modeling with Gated Convolutional Networks](https://md.research.elamur.ai/free/en/papers/dauphin-2016-language-modeling-gated-convolutional-networks/) — gated linear units and parallel convolutional language models.
- [XLNet](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/) — permutation-based autoregressive pretraining.
- [SpanBERT](https://md.research.elamur.ai/free/en/papers/joshi-2019-spanbert-improving-pre-training-representing/) — span masking and a boundary objective.
- [Unified Language Model Pre-training](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/) — one network for understanding and generation via attention masks.
- [BioBERT](https://md.research.elamur.ai/free/en/papers/lee-2019-biobert-pre-trained-biomedical-language/) — continued pretraining for biomedical text.
- [SciBERT](https://md.research.elamur.ai/free/en/papers/beltagy-2019-scibert-pretrained-language-model-scientific/) — a scientific-domain encoder with its own vocabulary.
- [CamemBERT](https://md.research.elamur.ai/free/en/papers/martin-2019-camembert-tasty-french-language-model/) — a monolingual French model trained on web text.
- [Language Models as Knowledge Bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) — the LAMA knowledge probe.
- [CTRL](https://md.research.elamur.ai/free/en/papers/keskar-2019-ctrl-conditional-transformer-language-model/) — controllable generation through control codes.
- [Codex and HumanEval](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/) — execution-based testing of generated code.
- [On the opportunities and risks of foundation models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/) — the Stanford report that coined the term.
- [Liu et al.'s prompting survey](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/) — "pre-train, prompt, and predict" as a paradigm.
- [LM-BFF](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/) — prompt-based fine-tuning for mid-sized models.
- [Chain-of-Thought Prompting](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/) — reasoning steps in demonstrations.
- [Self-Consistency](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/) — voting over sampled reasoning chains.
- [Rethinking the Role of Demonstrations](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/) — what in-context learning depends on.
- [Beyond the Imitation Game](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/) — the BIG-bench collaborative benchmark.
- [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/) — a frontier multimodal model and predictable scaling.
- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/) — a broad, regularly updated overview.
- [Lost in the Middle](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/) — position effects in long contexts.
- [Toolformer](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/) — self-supervised learning of API use.
- [DeepSeekMath](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/) — mathematical pretraining data and the GRPO algorithm.
- [DeepSeek-R1](https://md.research.elamur.ai/free/en/papers/deepseek-2025-ai-deepseek-r1-incentivizing-reasoning-capability/) — reasoning learned through reinforcement learning with verifiable rewards.

## All 1,039 papers on this topic

**1** [2](https://md.research.elamur.ai/free/en/topics/language-models/page/2/) … [42](https://md.research.elamur.ai/free/en/topics/language-models/page/42/) [next →](https://md.research.elamur.ai/free/en/topics/language-models/page/2/)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/) — Jacob Devlin, Ming-Wei Chang, Kenton Lee, et al. · 2019 · 23 422 citations
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://md.research.elamur.ai/free/en/papers/he-2019-momentum-contrast-unsupervised-visual-representation/) — Kaiming He, Haoqi Fan, Yuxin Wu, et al. · 2019 · 13 143 citations
- [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/) — OpenAI, Josh Achiam, Steven Adler, et al. · 2023 · 12 694 citations
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/) — Jason Wei, Xuezhi Wang, Dale Schuurmans, et al. · 2022 · 11 278 citations
- [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/) — Tom B. Brown, Benjamin Mann, Nick Ryder, et al. · 2020 · 10 782 citations
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://md.research.elamur.ai/free/en/papers/lee-2019-biobert-pre-trained-biomedical-language/) — Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, et al. · 2019 · 8 281 citations
- [ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education](https://md.research.elamur.ai/free/en/papers/kasneci-2023-chatgpt-good-opportunities-challenges-large/) — Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, et al. · 2023 · 7 429 citations
- [LoRA: Low-Rank Adaptation of Large Language Models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/) — Edward J. Hu, Yelong Shen, Phillip Wallis, et al. · 2021 · 5 972 citations
- [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/) — Long Ouyang, Jeff Wu, Xu Jiang, et al. · 2022 · 5 770 citations
- [LLaMA: Open and Efficient Foundation Language Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/) — Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. · 2023 · 5 609 citations
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/) — Pengfei Liu, Weizhe Yuan, Jinlan Fu, et al. · 2021 · 5 375 citations
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/deepseek-2025-ai-deepseek-r1-incentivizing-reasoning-capability/) — DeepSeek-AI, Daya Guo, Dejian Yang, et al. · 2025 · 4 593 citations
- [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/) — Takeshi Kojima, Shixiang Shane Gu, Machel Reid, et al. · 2022 · 3 780 citations
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/) — Zhilin Yang, Zihang Dai, Yiming Yang, et al. · 2019 · 3 740 citations
- [A Survey on Evaluation of Large Language Models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/) — Yupeng Chang, Xu Wang, Jindong Wang, et al. · 2023 · 3 640 citations
- [SciBERT: A Pretrained Language Model for Scientific Text](https://md.research.elamur.ai/free/en/papers/beltagy-2019-scibert-pretrained-language-model-scientific/) — Iz Beltagy, Kyle Lo, Arman Cohan · 2019 · 3 548 citations
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/) — Lei Huang, Weijiang Yu, Weitao Ma, et al. · 2023 · 3 221 citations
- [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/) — Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, et al. · 2021 · 3 044 citations
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/) — Xuezhi Wang, Jason Wei, Dale Schuurmans, et al. · 2022 · 3 035 citations
- [Language Models as Knowledge Bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) — Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, et al. · 2019 · 2 977 citations
- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/) — Wayne Xin Zhao, Kun Zhou, Junyi Li, et al. · 2023 · 2 847 citations
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/) — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al. · 2023 · 2 738 citations
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://md.research.elamur.ai/free/en/papers/tan-2019-lxmert-learning-cross-modality-encoder/) — Hao Tan, Mohit Bansal · 2019 · 2 714 citations
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://md.research.elamur.ai/free/en/papers/gu-2020-domain-specific-language-model-pretraining/) — Yu Gu; Robert Tinn; Hao Cheng; Michael Lucas; Naoto Usuyama; Xiaodong Liu; Tristan Naumann; Jianfeng Gao;, Hoifung Poon · 2020 · 2 636 citations
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://md.research.elamur.ai/free/en/papers/joshi-2019-spanbert-improving-pre-training-representing/) — Mandar Joshi, Danqi Chen, Yinhan Liu, et al. · 2019 · 2 580 citations

**1** [2](https://md.research.elamur.ai/free/en/topics/language-models/page/2/) … [42](https://md.research.elamur.ai/free/en/topics/language-models/page/42/) [next →](https://md.research.elamur.ai/free/en/topics/language-models/page/2/)

- [← previous topic Knowledge graphs & ontologies](https://md.research.elamur.ai/free/en/topics/knowledge-graphs/)
- [next topic → Latin](https://md.research.elamur.ai/free/en/topics/latin/)
- [↑ back to topics](https://md.research.elamur.ai/free/en/topics/)