---
title: "Text classification — methods, history and key papers"
description: "Text classification is the task of assigning one or more labels from a fixed, predefined set to a piece of text, such as a sentence, review…"
canonical: "https://research.elamur.ai/free/en/concepts/text-classification/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/concepts/text-classification/)

[home](https://md.research.elamur.ai/free/en/) / [concepts](https://md.research.elamur.ai/free/en/concepts/) / Text classification

# 🗃️ Text classification

Field · 42 key papers · [all concepts](https://md.research.elamur.ai/free/en/concepts/)

Part of: [🗨️ Natural language processing](https://md.research.elamur.ai/free/en/concepts/natural-language-processing/) · [🗂️ Topic modeling](https://md.research.elamur.ai/free/en/concepts/topic-modeling/) · [🪄 Few-shot learning](https://md.research.elamur.ai/free/en/concepts/few-shot-learning/) · [📊 Machine learning](https://md.research.elamur.ai/free/en/concepts/machine-learning/)

Text classification is the task of assigning one or more labels from a fixed, predefined set to a piece of text, such as a sentence, review, message or document. The decision is usually made by a model learned from examples that already carry labels.

Also: text categorization, document classification, automated text categorization

About 18 minutes to read · every paper named here has its own page in this library

**On this page**

1. [What it is](#a-what-it-is)
2. [How it works](#a-how-it-works)
3. [History](#a-history)
4. [Methods](#a-methods)
5. [Datasets and benchmarks](#a-datasets-and-benchmarks)
6. [People and institutions](#a-people-and-institutions)
7. [Open questions](#a-open-questions)
8. [The library in figures](#a-in-figures)

## What it is

A spam filter decides whether each incoming email is spam or not. A news site files every article under a section such as sports or politics. A company sorts product reviews into positive and negative. Each is a case of text classification: a program reads a text and returns a label from a list fixed in advance. It is one of the oldest and most widely deployed tasks in [natural language processing](https://md.research.elamur.ai/free/en/concepts/natural-language-processing/) and text mining. Other tasks are often handled in the same form: natural language inference, for instance, can be treated as choosing a label for a pair of sentences ([Schick & Schütze 2021](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/)).

A standard survey frames the task as approximating an unknown function. That function assigns true or false to every pair of a document and a category (Sebastiani 2002). In this framing a category is a bare symbol, and the classifier relies only on the text itself, not on outside metadata (Sebastiani 2002). Three settings are usually distinguished:

- Binary classification has two classes, such as spam versus legitimate mail, or positive versus negative opinion ([Pang et al. 2002](https://md.research.elamur.ai/free/en/papers/pang-2002-thumbs-up-sentiment-classification-machine/)).
- Multi-class classification has more than two mutually exclusive classes, for example star ratings from 1 to 5, or one of ten topics for a question ([Yang et al. 2016](https://md.research.elamur.ai/free/en/papers/yang-2016-hierarchical-attention-networks-document-classification/)).
- Multi-label classification lets one text carry several labels at once. One example is a newswire story coded with several topics and regions ([Lewis et al. 2004](https://md.research.elamur.ai/free/en/papers/lewis-2004-rcv1-benchmark-collection-text-categorization/)). Another is a clinical note given several diagnosis codes, a setting tested in one experiment of a general label-embedding method ([Wang et al. 2018](https://md.research.elamur.ai/free/en/papers/wang-2018-joint-embedding-words-labels-text/)).

A multi-label problem over a set of categories can be split into one independent binary decision per category, provided the categories can be treated as independent. For that reason much of the classical literature studies the binary case (Sebastiani 2002). Thorsten Joachims, for example, trained one binary classifier for each category ([Joachims 1998](https://md.research.elamur.ai/free/en/papers/joachims-1998-text-categorization-support-vector-machines/)).

The labels can describe almost any property of a text:

- its topic;
- the sentiment it expresses, a large area covered under [sentiment and emotion analysis](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/);
- the attitude toward a single aspect or target inside a sentence ([Wang et al. 2016](https://md.research.elamur.ai/free/en/papers/wang-2016-attention-based-lstm-aspect-level/));
- whether it is abusive or hateful ([Waseem & Hovy 2016](https://md.research.elamur.ai/free/en/papers/waseem-2016-hateful-symbols-or-hateful-people/));
- its genre or author (Stamatatos et al. 2000);
- the dialogue act an utterance performs ([Lee & Dernoncourt 2016](https://md.research.elamur.ai/free/en/papers/lee-2016-sequential-short-text-classification-recurrent/)).

The task is related to [topic modeling](https://md.research.elamur.ai/free/en/concepts/topic-modeling/) and [information retrieval](https://md.research.elamur.ai/free/en/topics/information-retrieval/), but it differs from both. The categories are given beforehand, rather than discovered from the data or ranked against a query.

The shelves its papers stand on Artificial Intelligence: 152 of the 154 papers here.

**The numbers**

| Shelf | Papers |
| --- | --- |
| [Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) | 152 papers |
| [Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) | 126 papers |
| [Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) | 13 papers |
| [Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) | 8 papers |
| [Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) | 8 papers |
| [General Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/general-social-sciences/) | 4 papers |
| [Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) | 4 papers |
| [Management Science and Operations Research](https://md.research.elamur.ai/free/en/shelves/decision-sciences/management-science-and-operations-research/) | 3 papers |
| [Political Science and International Relations](https://md.research.elamur.ai/free/en/shelves/social-sciences/political-science-and-international-relations/) | 3 papers |
| [Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) | 3 papers |
| [Communication](https://md.research.elamur.ai/free/en/shelves/social-sciences/communication/) | 2 papers |
| [Molecular Biology](https://md.research.elamur.ai/free/en/shelves/biochemistry-genetics-and-molecular-biology/molecular-biology/) | 2 papers |

## How it works

Sebastiani organized the field around three problems (Sebastiani 2002):

1. Turning raw text into a representation a learner can use.
2. Building a classifier from labelled examples.
3. Measuring how well that classifier works.

Older systems were sets of hand-written rules. During the 1990s these gave way to a generic learning procedure that induces the classifier from a labelled collection (Sebastiani 2002).

### Representing text

The classic representation is the bag of words. A document becomes a vector with one dimension per vocabulary term, recording whether the term occurs or how often. Word order is discarded. Such vectors are very high-dimensional and sparse. Joachims observed that text tasks easily involve more than ten thousand word stems, and that few of these features are useless on their own ([Joachims 1998](https://md.research.elamur.ai/free/en/papers/joachims-1998-text-categorization-support-vector-machines/)). Raw counts are commonly reweighted by term frequency–inverse document frequency (TF-IDF). This textbook scheme from information retrieval goes back to Spärck Jones (1972):

```
tfidf(t, d) = tf(t, d) · log( N / df(t) )
```

Here tf(t, d) is how often term t occurs in document d, and N is the number of documents in the collection. df(t) counts the documents in which t appears. The logarithm shrinks the weight of terms that appear almost everywhere. Vectors are often scaled to unit length afterwards, as in Joachims's experiments (Joachims 1999). Word n-grams (pairs or triples of adjacent words) can be added as extra features. Character n-grams are an alternative that copes better with misspellings and deliberate obfuscation ([Schmidt & Wiegand 2017](https://md.research.elamur.ai/free/en/papers/schmidt-2017-survey-hate-speech-detection-natural/)).

Neural models replace sparse counts with dense vectors. Each word, or each piece produced by [subword tokenization](https://md.research.elamur.ai/free/en/concepts/subword-tokenization/), is mapped to a learned [word embedding](https://md.research.elamur.ai/free/en/concepts/word-embeddings/). A network then composes these vectors into a representation of the whole text ([Kim 2014](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/)).

### Learning a decision rule

Given vectors and labels, a learner fits a function that maps a new vector to a class. Most classical learners are linear. They compute a weighted sum of features and compare it with a threshold. Logistic regression, in the binary case, turns that sum into a probability:

```
P(y = 1 | x) = 1 / (1 + exp(−(w · x + b)))
```

Here x is the feature vector of a text, w is a vector of learned weights with one weight per feature, b is a bias term, and y is the label. Training adjusts w and b so that the labelled examples receive high probability. A neural classifier follows the same pattern. Its last layer is usually a softmax, the multi-class extension of this formula, placed on top of a learned text representation.

## History

### From rules to learning

Before the 1990s, text categorization systems were largely built by knowledge engineering: experts wrote the rules. Sebastiani's survey records how learned classifiers took over during that decade (Sebastiani 2002). Their effectiveness became comparable to that of trained human indexers, at far lower labour cost. The gains came largely from boosting and support vector machines (SVMs) (Sebastiani 2002). Rocchio's relevance-feedback method (Rocchio 1971) and nearest-neighbour classifiers served as common baselines.

Several papers from 1998 shaped the following decade. Joachims argued that SVMs suit text, because their capacity is controlled by the margin rather than by the number of features. In his experiments they beat naive Bayes, Rocchio, k-nearest neighbours (k-NN) and C4.5 decision trees on two collections ([Joachims 1998](https://md.research.elamur.ai/free/en/papers/joachims-1998-text-categorization-support-vector-machines/)). In the same year, Andrew McCallum and Kamal Nigam separated two incompatible models that had both been called naive Bayes (McCallum & Nigam 1998).

Labels were costly, so attention soon turned to unlabelled text. Nigam and colleagues combined naive Bayes with expectation-maximization (Nigam et al. 2000). Building on Vapnik's transductive framework, Joachims developed transductive SVMs with a training algorithm that scales to large collections. These SVMs fit a separator to the labelled data and the waiting test set together (Joachims 1999). In 2002, Bo Pang, Lillian Lee and Shivakumar Vaithyanathan asked whether opinion could be classified like topic. They found it noticeably harder ([Pang et al. 2002](https://md.research.elamur.ai/free/en/papers/pang-2002-thumbs-up-sentiment-classification-machine/)).

### Neural models

Around 2014 the field moved to neural networks built on pretrained word vectors. Yoon Kim showed that a single convolutional layer over word2vec embeddings could match far more elaborate systems ([Kim 2014](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/)). In 2015 three alternatives followed:

- networks that read raw characters ([Zhang et al. 2015](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/));
- convolutions applied directly to one-hot word vectors ([Johnson & Zhang 2015](https://md.research.elamur.ai/free/en/papers/johnson-2015-effective-use-word-order-text/));
- deep averaging networks that ignore word order altogether ([Iyyer et al. 2015](https://md.research.elamur.ai/free/en/papers/iyyer-2015-deep-unordered-composition-rivals-syntactic/)).

Hierarchical models then encoded sentences first and documents second, with recurrent units ([Tang et al. 2015](https://md.research.elamur.ai/free/en/papers/tang-2015-document-modeling-gated-recurrent-neural/)) and with attention ([Yang et al. 2016](https://md.research.elamur.ai/free/en/papers/yang-2016-hierarchical-attention-networks-document-classification/)). In 2017 a very deep character-level convolutional network (ConvNet) showed that depth can pay off for text as it had for images ([Conneau et al. 2017](https://md.research.elamur.ai/free/en/papers/conneau-2017-very-deep-convolutional-networks-text/)). In parallel, fastText showed that a well-engineered linear model remains a hard baseline to beat (Joulin et al. 2017).

### Pretraining and prompting

The next shift was to pretrain a [language model](https://md.research.elamur.ai/free/en/concepts/language-model/) on raw text and then fine-tune it on the classification task. Universal Language Model Fine-tuning (ULMFiT) applied this recipe to text classification in 2018 (Howard & Ruder 2018). BERT, built on the [Transformer](https://md.research.elamur.ai/free/en/concepts/transformer/) encoder of Vaswani and colleagues (2017), made it the default. It added a single output layer and set new results on eleven benchmark tasks ([Devlin et al. 2019](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)). Later work recast classification as filling a blank in a natural-language prompt ([Schick & Schütze 2021](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/)) or as checking whether a text entails a label description ([Yin et al. 2019](https://md.research.elamur.ai/free/en/papers/yin-2019-benchmarking-zero-shot-text-classification/)). Both moves led toward few-shot and zero-shot use of [large language models](https://md.research.elamur.ai/free/en/concepts/large-language-models/).

Papers on this subject by year 154 dated papers from 1998 to 2026; the busiest year is 2020 with 19, and 68% of them appeared in 2018 or later.

**The numbers**

| Year | Papers |
| --- | --- |
| 2026 | 8 |
| 2025 | 1 |
| 2024 | 6 |
| 2023 | 12 |
| 2022 | 9 |
| 2021 | 19 |
| 2020 | 19 |
| 2019 | 16 |
| 2018 | 15 |
| 2017 | 9 |
| 2016 | 11 |
| 2015 | 6 |
| 2014 | 4 |
| 2012 | 3 |
| 2011 | 1 |
| 2010 | 4 |
| 2009 | 1 |
| 2007 | 2 |
| 2005 | 2 |
| 2004 | 1 |
| 2003 | 2 |
| 2002 | 1 |
| 2000 | 1 |
| 1998 | 1 |

Landmark papers on this subject: the 12 most cited, in time The 12 most cited papers on this subject span 1998–2017; the dot grows with the citations.

**The numbers**

| Paper | Citations |
| --- | --- |
| [1998 — Text Categorization with Support Vector Machines: Learning with Many Relevant Features](https://md.research.elamur.ai/free/en/papers/joachims-1998-text-categorization-support-vector-machines/) | 8,311 |
| [2002 — Thumbs up? Sentiment Classification using Machine Learning Techniques](https://md.research.elamur.ai/free/en/papers/pang-2002-thumbs-up-sentiment-classification-machine/) | 7,762 |
| [2004 — RCV1: A New Benchmark Collection for Text Categorization Research](https://md.research.elamur.ai/free/en/papers/lewis-2004-rcv1-benchmark-collection-text-categorization/) | 2,752 |
| [2005 — Seeing Stars: Exploiting Class Relationships for Sentiment Categorization with Respect to Rating Scales](https://md.research.elamur.ai/free/en/papers/pang-2005-seeing-stars-exploiting-class-relationships/) | 2,733 |
| [2007 — Biographies, Bollywood, Boom-boxes and Blenders: Domain Adaptation for Sentiment Classification](https://md.research.elamur.ai/free/en/papers/blitzer-2007-biographies-bollywood-boom-boxes-blenders/) | 2,329 |
| [2010 — Twitter as a Corpus for Sentiment Analysis and Opinion Mining](https://md.research.elamur.ai/free/en/papers/pak-2010-twitter-as-corpus-sentiment-analysis/) | 2,915 |
| [2014 — Convolutional Neural Networks for Sentence Classification](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/) | 16,269 |
| [2015 — Character-level Convolutional Networks for Text Classification](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/) | 4,781 |
| [2016 — Hierarchical Attention Networks for Document Classification](https://md.research.elamur.ai/free/en/papers/yang-2016-hierarchical-attention-networks-document-classification/) | 5,423 |
| [2016 — Attention-based LSTM for Aspect-level Sentiment Classification](https://md.research.elamur.ai/free/en/papers/wang-2016-attention-based-lstm-aspect-level/) | 2,619 |
| [2016 — Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter](https://md.research.elamur.ai/free/en/papers/waseem-2016-hateful-symbols-or-hateful-people/) | 2,167 |
| [2017 — A Survey on Hate Speech Detection using Natural Language Processing](https://md.research.elamur.ai/free/en/papers/schmidt-2017-survey-hate-speech-detection-natural/) | 1,805 |

Citations of the papers on this subject, by when they were published Papers published in 2015–2019 gather 42% of the 109,479 citations of the papers on this subject.

**The numbers**

| Published | Papers and their citations |
| --- | --- |
| 2025–2029 | 9 papers, 7 citations |
| 2020–2024 | 65 papers, 10,441 citations |
| 2015–2019 | 57 papers, 46,254 citations |
| 2010–2014 | 12 papers, 27,293 citations |
| 2005–2009 | 5 papers, 6,113 citations |
| 2000–2004 | 5 papers, 11,060 citations |
| 1995–1999 | 1 papers, 8,311 citations |

## Methods

### Naive Bayes

Naive Bayes applies Bayes' rule and assumes that words are independent of one another once the class is known. In the multinomial event model, a document is a sequence of word draws from a class-specific distribution:

```
ĉ = argmax over c of  P(c) · ∏ᵢ P(wᵢ | c)

P(w | c) = (count(w, c) + 1) / (Σ_w′ count(w′, c) + |V|)
```

Here ĉ is the predicted class, P(c) is the prior probability of class c, w₁ … wₙ are the word tokens of the document, count(w, c) is how often word w occurs in the training documents of class c, and |V| is the vocabulary size. The added one is Laplace smoothing, which keeps unseen words from zeroing out the product. In practice the product is computed as a sum of logarithms.

The multi-variate Bernoulli model instead records only whether each vocabulary word is present or absent. It also includes explicit factors for absent words. McCallum and Nigam derived both models from one mixture framework (McCallum & Nigam 1998). Bernoulli sometimes won with very small vocabularies. The multinomial model generally won as the vocabulary grew, and across five corpora it cut error by 27% on average (McCallum & Nigam 1998).

### Support vector machines and logistic regression

SVMs find the hyperplane that separates two classes with the widest margin. Joachims argued that sparse, high-dimensional and often linearly separable text vectors are exactly the case margins handle well. On his data, no overfitting appeared even with all features, so separate feature selection became unnecessary ([Joachims 1998](https://md.research.elamur.ai/free/en/papers/joachims-1998-text-categorization-support-vector-machines/)). The table gives his microaveraged precision/recall-breakeven scores on the Reuters ModApte split.

| Method on Reuters-21578 (ModApte) | Breakeven |
| --- | --- |
| Naive Bayes | 72.0 |
| C4.5 | 79.4 |
| Rocchio | 79.9 |
| k-NN | 82.3 |
| SVM, polynomial kernel | 86.0 |
| SVM, RBF kernel | 86.4 |

Source: ([Joachims 1998](https://md.research.elamur.ai/free/en/papers/joachims-1998-text-categorization-support-vector-machines/)).

On the RCV1 benchmark, SVMs again came out ahead of k-NN and Rocchio-style classifiers at every category frequency ([Lewis et al. 2004](https://md.research.elamur.ai/free/en/papers/lewis-2004-rcv1-benchmark-collection-text-categorization/)). The same study found that the effectiveness gap between two SVM variants came mainly from how decision thresholds were chosen ([Lewis et al. 2004](https://md.research.elamur.ai/free/en/papers/lewis-2004-rcv1-benchmark-collection-text-categorization/)).

Sida Wang and Christopher Manning showed that the ranking of naive Bayes and SVMs depends on the data. Multinomial naive Bayes did better on short snippets, while SVMs did better on full-length reviews ([Wang & Manning 2012](https://md.research.elamur.ai/free/en/papers/wang-2012-baselines-bigrams-simple-good-sentiment/)). Their hybrid, NBSVM (a naive Bayes–SVM combination), trains an SVM on features scaled by naive Bayes log-count ratios. It reached 91.22% accuracy on IMDB ([Wang & Manning 2012](https://md.research.elamur.ai/free/en/papers/wang-2012-baselines-bigrams-simple-good-sentiment/)). Logistic regression is the other standard linear learner. Over character n-grams it served as a strong hate speech baseline ([Waseem & Hovy 2016](https://md.research.elamur.ai/free/en/papers/waseem-2016-hateful-symbols-or-hateful-people/)). A one-versus-rest variant with L2 regularization was used for a three-way hate, offensive and neither task ([Davidson et al. 2017](https://md.research.elamur.ai/free/en/papers/davidson-2017-automated-hate-speech-detection-problem/)).

### Convolutional networks

Kim's convolutional neural network (CNN) slides filters of widths 3, 4 and 5 over the matrix of word embeddings. It keeps each filter's maximum response and classifies with a softmax layer regularized by dropout ([Kim 2014](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/)). Pretrained vectors helped much more than random initialization. On the MR movie-review set, the randomly initialized variant reached only 76.1% accuracy, against 81.5% for the variant that fine-tunes word2vec vectors ([Kim 2014](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/)).

Character-level ConvNets took the opposite route. They used a 70-symbol alphabet with no notion of words ([Zhang et al. 2015](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/)). Their main finding concerned data size. TF-IDF n-gram models stayed strong up to a few hundred thousand training samples, and the character model pulled ahead only at millions ([Zhang et al. 2015](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/)). VDCNN (very deep convolutional neural network) extended this idea with up to 49 convolutional layers. Accuracy improved up to 29 layers, and the deepest networks needed residual shortcuts ([Conneau et al. 2017](https://md.research.elamur.ai/free/en/papers/conneau-2017-very-deep-convolutional-networks-text/)).

Rie Johnson and Tong Zhang fed one-hot vectors directly into the convolution, so the network learned embeddings of whole text regions. They report this as the first successful use of word order for topic classification ([Johnson & Zhang 2015](https://md.research.elamur.ai/free/en/papers/johnson-2015-effective-use-word-order-text/)).

### Recurrent, hierarchical and attention models

Recurrent neural networks (RNNs) read a text word by word and carry a hidden state forward. For long documents, a two-level design proved effective. Duyu Tang, Bing Qin and Ting Liu built sentence vectors with a CNN or a long short-term memory (LSTM) network, then composed them with a gated recurrent network. They found that a plain document-level RNN worked poorly, while gating helped a great deal ([Tang et al. 2015](https://md.research.elamur.ai/free/en/papers/tang-2015-document-modeling-gated-recurrent-neural/)).

The Hierarchical Attention Network adds attention at both the word and the sentence level. The weights are scored against learned context vectors, and they can be shown to readers as an explanation of the decision ([Yang et al. 2016](https://md.research.elamur.ai/free/en/papers/yang-2016-hierarchical-attention-networks-document-classification/)). Label-embedding attention places labels in the same space as words and weights each word by how well it matches each label ([Wang et al. 2018](https://md.research.elamur.ai/free/en/papers/wang-2018-joint-embedding-words-labels-text/)). For a fixed two-level taxonomy, HDLTex (Hierarchical Deep Learning for Text classification) trains a parent network for the broad field and a child network for each subfield (Kowsari et al. 2017).

Aspect-level sentiment became a testbed for target-aware architectures:

- recursive networks over parses re-rooted at the target ([Dong et al. 2014](https://md.research.elamur.ai/free/en/papers/dong-2014-adaptive-recursive-neural-network-target/));
- target-dependent LSTMs ([Tang et al. 2016](https://md.research.elamur.ai/free/en/papers/tang-2016-effective-lstms-target-dependent-sentiment/));
- aspect embeddings that steer attention ([Wang et al. 2016](https://md.research.elamur.ai/free/en/papers/wang-2016-attention-based-lstm-aspect-level/));
- multi-hop memory networks ([Tang et al. 2016](https://md.research.elamur.ai/free/en/papers/tang-2016-aspect-level-sentiment-classification-deep/));
- multi-grained attention ([Fan et al. 2018](https://md.research.elamur.ai/free/en/papers/fan-2018-multi-grained-attention-network-aspect/));
- target-specific transformations ([Li et al. 2018](https://md.research.elamur.ai/free/en/papers/li-2018-transformation-networks-target-oriented-sentiment/));
- graph convolutions over parses produced by [dependency parsing](https://md.research.elamur.ai/free/en/concepts/dependency-parsing/) ([Zhang et al. 2019](https://md.research.elamur.ai/free/en/papers/zhang-2019-aspect-based-sentiment-classification-aspect/); [Tang et al. 2020](https://md.research.elamur.ai/free/en/papers/tang-2020-dependency-graph-enhanced-dual-transformer/)).

Against this growing complexity, the deep averaging network made a deflationary point. It averages the word embeddings, applies a few nonlinear layers, and comes close to syntax-aware models while training in minutes ([Iyyer et al. 2015](https://md.research.elamur.ai/free/en/papers/iyyer-2015-deep-unordered-composition-rivals-syntactic/)).

### Fine-tuned transformers

BERT pretrains a bidirectional Transformer with two objectives: masked-token prediction and next-sentence prediction. For classification, the representation of [CLS], a special token added to the input for this purpose, is passed to a new output layer, and the whole network is fine-tuned ([Devlin et al. 2019](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)). Chi Sun and colleagues studied the choices this leaves open ([Sun et al. 2019](https://md.research.elamur.ai/free/en/papers/sun-2019-fine-tune-bert-text-classification/)). They recommend:

- first continuing pretraining on in-domain text, which reduced error by 18.57% on average relative to standard fine-tuning without that extra stage;
- optionally fine-tuning on several related tasks together;
- finally fine-tuning on the target task, with learning rates that decrease toward the lower layers.

Other methods build on the pretrained encoder. One example is MixText, which interpolates hidden states of pairs of texts for semi-supervised training ([Chen et al. 2020](https://md.research.elamur.ai/free/en/papers/chen-2020-mixtext-linguistically-informed-interpolation-hidden/)).

### Few-shot and zero-shot classification

Pattern-Exploiting Training (PET), from Timo Schick and Hinrich Schütze, rewrites each input as a cloze question. A verbalizer then maps every label to one vocabulary word, so the masked language model's own prediction head acts as the classifier ([Schick & Schütze 2021](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/)). PET fine-tunes one model per pattern on a handful of labelled examples. The ensemble then labels unlabelled text, and a final classifier is distilled from those soft labels ([Schick & Schütze 2021](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/)). This line of work belongs to [few-shot learning](https://md.research.elamur.ai/free/en/concepts/few-shot-learning/) and prompt engineering.

For the zero-shot case, Wenpeng Yin, Jamaal Hay and Dan Roth cast each candidate label as a hypothesis, such as "the text is about sports". They then asked a model trained on textual entailment whether the input supports it ([Yin et al. 2019](https://md.research.elamur.ai/free/en/papers/yin-2019-benchmarking-zero-shot-text-classification/)).

### Learning with few labels, other domains and other languages

Many methods target the label bottleneck without large pretrained models:

- EM with naive Bayes: on newsgroups, adding 10,000 unlabelled articles cut the labelled data needed for 70% accuracy from 2,000 articles to about 600 (Nigam et al. 2000).
- Transductive SVMs: on WebKB, the average precision/recall breakeven rose from 57.2 with inductive SVMs to 62.4 (Joachims 1999).
- Virtual adversarial training on word embeddings: it regularizes LSTM classifiers and exploits unlabelled text ([Miyato et al. 2016](https://md.research.elamur.ai/free/en/papers/miyato-2016-adversarial-training-methods-semi-supervised/)).
- Easy Data Augmentation (EDA): four simple word edits. Models trained with it on half the data matched baselines trained on all of it, although the benefit faded with pretrained models ([Wei & Zou 2019](https://md.research.elamur.ai/free/en/papers/wei-2019-eda-easy-data-augmentation-techniques/)).
- Distant supervision: emoticons served as labels for about 10 million tweets, used to learn sentiment-specific embeddings ([Tang et al. 2014](https://md.research.elamur.ai/free/en/papers/tang-2014-learning-sentiment-specific-word-embedding/)).

Domain shift is a related problem. A classifier trained on book reviews loses accuracy on kitchen appliances. Structural correspondence learning with mutual-information pivots addressed such shifts. On average it reduced the relative error they cause by 46% compared with a classifier without adaptation, and by 30% compared with the original method ([Blitzer et al. 2007](https://md.research.elamur.ai/free/en/papers/blitzer-2007-biographies-bollywood-boom-boxes-blenders/)). Adversarial multi-task learning keeps shared and task-private features apart when many small datasets are trained together ([Liu et al. 2017](https://md.research.elamur.ai/free/en/papers/liu-2017-adversarial-multi-task-learning-text/)).

Cross-lingual transfer, part of [multilingual natural language processing](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/), has two prominent approaches. One is co-training over machine-translated English and Chinese views of each review ([Wan 2009](https://md.research.elamur.ai/free/en/papers/wan-2009-co-training-cross-lingual-sentiment/)). The other is an adversarial averaging network that learns features hiding the input language ([Chen et al. 2018](https://md.research.elamur.ai/free/en/papers/chen-2018-adversarial-deep-averaging-networks-cross/)). Both help in [low-resource languages](https://md.research.elamur.ai/free/en/topics/low-resource-languages/).

### Efficiency

Linear models remain attractive where speed and memory matter. Product quantization of fastText's embedding tables shrinks models to well under a megabyte with little loss in accuracy ([Joulin et al. 2016](https://md.research.elamur.ai/free/en/papers/joulin-2016-fasttext-zip-compressing-text-classification/)). This connects text classification to work on [efficient models](https://md.research.elamur.ai/free/en/topics/efficient-models/).

## Datasets and benchmarks

### Collections

Shared collections defined much of the field's progress. They are covered more broadly under [evaluation and benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/).

| Collection | Task | Size or structure (as reported) |
| --- | --- | --- |
| Reuters-21578, ModApte split | Newswire topics, multi-label | 9,603 train / 3,299 test documents, 90 categories ([Joachims 1998](https://md.research.elamur.ai/free/en/papers/joachims-1998-text-categorization-support-vector-machines/)) |
| Ohsumed, 1991 abstracts | Medical topics | 10,000 train / 10,000 test, 23 disease categories ([Joachims 1998](https://md.research.elamur.ai/free/en/papers/joachims-1998-text-categorization-support-vector-machines/)) |
| RCV1 / RCV1-v2 | Newswire, hierarchical multi-label | Over 800,000 stories, Topic/Industry/Region codes ([Lewis et al. 2004](https://md.research.elamur.ai/free/en/papers/lewis-2004-rcv1-benchmark-collection-text-categorization/)) |
| Polarity corpus | Movie review sentiment | 2,053 reviews from 144 reviewers ([Pang et al. 2002](https://md.research.elamur.ai/free/en/papers/pang-2002-thumbs-up-sentiment-classification-machine/)) |
| Multi-domain Amazon reviews | Cross-domain sentiment | 4 domains, 1,000 positive and 1,000 negative each ([Blitzer et al. 2007](https://md.research.elamur.ai/free/en/papers/blitzer-2007-biographies-bollywood-boom-boxes-blenders/)) |
| AG News, DBPedia, Yelp, Yahoo! Answers, Amazon, Sogou | Large-scale topic and sentiment | Amazon Polarity has 1.8M training samples per class ([Zhang et al. 2015](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/)) |
| SemEval 2014 laptops and restaurants | Aspect-level sentiment | Three polarity classes ([Wang et al. 2016](https://md.research.elamur.ai/free/en/papers/wang-2016-attention-based-lstm-aspect-level/)) |
| Waseem and Hovy Twitter corpus | Racist/sexist/neither | 16,914 annotated tweets ([Waseem & Hovy 2016](https://md.research.elamur.ai/free/en/papers/waseem-2016-hateful-symbols-or-hateful-people/)) |

RCV1 was assembled because older test collections were small, poorly documented and overused, so that a whole community could overfit to them. Its authors also documented coding errors and released a cleaned version ([Lewis et al. 2004](https://md.research.elamur.ai/free/en/papers/lewis-2004-rcv1-benchmark-collection-text-categorization/)). Sentence-level suites became the standard testbed for short-text models ([Kim 2014](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/)):

- MR (movie-review sentiment);
- SST (the Stanford Sentiment Treebank);
- Subj (subjectivity);
- TREC (question type);
- CR (customer reviews);
- MPQA (opinion polarity).

The large datasets of Zhang, Zhao and LeCun became the standard for document-level models ([Conneau et al. 2017](https://md.research.elamur.ai/free/en/papers/conneau-2017-very-deep-convolutional-networks-text/)). For zero-shot evaluation, Yin and colleagues fixed splits of seen and unseen classes across topic, emotion and situation labels ([Yin et al. 2019](https://md.research.elamur.ai/free/en/papers/yin-2019-benchmarking-zero-shot-text-classification/)).

### Evaluation measures

For one class, every prediction falls into one of four cells. A true positive (TP) is a text correctly assigned to the class, and a false positive (FP) is a text wrongly assigned to it. A false negative (FN) is a member of the class that the classifier missed, and a true negative (TN) is a non-member correctly left out. The standard measures, in the textbook form popularized by van Rijsbergen (1979), are:

```
accuracy  = (TP + TN) / (TP + TN + FP + FN)
precision = TP / (FP + TP)
recall    = TP / (FN + TP)
F1        = 2 · precision · recall / (precision + recall)
```

Precision is the share of predicted members that are correct. Recall is the share of true members that are found. F1 is their harmonic mean, which is low whenever either of the two is low.

With many categories, the measures can be averaged in two ways. Microaveraging pools the counts of all categories before computing the measure, so frequent classes dominate the result. Macroaveraging computes the measure per class and then averages, so every class counts equally. Older work often reported the precision/recall breakeven point, where the two measures are equal ([Joachims 1998](https://md.research.elamur.ai/free/en/papers/joachims-1998-text-categorization-support-vector-machines/)). Twitter sentiment work commonly reports macro-F1 ([Tang et al. 2014](https://md.research.elamur.ai/free/en/papers/tang-2014-learning-sentiment-specific-word-embedding/)). Multi-label clinical coding adds ranking measures, such as the area under the receiver operating characteristic curve (AUC) and precision at 5 ([Wang et al. 2018](https://md.research.elamur.ai/free/en/papers/wang-2018-joint-embedding-words-labels-text/)).

### Class imbalance

Accuracy misleads when one class is rare, since always predicting the majority class already scores high. Several examples show how common this is:

- In the Reuters binary tasks of Nigam and colleagues, the positive class made up between 1% and 30% of documents (Nigam et al. 2000).
- In a lexicon-sampled Twitter corpus, only about 5% of tweets were majority-labelled as hate speech ([Davidson et al. 2017](https://md.research.elamur.ai/free/en/papers/davidson-2017-automated-hate-speech-detection-problem/)).
- On that same corpus, overall precision and recall were around 0.9, yet hate-class precision was only 0.44 ([Davidson et al. 2017](https://md.research.elamur.ai/free/en/papers/davidson-2017-automated-hate-speech-detection-problem/)).
- The racist class of the Waseem and Hovy corpus came from only 9 users ([Waseem & Hovy 2016](https://md.research.elamur.ai/free/en/papers/waseem-2016-hateful-symbols-or-hateful-people/)).

Per-class and macro-averaged scores, tuned decision thresholds ([Lewis et al. 2004](https://md.research.elamur.ai/free/en/papers/lewis-2004-rcv1-benchmark-collection-text-categorization/)) and deliberately balanced data construction ([Hartvigsen et al. 2022](https://md.research.elamur.ai/free/en/papers/hartvigsen-2022-toxigen-large-scale-machine-generated/)) are the usual responses.

### Label noise

Gold labels are themselves uncertain. Sebastiani notes that trained human indexers disagree with one another (Sebastiani 2002). In hate speech, who does the labelling matters. Unscreened crowd workers flagged hate more often than expert annotators did. In cross-validation within each label set, the best features reached about 91.2–91.5 F1 on expert labels, against about 86.3–86.5 on crowd labels ([Waseem 2016](https://md.research.elamur.ai/free/en/papers/waseem-2016-are-you-racist-or-am/)). Dialect can bias labels as well. African American English tweets were disproportionately marked offensive, and classifiers trained on those labels reproduced the disparity ([Sap et al. 2019](https://md.research.elamur.ai/free/en/papers/sap-2019-risk-racial-bias-hate-speech/)). Labels obtained by distant supervision, such as emoticons, are cheap but noisy by construction ([Tang et al. 2014](https://md.research.elamur.ai/free/en/papers/tang-2014-learning-sentiment-specific-word-embedding/)).

## People and institutions

[Bo Pang](https://md.research.elamur.ai/free/en/authors/bo-pang/) and [Lillian Lee](https://md.research.elamur.ai/free/en/authors/lillian-lee/), with Shivakumar Vaithyanathan, wrote the movie-review sentiment study described under History ([Pang et al. 2002](https://md.research.elamur.ai/free/en/papers/pang-2002-thumbs-up-sentiment-classification-machine/)). Lee is at [Cornell University](https://md.research.elamur.ai/free/en/universities/cornell-university/).

At [Harbin Institute of Technology](https://md.research.elamur.ai/free/en/universities/harbin-institute-of-technology/), [Ting Liu](https://md.research.elamur.ai/free/en/authors/ting-liu/) and [Bing Qin](https://md.research.elamur.ai/free/en/authors/bing-qin/) worked with [Duyu Tang](https://md.research.elamur.ai/free/en/authors/duyu-tang/) on a run of neural sentiment classifiers: sentiment-specific embeddings, gated document models, target-dependent LSTMs and deep memory networks ([Tang et al. 2014](https://md.research.elamur.ai/free/en/papers/tang-2014-learning-sentiment-specific-word-embedding/); [Tang et al. 2015](https://md.research.elamur.ai/free/en/papers/tang-2015-document-modeling-gated-recurrent-neural/); [Tang et al. 2016](https://md.research.elamur.ai/free/en/papers/tang-2016-effective-lstms-target-dependent-sentiment/); [Tang et al. 2016](https://md.research.elamur.ai/free/en/papers/tang-2016-aspect-level-sentiment-classification-deep/)). Tang also co-authored the adaptive recursive network with [Furu Wei](https://md.research.elamur.ai/free/en/authors/furu-wei/) and [Ming Zhou](https://md.research.elamur.ai/free/en/authors/ming-zhou/) ([Dong et al. 2014](https://md.research.elamur.ai/free/en/papers/dong-2014-adaptive-recursive-neural-network-target/)). Wei is at [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/).

[Yann LeCun](https://md.research.elamur.ai/free/en/authors/yann-lecun/) co-authored both the character-level ConvNet and its very deep successor ([Zhang et al. 2015](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/); [Conneau et al. 2017](https://md.research.elamur.ai/free/en/papers/conneau-2017-very-deep-convolutional-networks-text/)). [Zichao Yang](https://md.research.elamur.ai/free/en/authors/zichao-yang/) and [Diyi Yang](https://md.research.elamur.ai/free/en/authors/diyi-yang/) worked together on hierarchical attention networks and on MixText ([Yang et al. 2016](https://md.research.elamur.ai/free/en/papers/yang-2016-hierarchical-attention-networks-document-classification/); [Chen et al. 2020](https://md.research.elamur.ai/free/en/papers/chen-2020-mixtext-linguistically-informed-interpolation-hidden/)). Both were at [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) at the time of the hierarchical attention work, and Diyi Yang is now at [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/). [Xuanjing Huang](https://md.research.elamur.ai/free/en/authors/xuanjing-huang/) of [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) co-authored adversarial multi-task learning and the BERT fine-tuning study ([Liu et al. 2017](https://md.research.elamur.ai/free/en/papers/liu-2017-adversarial-multi-task-learning-text/); [Sun et al. 2019](https://md.research.elamur.ai/free/en/papers/sun-2019-fine-tune-bert-text-classification/)). [Lidong Bing](https://md.research.elamur.ai/free/en/authors/lidong-bing/) contributed transformation networks for target-oriented sentiment ([Li et al. 2018](https://md.research.elamur.ai/free/en/papers/li-2018-transformation-networks-target-oriented-sentiment/)).

[Timo Schick](https://md.research.elamur.ai/free/en/authors/timo-schick/) and [Hinrich Schütze](https://md.research.elamur.ai/free/en/authors/hinrich-schutze/) of Ludwig-Maximilians-Universität München developed PET ([Schick & Schütze 2021](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/)). [Claire Cardie](https://md.research.elamur.ai/free/en/authors/claire-cardie/) of Cornell co-authored adversarial cross-lingual sentiment classification ([Chen et al. 2018](https://md.research.elamur.ai/free/en/papers/chen-2018-adversarial-deep-averaging-networks-cross/)).

On abusive language, [Zeerak Waseem](https://md.research.elamur.ai/free/en/authors/zeerak-waseem/) built the early Twitter hate speech corpora and studied how annotators shape them ([Waseem & Hovy 2016](https://md.research.elamur.ai/free/en/papers/waseem-2016-hateful-symbols-or-hateful-people/); [Waseem 2016](https://md.research.elamur.ai/free/en/papers/waseem-2016-are-you-racist-or-am/)). [Maarten Sap](https://md.research.elamur.ai/free/en/authors/maarten-sap/) and [Noah A. Smith](https://md.research.elamur.ai/free/en/authors/noah-a-smith/) exposed dialect bias in toxicity labels ([Sap et al. 2019](https://md.research.elamur.ai/free/en/papers/sap-2019-risk-racial-bias-hate-speech/)). Smith is at the [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/). Sap later co-authored the ToxiGen dataset ([Hartvigsen et al. 2022](https://md.research.elamur.ai/free/en/papers/hartvigsen-2022-toxigen-large-scale-machine-generated/)).

Earlier foundations came from several researchers:

- Thorsten Joachims on SVMs ([Joachims 1998](https://md.research.elamur.ai/free/en/papers/joachims-1998-text-categorization-support-vector-machines/));
- Andrew McCallum and Kamal Nigam on naive Bayes and EM (McCallum & Nigam 1998; Nigam et al. 2000);
- Fabrizio Sebastiani's survey (Sebastiani 2002);
- David D. Lewis and Yiming Yang on RCV1 ([Lewis et al. 2004](https://md.research.elamur.ai/free/en/papers/lewis-2004-rcv1-benchmark-collection-text-categorization/)).

Authors with the most papers on this subject Ting Liu leads with 6 papers; the 12 authors shown wrote 39 of the papers on this subject between them, counting each paper once per author.

**The numbers**

| Author | Papers here · their citations |
| --- | --- |
| [Ting Liu](https://md.research.elamur.ai/free/en/authors/ting-liu/) | 6 papers · 5,110 citations |
| [Duyu Tang](https://md.research.elamur.ai/free/en/authors/duyu-tang/) | 5 papers · 6,137 citations |
| [Bing Qin](https://md.research.elamur.ai/free/en/authors/bing-qin/) | 4 papers · 4,921 citations |
| [Zeerak Waseem](https://md.research.elamur.ai/free/en/authors/zeerak-waseem/) | 3 papers · 3,137 citations |
| [Lidong Bing](https://md.research.elamur.ai/free/en/authors/lidong-bing/) | 3 papers · 2,219 citations |
| [Mohit Iyyer](https://md.research.elamur.ai/free/en/authors/mohit-iyyer/) | 3 papers · 1,899 citations |
| [Noah A. Smith](https://md.research.elamur.ai/free/en/authors/noah-a-smith/) | 3 papers · 1,588 citations |
| [Xuanjing Huang](https://md.research.elamur.ai/free/en/authors/xuanjing-huang/) | 3 papers · 1,310 citations |
| [Claire Cardie](https://md.research.elamur.ai/free/en/authors/claire-cardie/) | 3 papers · 605 citations |
| [Bo Pang](https://md.research.elamur.ai/free/en/authors/bo-pang/) | 2 papers · 10,495 citations |
| [Lillian Lee](https://md.research.elamur.ai/free/en/authors/lillian-lee/) | 2 papers · 10,495 citations |
| [Yann LeCun](https://md.research.elamur.ai/free/en/authors/yann-lecun/) | 2 papers · 5,896 citations |

Where the authors on this subject work now Alibaba Group is the current home of authors of 10 papers on this subject; counted are the authors with a page here, by their latest institution.

**The numbers**

| Institution | Papers |
| --- | --- |
| [Alibaba Group](https://md.research.elamur.ai/free/en/companies/alibaba-group/) | 10 papers |
| [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) | 10 papers |
| [Harbin Institute of Technology](https://md.research.elamur.ai/free/en/universities/harbin-institute-of-technology/) | 10 papers |
| [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) | 9 papers |
| [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) | 9 papers |
| [University of Illinois Urbana-Champaign](https://md.research.elamur.ai/free/en/universities/university-of-illinois-urbana-champaign/) | 9 papers |
| [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) | 8 papers |
| [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) | 7 papers |
| [Google](https://md.research.elamur.ai/free/en/companies/google/) | 7 papers |
| [Northeastern University](https://md.research.elamur.ai/free/en/universities/northeastern-university/) | 7 papers |

Countries of the authors on this subject United States comes first: authors working there now wrote 63 papers on this subject.

**The numbers**

| Country | Papers |
| --- | --- |
| [United States](https://md.research.elamur.ai/free/en/countries/united-states/) | 63 papers |
| [China](https://md.research.elamur.ai/free/en/countries/china/) | 49 papers |
| [United Kingdom](https://md.research.elamur.ai/free/en/countries/united-kingdom/) | 16 papers |
| [Germany](https://md.research.elamur.ai/free/en/countries/germany/) | 12 papers |
| [Singapore](https://md.research.elamur.ai/free/en/countries/singapore/) | 12 papers |
| Hong Kong | 10 papers |
| [Canada](https://md.research.elamur.ai/free/en/countries/canada/) | 7 papers |
| [Italy](https://md.research.elamur.ai/free/en/countries/italy/) | 7 papers |
| [Australia](https://md.research.elamur.ai/free/en/countries/australia/) | 6 papers |
| [Denmark](https://md.research.elamur.ai/free/en/countries/denmark/) | 5 papers |

Where the papers on this subject were published Annual Meeting of the Association for Computational Linguistics (ACL) carries the most, 27 of 154.

**The numbers**

| Venue | Papers |
| --- | --- |
| [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/) | 27 papers |
| [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) | 24 papers |
| [Language Resources and Evaluation Conference (LREC)](https://md.research.elamur.ai/free/en/venues/lrec/) | 11 papers |
| [Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)](https://md.research.elamur.ai/free/en/venues/naacl/) | 10 papers |
| [International Conference on Computational Linguistics (COLING)](https://md.research.elamur.ai/free/en/venues/coling/) | 9 papers |
| [Arabic Natural Language Processing Workshop (WANLP)](https://md.research.elamur.ai/free/en/venues/wanlp/) | 7 papers |
| [Conference of the European Chapter of the Association for Computational Linguistics (EACL)](https://md.research.elamur.ai/free/en/venues/eacl/) | 3 papers |
| [Findings of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/findings/) | 3 papers |
| [International Joint Conference on Natural Language Processing (IJCNLP)](https://md.research.elamur.ai/free/en/venues/ijcnlp/) | 3 papers |
| [Proceedings of the AAAI Conference on Artificial Intelligence](https://md.research.elamur.ai/free/en/venues/proceedings-of-the-aaai-conference-on-artificial-intelligence/) | 3 papers |

## Open questions

### Robustness

High benchmark scores do not mean a classifier is robust. TextFooler swaps influential words for synonyms. It pushed most target models, BERT included, below 10% accuracy while changing under a fifth of the words ([Jin et al. 2019](https://md.research.elamur.ai/free/en/papers/jin-2019-bert-really-robust-strong-baseline/)). BAE (BERT-based Adversarial Examples) uses a masked language model to propose substitutions and insertions that fit the context. Its outputs were judged more natural than TextFooler's and remained effective ([Garg & Ramakrishnan 2020](https://md.research.elamur.ai/free/en/papers/garg-2020-bae-bert-based-adversarial-examples/)). Adversarial retraining helped only partly ([Jin et al. 2019](https://md.research.elamur.ai/free/en/papers/jin-2019-bert-really-robust-strong-baseline/)). How to build classifiers whose decisions hold up under meaning-preserving edits remains open.

### Fairness and what a label means

For socially sensitive labels, the text alone may not settle the answer. Dialect and speaker identity change how annotators judge offensiveness ([Sap et al. 2019](https://md.research.elamur.ai/free/en/papers/sap-2019-risk-racial-bias-hate-speech/)). Merely offensive language is easily confused with hate speech ([Davidson et al. 2017](https://md.research.elamur.ai/free/en/papers/davidson-2017-automated-hate-speech-detection-problem/)). Classifiers often treat any mention of a minority group as a toxic signal ([Hartvigsen et al. 2022](https://md.research.elamur.ai/free/en/papers/hartvigsen-2022-toxigen-large-scale-machine-generated/)). A shared definition and comparable datasets are still missing in this area ([Schmidt & Wiegand 2017](https://md.research.elamur.ai/free/en/papers/schmidt-2017-survey-hate-speech-detection-natural/)). These questions overlap with work on [bias, fairness and safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/).

### Scarce labels and domain shift

PET ([Schick & Schütze 2021](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/)) and entailment-based zero-shot classification ([Yin et al. 2019](https://md.research.elamur.ai/free/en/papers/yin-2019-benchmarking-zero-shot-text-classification/)) reduce the need for annotation. Their sensitivity to the wording of patterns and label descriptions is not settled. Neither is the question of when a small fine-tuned model is still the better choice. Domain shift persists even with pretraining. Continued in-domain pretraining still brought large gains in one controlled study ([Sun et al. 2019](https://md.research.elamur.ai/free/en/papers/sun-2019-fine-tune-bert-text-classification/)).

### Interpretability and the value of baselines

Attention weights are offered as explanations ([Yang et al. 2016](https://md.research.elamur.ai/free/en/papers/yang-2016-hierarchical-attention-networks-document-classification/); [Wang et al. 2018](https://md.research.elamur.ai/free/en/papers/wang-2018-joint-embedding-words-labels-text/)). Whether they reflect how the model actually decides belongs to the wider study of [interpretability](https://md.research.elamur.ai/free/en/topics/interpretability/).

Finally, the field keeps rediscovering that simple models are hard to beat:

- TF-IDF n-grams on medium-sized data ([Zhang et al. 2015](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/));
- NBSVM on sentiment ([Wang & Manning 2012](https://md.research.elamur.ai/free/en/papers/wang-2012-baselines-bigrams-simple-good-sentiment/));
- averaged embeddings with a few nonlinear layers ([Iyyer et al. 2015](https://md.research.elamur.ai/free/en/papers/iyyer-2015-deep-unordered-composition-rivals-syntactic/)).

Each new architecture is therefore expected to show a clear margin over well-tuned baselines.

## The library in figures

Neighbouring subjects: papers shared Natural language processing shares the most: 112 of the 154 papers here (73%) stand on both.

**The numbers**

| Shelf | Papers shared |
| --- | --- |
| [Natural language processing](https://md.research.elamur.ai/free/en/concepts/natural-language-processing/) | 112 shared · 73% |
| [Few-shot learning](https://md.research.elamur.ai/free/en/concepts/few-shot-learning/) | 18 shared · 12% |
| [Large language models](https://md.research.elamur.ai/free/en/concepts/large-language-models/) | 12 shared · 8% |
| [Machine learning](https://md.research.elamur.ai/free/en/concepts/machine-learning/) | 11 shared · 7% |
| [Morphology](https://md.research.elamur.ai/free/en/concepts/morphology/) | 5 shared · 3% |
| [Natural language understanding](https://md.research.elamur.ai/free/en/concepts/natural-language-understanding/) | 4 shared · 3% |
| [Question answering](https://md.research.elamur.ai/free/en/concepts/question-answering/) | 3 shared · 2% |
| [Parsing](https://md.research.elamur.ai/free/en/concepts/parsing/) | 2 shared · 1% |
| [Chatbots](https://md.research.elamur.ai/free/en/concepts/chatbot/) | 1 shared · 1% |
| [Information extraction](https://md.research.elamur.ai/free/en/concepts/information-extraction/) | 1 shared · 1% |

## Key papers

The works this page cites, the most cited first; citations as counted by OpenAlex plus the papers of this library that cite the work, read 2026-10-24.

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/) — Jacob Devlin, Ming-Wei Chang, Kenton Lee et al. · 2019 · 55,571 citations
- [Convolutional Neural Networks for Sentence Classification](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/) — Yoon Kim · 2014 · 16,269 citations
- [Text Categorization with Support Vector Machines: Learning with Many Relevant Features](https://md.research.elamur.ai/free/en/papers/joachims-1998-text-categorization-support-vector-machines/) — Thorsten Joachims · 1998 · 8,311 citations
- [Thumbs up? Sentiment Classification using Machine Learning Techniques](https://md.research.elamur.ai/free/en/papers/pang-2002-thumbs-up-sentiment-classification-machine/) — Bo Pang, Lillian Lee, Shivakumar Vaithyanathan · 2002 · 7,762 citations
- [Hierarchical Attention Networks for Document Classification](https://md.research.elamur.ai/free/en/papers/yang-2016-hierarchical-attention-networks-document-classification/) — Zichao Yang, Diyi Yang, Chris Dyer et al. · 2016 · 5,423 citations
- [Character-level Convolutional Networks for Text Classification](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/) — Xiang Zhang, Junbo Zhao, Yann LeCun · 2015 · 4,781 citations
- [RCV1: A New Benchmark Collection for Text Categorization Research](https://md.research.elamur.ai/free/en/papers/lewis-2004-rcv1-benchmark-collection-text-categorization/) — David D. Lewis, Yiming Yang, Tony G. Rose et al. · 2004 · 2,752 citations
- [Attention-based LSTM for Aspect-level Sentiment Classification](https://md.research.elamur.ai/free/en/papers/wang-2016-attention-based-lstm-aspect-level/) — Yequan Wang, Minlie Huang, Xiaoyan Zhu et al. · 2016 · 2,619 citations
- [Biographies, Bollywood, Boom-boxes and Blenders: Domain Adaptation for Sentiment Classification](https://md.research.elamur.ai/free/en/papers/blitzer-2007-biographies-bollywood-boom-boxes-blenders/) — John Blitzer, Mark Dredze, Fernando Pereira · 2007 · 2,329 citations
- [Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter](https://md.research.elamur.ai/free/en/papers/waseem-2016-hateful-symbols-or-hateful-people/) — Zeerak Waseem, Dirk Hovy · 2016 · 2,167 citations
- [A Survey on Hate Speech Detection using Natural Language Processing](https://md.research.elamur.ai/free/en/papers/schmidt-2017-survey-hate-speech-detection-natural/) — Anna Schmidt, Michael Wiegand · 2017 · 1,805 citations
- [Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/) — Timo Schick, Hinrich Schütze · 2021 · 1,803 citations
- [Document Modeling with Gated Recurrent Neural Network for Sentiment Classification](https://md.research.elamur.ai/free/en/papers/tang-2015-document-modeling-gated-recurrent-neural/) — Duyu Tang, Bing Qin, Ting Liu · 2015 · 1,684 citations
- [Learning Sentiment-Specific Word Embedding for Twitter Sentiment Classification](https://md.research.elamur.ai/free/en/papers/tang-2014-learning-sentiment-specific-word-embedding/) — Duyu Tang, Furu Wei, Nan Yang et al. · 2014 · 1,346 citations
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://md.research.elamur.ai/free/en/papers/jin-2019-bert-really-robust-strong-baseline/) — Di Jin, Zhijing Jin, Joey Tianyi Zhou et al. · 2019 · 1,326 citations
- [Adaptive Recursive Neural Network for Target-dependent Twitter Sentiment Classification](https://md.research.elamur.ai/free/en/papers/dong-2014-adaptive-recursive-neural-network-target/) — Li Dong, Furu Wei, Chuanqi Tan et al. · 2014 · 1,216 citations
- [FastText.zip: Compressing text classification models](https://md.research.elamur.ai/free/en/papers/joulin-2016-fasttext-zip-compressing-text-classification/) — Armand Joulin, Edouard Grave, Piotr Bojanowski et al. · 2016 · 1,214 citations
- [Aspect Level Sentiment Classification with Deep Memory Network](https://md.research.elamur.ai/free/en/papers/tang-2016-aspect-level-sentiment-classification-deep/) — Duyu Tang, Bing Qin, Ting Liu · 2016 · 1,194 citations
- [The Risk of Racial Bias in Hate Speech Detection](https://md.research.elamur.ai/free/en/papers/sap-2019-risk-racial-bias-hate-speech/) — Maarten Sap, Dallas Card, Saadia Gabriel et al. · 2019 · 1,152 citations
- [Baselines and Bigrams: Simple, Good Sentiment and Topic Classification](https://md.research.elamur.ai/free/en/papers/wang-2012-baselines-bigrams-simple-good-sentiment/) — Sida Wang, Christopher Manning · 2012 · 1,127 citations
- [Very Deep Convolutional Networks for Text Classification](https://md.research.elamur.ai/free/en/papers/conneau-2017-very-deep-convolutional-networks-text/) — Alexis Conneau, Holger Schwenk, Loïc Barrault et al. · 2017 · 1,115 citations
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://md.research.elamur.ai/free/en/papers/davidson-2017-automated-hate-speech-detection-problem/) — Thomas Davidson, Dana Warmsley, Michael Macy et al. · 2017 · 1,015 citations
- [Effective Use of Word Order for Text Categorization with Convolutional Neural Networks](https://md.research.elamur.ai/free/en/papers/johnson-2015-effective-use-word-order-text/) — Rie Johnson, Tong Zhang · 2015 · 970 citations
- [Deep Unordered Composition Rivals Syntactic Methods for Text Classification](https://md.research.elamur.ai/free/en/papers/iyyer-2015-deep-unordered-composition-rivals-syntactic/) — Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber et al. · 2015 · 920 citations
- [Aspect-based Sentiment Classification with Aspect-specific Graph Convolutional Networks](https://md.research.elamur.ai/free/en/papers/zhang-2019-aspect-based-sentiment-classification-aspect/) — Chen Zhang, Qiuchi Li, Dawei Song · 2019 · 808 citations
- [Are You a Racist or Am I Seeing Things? Annotator Influence on Hate Speech Detection on Twitter](https://md.research.elamur.ai/free/en/papers/waseem-2016-are-you-racist-or-am/) — Zeerak Waseem · 2016 · 767 citations
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://md.research.elamur.ai/free/en/papers/wei-2019-eda-easy-data-augmentation-techniques/) — Jason Wei, Kai Zou · 2019 · 755 citations
- [Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach](https://md.research.elamur.ai/free/en/papers/yin-2019-benchmarking-zero-shot-text-classification/) — Wenpeng Yin, Jamaal Hay, Dan Roth · 2019 · 731 citations
- [Adversarial Multi-task Learning for Text Classification](https://md.research.elamur.ai/free/en/papers/liu-2017-adversarial-multi-task-learning-text/) — Pengfei Liu, Xipeng Qiu, Xuanjing Huang · 2017 · 725 citations
- [Effective LSTMs for Target-Dependent Sentiment Classification](https://md.research.elamur.ai/free/en/papers/tang-2016-effective-lstms-target-dependent-sentiment/) — Duyu Tang, Bing Qin, Xiaocheng Feng et al. · 2016 · 697 citations
- [BAE: BERT-based Adversarial Examples for Text Classification](https://md.research.elamur.ai/free/en/papers/garg-2020-bae-bert-based-adversarial-examples/) — Siddhant Garg, Goutham Ramakrishnan · 2020 · 665 citations
- [Transformation Networks for Target-Oriented Sentiment Classification](https://md.research.elamur.ai/free/en/papers/li-2018-transformation-networks-target-oriented-sentiment/) — Xin Li, Lidong Bing, Wai Lam et al. · 2018 · 597 citations
- [Co-Training for Cross-Lingual Sentiment Classification](https://md.research.elamur.ai/free/en/papers/wan-2009-co-training-cross-lingual-sentiment/) — Xiaojun Wan · 2009 · 595 citations
- [Multi-grained Attention Network for Aspect-Level Sentiment Classification](https://md.research.elamur.ai/free/en/papers/fan-2018-multi-grained-attention-network-aspect/) — Feifan Fan, Yansong Feng, Dongyan Zhao · 2018 · 573 citations
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://md.research.elamur.ai/free/en/papers/hartvigsen-2022-toxigen-large-scale-machine-generated/) — Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi et al. · 2022 · 571 citations
- [Sequential Short-Text Classification with Recurrent and Convolutional Neural Networks](https://md.research.elamur.ai/free/en/papers/lee-2016-sequential-short-text-classification-recurrent/) — Ji Young Lee, Franck Dernoncourt · 2016 · 512 citations
- [Joint Embedding of Words and Labels for Text Classification](https://md.research.elamur.ai/free/en/papers/wang-2018-joint-embedding-words-labels-text/) — Guoyin Wang, Chunyuan Li, Wenlin Wang et al. · 2018 · 464 citations
- [Adversarial Training Methods for Semi-Supervised Text Classification](https://md.research.elamur.ai/free/en/papers/miyato-2016-adversarial-training-methods-semi-supervised/) — Takeru Miyato, Andrew M. Dai, Ian Goodfellow · 2016 · 427 citations
- [MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification](https://md.research.elamur.ai/free/en/papers/chen-2020-mixtext-linguistically-informed-interpolation-hidden/) — Jiaao Chen, Zichao Yang, Diyi Yang · 2020 · 413 citations
- [How to Fine-Tune BERT for Text Classification?](https://md.research.elamur.ai/free/en/papers/sun-2019-fine-tune-bert-text-classification/) — Chi Sun, Xipeng Qiu, Yige Xu et al. · 2019 · 404 citations
- [Dependency Graph Enhanced Dual-transformer Structure for Aspect-based Sentiment Classification](https://md.research.elamur.ai/free/en/papers/tang-2020-dependency-graph-enhanced-dual-transformer/) — Hao Tang, Donghong Ji, Chenliang Li et al. · 2020 · 397 citations
- [Adversarial Deep Averaging Networks for Cross-Lingual Sentiment Classification](https://md.research.elamur.ai/free/en/papers/chen-2018-adversarial-deep-averaging-networks-cross/) — Xilun Chen, Yu Sun, Ben Athiwaratkun et al. · 2018 · 393 citations

## Related concepts

- [Few-shot learning](https://md.research.elamur.ai/free/en/concepts/few-shot-learning/) — Few-shot learning (FSL) is the study and practice of building machine-learning systems that can perform a new task, such as recognising a…
- [Topic modeling](https://md.research.elamur.ai/free/en/concepts/topic-modeling/) — Topic modeling is a family of mostly unsupervised methods that discover recurring themes in a document collection. Each theme, or topic, is…
- [Named entity recognition](https://md.research.elamur.ai/free/en/concepts/named-entity-recognition/) — Named entity recognition locates stretches of text that refer to specific real-world things, such as people, organizations, places…

## More papers

Topics: [😊 Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) · [🛡️ Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/)

Shelves: [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/)

Updated 24 October 2026

- [← previous concept SYM: symbol](https://md.research.elamur.ai/free/en/concepts/upos-sym/)
- [next concept → Text summarization](https://md.research.elamur.ai/free/en/concepts/text-summarization/)
- [↑ back to concepts](https://md.research.elamur.ai/free/en/concepts/)