---
title: "A Dataset and Classifier for Recognizing Social Media English"
description: "First, they build an evaluation resource: 10,502 geotagged public tweets drawn uniformly… 16.5 points. 14 references, cited by 10 papers here."
authors: ["Su Lin Blodgett", "Johnny Wei", "Brendan O'Connor"]
year: 2017
venue: "Proceedings of the 3rd Workshop on Noisy User-generated Text"
source: "https://aclanthology.org/W17-4408/"
doi: "10.18653/v1/w17-4408"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/blodgett-2017-dataset-classifier-recognizing-social-media/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/blodgett-2017-dataset-classifier-recognizing-social-media/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/4/) / A Dataset and Classifier for Recognizing Social Media English

# A Dataset and Classifier for Recognizing Social Media English

**Authors:** [Su Lin Blodgett](https://md.research.elamur.ai/free/en/authors/su-lin-blodgett/), [Johnny Wei](https://md.research.elamur.ai/free/en/authors/johnny-wei/) and [Brendan O'Connor](https://md.research.elamur.ai/free/en/authors/brendan-o-connor/) · **Year:** 2017 · **Venue:** [Proceedings of the 3rd Workshop on Noisy User-generated Text](https://md.research.elamur.ai/free/en/venues/wnut/) ([Workshop on Noisy User-generated Text 2017](https://md.research.elamur.ai/free/en/venues/wnut/2017/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/W17-4408/) · [PDF](https://aclanthology.org/W17-4408.pdf) · [DOI 10.18653/v1/w17-4408](https://doi.org/10.18653/v1/w17-4408) · [OpenAlex](https://openalex.org/W2757188127)

**Citations:** 26 citations (OpenAlex, 2026-10-14) · [10](https://md.research.elamur.ai/free/en/papers/blodgett-2017-dataset-classifier-recognizing-social-media/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Off-the-shelf language identifiers do well on edited text but often miss English on Twitter. Short posts, playful spellings, slang and dialects such as African-American English (AAE) get labelled as some other language. The authors address this in two ways. First, they build an evaluation resource: 10,502 geotagged public tweets drawn uniformly at random, so no earlier classifier shaped the sample. Humans labelled each tweet as English, Not English or Ambiguous, and also marked code-switching, named-entity-only content and machine-generated posts. Second, they take the demographic mixed-membership language model of Blodgett et al. (2016), trained on U.S. tweets, and use it as a recall booster. It can override a base identifier's non-English decision when the probability mass on the U.S. English-associated topics is high. This boost improves recall for CLD2, Twitter's internal identifier, langid.py and a neural model, with almost no precision cost. It beats self-training and helps most on very short messages and on English from many non-U.S. countries. The dataset and ensemble are released to support fairer handling of non-standard English.

## Key points
- The dataset has 10,502 tweets (5,086 English, 4,646 Not English, 770 Ambiguous). The evaluation subset of 8,366 tweets drops ambiguous and automatically generated posts. Extra tags cover 162 code-switched tweets, 132 ambiguous because of named entities, and 1,371 auto-generated tweets.
- The ensemble rule works as follows. If the base classifier says English, the label is kept. Otherwise the tweet becomes English when the African-American, Hispanic and white topics of the demographic model together hold at least 90% posterior. Adding this rule lowered precision by at most 0.4%.
- The rule raised recall in every setting:
  - CLD2: 0.863 → 0.924
  - langid.py trained on Twitter70 (74,259 recoverable tweets of Twitter's 120,575-tweet set): 0.940 → 0.957
  - the hierarchical neural model trained on Wikipedia fragments: 0.415 → 0.773
- Self-training langid.py on 2.2M confidently auto-labelled tweets barely moved recall (0.886 → 0.894). Retraining it on in-domain Twitter70 data gave a 5.4-point gain.
- For English tweets, langid.py's recall gap between the shortest (≤5 words) and longest messages was 16.5 points. With the ensemble the gap shrank to 5.6 points. Recall on the shortest English tweets rose from 80.7% to 91.9%, and recall improved in 10 of the 15 countries analysed.

## Interesting ideas
- Build the evaluation set by uniform random sampling rather than from items an existing tool already flagged, so the test cannot inherit that tool's blind spots.
- Tag hard cases (ambiguous, code-switched, machine-generated) as separate descriptive labels instead of discarding them, then report scores on a clean subset.
- Use a one-directional override that can only flip a decision one way, and only above a high threshold. This trades a little precision for large recall gains in a controlled, explainable manner.
- Inspect high-confidence pseudo-labels before trusting self-training. Here even very confident "non-English" outputs were mostly English, which explains why self-training failed.
- Slice evaluation by message length and by author origin to expose where a system silently underperforms.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): sample the gold or negative-control sets uniformly from raw text instead of from sentences the witness parsers already handle, so failures on informal English stay visible.
- **I2** (learn): mark silver sentences as boilerplate, mixed-language or entity-only, and evaluate with and without them, so parser scores are not diluted or inflated by text that has no real syntax.
- **I3** (tokens): a fallback lexicon or model for social spellings (elongated words, abbreviations like "fr") could override "unknown word" tagger decisions only when confident, leaving standard text untouched.
- **I4** (learn): before adding agreed silver trees to training, hand-check a sample of the most confident ones from out-of-domain text, since all the witness parsers may share the same domain bias.
- **I5** (tree): report LAS in bins of sentence length and by source genre or dialect, to see whether gparse and the perceptron parser degrade on short, fragmentary or non-standard input.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Massachusetts Amherst](https://md.research.elamur.ai/free/en/universities/university-of-massachusetts-amherst/) — University, Amherst Center, United States · 106 works · 66,692 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works

## Cite

Cite as: Su Lin Blodgett, Johnny Wei and Brendan O'Connor. 2017. A Dataset and Classifier for Recognizing Social Media English. Proceedings of the 3rd Workshop on Noisy User-generated Text.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{blodgett2017dataset,
  title = {{A Dataset and Classifier for Recognizing Social Media English}},
  author = {Su Lin Blodgett, Johnny Wei and Brendan O'Connor},
  booktitle = {Proceedings of the 3rd Workshop on Noisy User-generated Text},
  year = {2017},
  doi = {10.18653/v1/w17-4408},
  url = {https://doi.org/10.18653/v1/w17-4408}
}
```

- [← previous No case for Case in locality: Case does not help interpretation when intervention blocks A-bar chains](https://md.research.elamur.ai/free/en/papers/friedmann-2017-no-case-case-locality-case/)
- [next → SiTAKA at SemEval-2017 Task 4: Sentiment Analysis in Twitter Based on a Rich Set of Features](https://md.research.elamur.ai/free/en/papers/jabreel-2017-sitaka-semeval-2017-task-4/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/4/)

## References

14 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jennifer Williams and Charlie Dagli. 2017. Twitter language identification of similar languages and dialects without ground truth. In Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). ACL. [find](https://scholar.google.com/scholar?q=Jennifer+Williams+and+Charlie+Dagli%2E+2017%2E+Twitter+language+identification+of+similar+languages+and+dialects+without+ground+truth%2E+In+Proceedings+of+the+Fourth+Workshop+on+NLP+for+Similar+Languages%2C+V)
2. Shane Bergsma, Paul McNamee, Mossaab Bagdouri, Clayton Fink, and Theresa Wilson. 2012. Language identification for creating language-specific twitter collections. In Proceedings of the second workshop on language in social media, pages 65–74. ACL. [find](https://scholar.google.com/scholar?q=Shane+Bergsma%2C+Paul+McNamee%2C+Mossaab+Bagdouri%2C+Clayton+Fink%2C+and+Theresa+Wilson%2E+2012%2E+Language+identification+for+creating+language%2Dspecific+twitter+collections%2E+In+Proceedings+of+the+second+workshop)
3. Yi Yang and Jacob Eisenstein. 2016. [Part-of-speech tagging for historical English](https://md.research.elamur.ai/free/en/papers/yang-2016-part-speech-tagging-historical-english/). In Proceedings of NAACL-HLT, San Diego, California. ACL.
4. John Blitzer, Mark Dredze, Fernando Pereira, et al. 2007. [Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification](https://md.research.elamur.ai/free/en/papers/blitzer-2007-biographies-bollywood-boom-boxes-blenders/). In ACL, volume 7, pages 440–447.
5. John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. [Domain adaptation with structural correspondence learning](https://md.research.elamur.ai/free/en/papers/blitzer-2006-domain-adaptation-structural-correspondence-learning/). In Proceedings of EMNLP. ACL.
6. Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016. [Demographic dialectal variation in social media: A case study of African-American English](https://md.research.elamur.ai/free/en/papers/blodgett-2016-demographic-dialectal-variation-social-media/). In Proceedings of EMNLP, Austin, Texas. ACL.
7. Baden Hughes, Timothy Baldwin, Steven Bird, Jeremy Nicholson, and Andrew MacKinlay. 2006. Reconsidering language identification for written language resources. In Proceedings of LREC. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Baden+Hughes%2C+Timothy+Baldwin%2C+Steven+Bird%2C+Jeremy+Nicholson%2C+and+Andrew+MacKinlay%2E+2006%2E+Reconsidering+language+identification+for+written+language+resources%2E+In+Proceedings+of+LREC%2E+European+Languag)
8. Aaron Jaech, George Mulcaire, Shobhit Hathi, Mari Ostendorf, and Noah A. Smith. 2016. Hierarchical character-word models for language identification. In Proceedings of EMNLP, Austin, TX, USA. ACL. [find](https://scholar.google.com/scholar?q=Aaron+Jaech%2C+George+Mulcaire%2C+Shobhit+Hathi%2C+Mari+Ostendorf%2C+and+Noah+A%2E+Smith%2E+2016%2E+Hierarchical+character%2Dword+models+for+language+identification%2E+In+Proceedings+of+EMNLP%2C+Austin%2C+TX%2C+USA%2E+ACL%2E)
9. Ben King and Steven P Abney. 2013. Labeling the languages of words in mixed-language documents using weakly supervised methods. In Proceedings of NAACL-HLT. [find](https://scholar.google.com/scholar?q=Ben+King+and+Steven+P+Abney%2E+2013%2E+Labeling+the+languages+of+words+in+mixed%2Dlanguage+documents+using+weakly+supervised+methods%2E+In+Proceedings+of+NAACL%2DHLT%2E)
10. Tom Kocmi and Ondřej Bojar. 2017. Lanidenn: Multilingual language identification on character window. To appear in Proceedings of EACL. [↗](https://doi.org/10.48550/arxiv.1701.03338)
11. Marco Lui and T. Baldwin. 2012. langid. py: An off-the-shelf language identification tool. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL 2012), Demo Session, Jeju, Republic of Korea. — [langid.py: An Off-the-shelf Language Identification Tool](https://md.research.elamur.ai/free/en/papers/lui-2012-langid-py-off-shelf-language/)
12. Marco Lui and Timothy Baldwin. 2014. Accurate language identification of twitter messages. In Proceedings of the 5th workshop on language analysis for social media (LASM)@ EACL, pages 17–25. [find](https://scholar.google.com/scholar?q=Marco+Lui+and+Timothy+Baldwin%2E+2014%2E+Accurate+language+identification+of+twitter+messages%2E+In+Proceedings+of+the+5th+workshop+on+language+analysis+for+social+media+%28LASM%29%40+EACL%2C+pages+17%E2%80%9325%2E)
13. Marco Lui and Paul Cook. 2013. [Classifying english documents by national dialect](https://md.research.elamur.ai/free/en/papers/lui-2013-classifying-english-documents-national-dialect/). In Proceedings of the Australasian Language Technology Association Workshop (ALTA).
14. Barbara Plank. 2009. A comparison of structural correspondence learning and self-training for discriminative parse selection. In Proceedings of the NAACL HLT Workshop on Semi-supervised Learning for NLP. ACL. [find](https://scholar.google.com/scholar?q=Barbara+Plank%2E+2009%2E+A+comparison+of+structural+correspondence+learning+and+self%2Dtraining+for+discriminative+parse+selection%2E+In+Proceedings+of+the+NAACL+HLT+Workshop+on+Semi%2Dsupervised+Learning+for+N)

*Updated 1 November 2026*