---
title: "The State and Fate of Linguistic Diversity and — Joshi 2020"
description: "Although the world has more than 7,000 languages, only a small share benefit from modern NLP, and many systems described as \"language agnostic\" have only…"
authors: ["Pratik Joshi", "Sebastin Santy", "Amar Budhiraja", "Kalika Bali", "Monojit Choudhury"]
year: 2020
venue: "arXiv:2004.09095"
source: "https://arxiv.org/abs/2004.09095"
doi: "10.48550/arxiv.2004.09095"
arxiv: "2004.09095"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/joshi-2020-state-fate-linguistic-diversity-inclusion/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/joshi-2020-state-fate-linguistic-diversity-inclusion/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/) / The State and Fate of Linguistic Diversity and Inclusion in the NLP World

# The State and Fate of Linguistic Diversity and Inclusion in the NLP World

**Authors:** Pratik Joshi, Sebastin Santy, Amar Budhiraja, [Kalika Bali](https://md.research.elamur.ai/free/en/authors/kalika-bali/), [Monojit Choudhury](https://md.research.elamur.ai/free/en/authors/monojit-choudhury/) · **Institutions:** [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) · [Microsoft Research (India)](https://md.research.elamur.ai/free/en/companies/microsoft-research/) · **Year:** 2020 · **Venue:** arXiv:2004.09095 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2004.09095) · [PDF](https://arxiv.org/pdf/2004.09095) · [DOI 10.48550/arxiv.2004.09095](https://doi.org/10.48550/arxiv.2004.09095) · [arXiv 2004.09095](https://arxiv.org/abs/2004.09095) · [OpenAlex](https://openalex.org/W3017311573)

**Citations:** 734 citations (OpenAlex, 2026-10-14) · [758](https://md.research.elamur.ai/free/en/papers/joshi-2020-state-fate-linguistic-diversity-inclusion/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
Although the world has more than 7,000 languages, only a small share benefit from modern NLP, and many systems described as "language agnostic" have only been tested on a few related, well-resourced languages. Joshi and colleagues at Microsoft Research India measure this gap. For each language they count two kinds of data. Labeled resources are datasets listed in the LDC catalog and the ELRA Map. Unlabeled resources are measured by Wikipedia pages. These two counts sort languages into a six-class taxonomy, from class 0, the "Left-Behinds", to class 5, the "Winners". The authors then use the classes to ask three questions. How do resources relate to speaker numbers? Which typological features do resource-rich languages fail to cover? How have ACL, NAACL, EACL, EMNLP, LREC and workshop papers spread their attention across languages over the decades? To answer the last one they use entropy measures and embeddings of conferences, languages and authors. The same hierarchy shows up in every analysis. The paper matters because it gives a shared vocabulary for resource inequality. Ukrainian falls in class 3, the "Rising Stars": it has plenty of unlabeled text but few labeled datasets.

## Key points
- Class 0 holds 2,191 languages (88.17% of the total, about 1B speakers). Class 5 has only 7 languages, including English, Spanish, German, Japanese and French, which together cover about 2.5B speakers.
- Class 3 contains Ukrainian, Indonesian, Cebuano, Afrikaans and Hebrew. These languages have a strong web presence, and the authors expect them to gain most from multilingual pre-training such as mBERT, provided that more labeled data is collected.
- The opening puzzle is revealed as Dutch versus Somali. Dutch has about 29M speakers, 69 LDC/ELRA items and 2M Wikipedia articles. Somali has about 18M speakers, 2 items and 5.5K articles.
- LREC and workshops cover the widest range of language classes, and newer conferences are more inclusive than older ones. Some typological features, such as 144E, are common in low-resource languages but rare in high-resource ones.
- An author–language MRR analysis finds that dedicated research communities exist for Japanese, Mandarin, Turkish and Hindi (MRR(10) > 0.75). Burmese (0.02), Igbo (0.13) and Javanese (0.23) get little attention despite having millions of speakers. The authors recommend adding diversity and inclusion questions to submission and reviewer forms.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{joshi2020state,
  title = {{The State and Fate of Linguistic Diversity and Inclusion in the NLP World}},
  author = {Joshi, Pratik and Santy, Sebastin and Budhiraja, Amar and Bali, Kalika and Choudhury, Monojit},
  year = {2020},
  doi = {10.48550/arxiv.2004.09095},
  eprint = {2004.09095},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2004.09095}
}
```

- [← previous How Can We Know What Language Models Know?](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/)
- [next → LEGAL-BERT: The Muppets straight out of Law School](https://md.research.elamur.ai/free/en/papers/chalkidis-2020-legal-bert-muppets-straight-out/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/)

## References

15 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3874–3884… [find](https://scholar.google.com/scholar?q=Roee+Aharoni%2C+Melvin+Johnson%2C+and+Orhan+Firat%2E+2019%2E+Massively+multilingual+neural+machine+translation%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+Association+for+Compu) [our card](https://md.research.elamur.ai/free/en/papers/aharoni-2019-massively-multilingual-neural-machine-translation/)
2. Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597–610. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe+and+Holger+Schwenk%2E+2019%2E+Massively+multilingual+sentence+embeddings+for+zero%2Dshot+cross%2Dlingual+transfer+and+beyond%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+7%3A597) [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2018-massively-multilingual-sentence-embeddings-zero/)
3. Emily M. Bender. 2011. On achieving and evaluating language-independence in NLP. In Vol 6: Interaction of Linguistics and Computational Linguistics. [find](https://scholar.google.com/scholar?q=Emily+M%2E+Bender%2E+2011%2E+On+achieving+and+evaluating+language%2Dindependence+in+NLP%2E+In+Vol+6%3A+Interaction+of+Linguistics+and+Computational+Linguistics%2E)
4. Steven Bird, Robert Dale, Bonnie J Dorr, Bryan Gibson, Mark Thomas Joseph, Min-Yen Kan, Dongwon Lee, Brett Powley, Dragomir R Radev, and Yee Fan Tan. 2008. The ACL anthology reference corpus: A reference dataset for bibliographic research in computational linguistics. LREC’08. [find](https://scholar.google.com/scholar?q=Steven+Bird%2C+Robert+Dale%2C+Bonnie+J+Dorr%2C+Bryan+Gibson%2C+Mark+Thomas+Joseph%2C+Min%2DYen+Kan%2C+Dongwon+Lee%2C+Brett+Powley%2C+Dragomir+R+Radev%2C+and+Yee+Fan+Tan%2E+2008%2E+The+ACL+anthology+reference+corpus%3A+A+refere) [our card](https://md.research.elamur.ai/free/en/papers/bird-2008-acl-anthology-reference-corpus-reference/)
5. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 7059– 7069. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Alexis+Conneau+and+Guillaume+Lample%2E+2019%2E+Cross%2Dlingual+language+model+pretraining%2E+In+H%2E+Wallach%2C+H%2E+Larochelle%2C+A%2E+Beygelzimer%2C+F%2E+d%27Alche%CC%81%2DBuc%2C+E%2E+Fox%2C+and+R%2E+Garnett%2C+editors%2C+Advances+in+Neural+)
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N) [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
7. Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. [find](https://scholar.google.com/scholar?q=Matthew+S%2E+Dryer+and+Martin+Haspelmath%2C+editors%2E+2013%2E+WALS+Online%2E+Max+Planck+Institute+for+Evolutionary+Anthropology%2C+Leipzig%2E)
8. David Kamholz, Jonathan Pool, and Susan Colowick. 2014. PanLex: Building a resource for panlingual lexical translation. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 3145– 3150, Reykjavik, Iceland. European Language Resources Association (… [find](https://scholar.google.com/scholar?q=David+Kamholz%2C+Jonathan+Pool%2C+and+Susan+Colowick%2E+2014%2E+PanLex%3A+Building+a+resource+for+panlingual+lexical+translation%2E+In+Proceedings+of+the+Ninth+International+Conference+on+Language+Resources+and+E)
9. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28–39, Vancouver. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Philipp+Koehn+and+Rebecca+Knowles%2E+2017%2E+Six+challenges+for+neural+machine+translation%2E+In+Proceedings+of+the+First+Workshop+on+Neural+Machine+Translation%2C+pages+28%E2%80%9339%2C+Vancouver%2E+Association+for+Comp)
10. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research, 9(Nov):2579–2605. [find](https://scholar.google.com/scholar?q=Laurens+van+der+Maaten+and+Geoffrey+Hinton%2E+2008%2E+Visualizing+data+using+t%2DSNE%2E+Journal+of+machine+learning+research%2C+9%28Nov%29%3A2579%E2%80%932605%2E)
11. Tomas Mikolov, Kai Chen, G. S. Corrado, and J. Dean. 2013. Efficient estimation of word representations in vector space. In ICLR. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Kai+Chen%2C+G%2E+S%2E+Corrado%2C+and+J%2E+Dean%2E+2013%2E+Efficient+estimation+of+word+representations+in+vector+space%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/)
12. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996– 5001, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Telmo+Pires%2C+Eva+Schlinger%2C+and+Dan+Garrette%2E+2019%2E+How+multilingual+is+multilingual+BERT%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+4996%E2%80%93+5001%2C) [our card](https://md.research.elamur.ai/free/en/papers/pires-2019-how-multilingual-is-multilingual-bert/)
13. Edoardo Maria Ponti, Helen O’Horan, Yevgeni Berzak, Ivan Vulić, Roi Reichart, Thierry Poibeau, Ekaterina Shutova, and Anna Korhonen. 2019. Modeling language variation and universals: A survey on typological linguistics for natural language processing. Computational Linguistics, 45(3):559–601. [find](https://scholar.google.com/scholar?q=Edoardo+Maria+Ponti%2C+Helen+O%E2%80%99Horan%2C+Yevgeni+Berzak%2C+Ivan+Vulic%CC%81%2C+Roi+Reichart%2C+Thierry+Poibeau%2C+Ekaterina+Shutova%2C+and+Anna+Korhonen%2E+2019%2E+Modeling+language+variation+and+universals%3A+A+survey+on+typo) [our card](https://md.research.elamur.ai/free/en/papers/ponti-2019-modeling-language-variation-universals-survey/)
14. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natu) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
15. Gary Simons and Steven Bird. 2003. The open language archives community: An infrastructure for distributed archiving of language resources. Computing Research Repository - CORR, 18:117–128. [find](https://scholar.google.com/scholar?q=Gary+Simons+and+Steven+Bird%2E+2003%2E+The+open+language+archives+community%3A+An+infrastructure+for+distributed+archiving+of+language+resources%2E+Computing+Research+Repository+%2D+CORR%2C+18%3A117%E2%80%93128%2E)