---
title: "IndicNLPSuite — Kakwani 2020"
description: "Indian languages have more than a billion speakers but few large public text collections or shared evaluation suites, which has slowed work on…"
authors: ["Divyanshu Kakwani", "Anoop Kunchukuttan", "Satish Golla", "Gokul N.C.", "Avik Bhattacharyya", "Mitesh M. Khapra", "Pratyush Kumar"]
year: 2020
venue: "Findings of the Association for Computational Linguistics: EMNLP 2020"
source: "https://aclanthology.org/2020.findings-emnlp.445/"
doi: "10.18653/v1/2020.findings-emnlp.445"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kakwani-2020-indicnlpsuite-monolingual-corpora-evaluation-benchmarks/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kakwani-2020-indicnlpsuite-monolingual-corpora-evaluation-benchmarks/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/) / IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages

# IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages

**Authors:** [Divyanshu Kakwani](https://md.research.elamur.ai/free/en/authors/divyanshu-kakwani/), [Anoop Kunchukuttan](https://md.research.elamur.ai/free/en/authors/anoop-kunchukuttan/), Satish Golla, Gokul N.C., Avik Bhattacharyya, [Mitesh M. Khapra](https://md.research.elamur.ai/free/en/authors/mitesh-m-khapra/), [Pratyush Kumar](https://md.research.elamur.ai/free/en/authors/pratyush-kumar/) · **Year:** 2020 · **Venue:** Findings of the Association for Computational Linguistics: EMNLP 2020 · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.findings-emnlp.445/) · [PDF](https://aclanthology.org/2020.findings-emnlp.445.pdf) · [DOI 10.18653/v1/2020.findings-emnlp.445](https://doi.org/10.18653/v1/2020.findings-emnlp.445) · [OpenAlex](https://openalex.org/W3099919888)

**Citations:** 446 citations (OpenAlex, 2026-10-14) · [284](https://md.research.elamur.ai/free/en/papers/kakwani-2020-indicnlpsuite-monolingual-corpora-evaluation-benchmarks/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
Indian languages have more than a billion speakers but few large public text collections or shared evaluation suites, which has slowed work on pre-trained models for them. The authors address this with IndicNLPSuite, a set of resources covering 11 Indo-Aryan and Dravidian languages plus Indian English. They crawled mainly news sites, magazines and blogs, extracted article text with BoilerPipe or custom BeautifulSoup scrapers, and added OSCAR CommonCrawl data. After Unicode normalization, sentence splitting and de-duplication, the resulting corpus, IndicCorp, contains about 8.8 billion tokens. On top of it they trained FastText word vectors (IndicFT) and multilingual ALBERT models (IndicBERT, base and large). They also built IndicGLUE, a benchmark that combines existing datasets, new translations of English tasks, and tasks generated semi-automatically from website and Wikipedia structure. IndicBERT matches or outperforms mBERT and XLM-R on most tasks, with the clearest gains in cross-lingual sentence retrieval. The work matters because it gives a large, diverse language group a common foundation for building and comparing models.

## Key points
- IndicCorp contains about 452.8M sentences and 8.8B tokens across 11 Indian languages and English. It is several times larger than OSCAR, with per-language size ratios from roughly 2× to 22×. Hindi is the largest component (1.86B tokens) and Assamese the smallest (32.6M).
- The suite has four parts:
  - IndicCorp (the corpora);
  - IndicFT (FastText embeddings, chosen to cope with rich morphology);
  - IndicBERT (compact ALBERT-based multilingual models);
  - IndicGLUE (the benchmark).
- IndicGLUE tasks include:
  - article genre classification, headline prediction and Wikipedia section-title prediction;
  - cloze-style multiple-choice QA;
  - WNLI and COPA translations;
  - NER, identifying paraphrases, classifying sentiment, labeling discourse modes, and retrieving sentences across languages.
- On public datasets, IndicBERT-base averages 77.39, compared with 74.42 for mBERT and 76.60 for XLM-R. On cross-lingual retrieval (Precision@10), IndicBERT large averages 34.10, against 27.12 for base, 21.46 for mBERT and 13.74 for XLM-R.
- mBERT does better on Wikipedia-derived tasks such as NER. Across languages, IndicBERT performs worst on Assamese and Odia, the two smallest corpora. The resources are released under CC BY-NC-SA 4.0.

## Interesting ideas
- **Keep Wikipedia out of pre-training.** The authors left Wikipedia out of the training corpus on purpose so it could serve as clean material for evaluation tasks. Separating pre-training sources from benchmark sources this way helps prevent contamination when building evaluation sets for any language.
- **Generate tasks from existing metadata.** Benchmark tasks were produced semi-automatically from structure already on the web, such as article genres from site categories, headlines, and Wikipedia section titles. This is an inexpensive way to get many NLU tasks for low-resource languages without large annotation budgets.
- **Zero-training sentence retrieval.** Cross-lingual retrieval is scored without fine-tuning: sentence vectors are mean-pooled from the last layer, and each language's vectors are centered to strip out language-specific offsets. This simple probe of multilingual alignment can be applied to any encoder.
- **Normalize Unicode first.** Text is canonicalized to a single Unicode form before tokenization, because Indic scripts allow several encodings of the same character. The same step is worth adding to any pipeline for scripts with variant encodings.
- **Read results through per-language corpus size.** Comparing performance language by language against corpus size showed which languages are held back mainly by data volume. This gives a quick way to decide where further crawling would pay off most.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · [Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kakwani2020indicnlpsuite,
  title = {{IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages}},
  author = {Kakwani, Divyanshu and Kunchukuttan, Anoop and Golla, Satish and N.C and Bhattacharyya, Avik and Khapra, Mitesh M. and Kumar, Pratyush},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.445},
  url = {https://doi.org/10.18653/v1/2020.findings-emnlp.445}
}
```

- [← previous A Novel Cascade Binary Tagging Framework for Relational Triple Extraction](https://md.research.elamur.ai/free/en/papers/wei-2020-novel-cascade-binary-tagging-framework/)
- [next → Fact or Fiction: Verifying Scientific Claims](https://md.research.elamur.ai/free/en/papers/wadden-2020-fact-or-fiction-verifying-scientific/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/)

## References

48 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Md. Shad Akhtar, Ayush Kumar, Asif Ekbal, and Pushpak Bhattacharyya. 2016. A hybrid deep learning architecture for sentiment analysis. In COLING 2016, 26th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, December 11-16, 2016, Osaka, Japan, page… [find](https://scholar.google.com/scholar?q=Md%2E+Shad+Akhtar%2C+Ayush+Kumar%2C+Asif+Ekbal%2C+and+Pushpak+Bhattacharyya%2E+2016%2E+A+hybrid+deep+learning+architecture+for+sentiment+analysis%2E+In+COLING+2016%2C+26th+International+Conference+on+Computational+Li)
2. Syed Sarfaraz Akhtar, Arihant Gupta, Avijit Vajpayee, Arjit Srivastava, and Manish Shrivastava. 2017. Word similarity datasets for Indian languages: Annotation and baseline systems. In Proceedings of the 11th Linguistic Annotation Workshop, pages 91–94, Valencia, Spain. Association for Computational… [find](https://scholar.google.com/scholar?q=Syed+Sarfaraz+Akhtar%2C+Arihant+Gupta%2C+Avijit+Vajpayee%2C+Arjit+Srivastava%2C+and+Manish+Shrivastava%2E+2017%2E+Word+similarity+datasets+for+Indian+languages%3A+Annotation+and+baseline+systems%2E+In+Proceedings+of+)
3. Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed word representations for multilingual nlp. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 183–192, Sofia, Bulgaria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rami+Al%2DRfou%2C+Bryan+Perozzi%2C+and+Steven+Skiena%2E+2013%2E+Polyglot%3A+Distributed+word+representations+for+multilingual+nlp%2E+In+Proceedings+of+the+Seventeenth+Conference+on+Computational+Natural+Language+Le) [our card](https://md.research.elamur.ai/free/en/papers/rfou-2013-polyglot-distributed-word-representations-multilingual/)
4. M. Anand Kumar, S. Singh, B. Kavirajan, and K.P. Soman. 2016. DPIL@FIRE 2016: Overview of shared task on detecting paraphrases in Indian Languages (DPIL). In CEUR Workshop Proceedings. [find](https://scholar.google.com/scholar?q=M%2E+Anand+Kumar%2C+S%2E+Singh%2C+B%2E+Kavirajan%2C+and+K%2EP%2E+Soman%2E+2016%2E+DPIL%40FIRE+2016%3A+Overview+of+shared+task+on+detecting+paraphrases+in+Indian+Languages+%28DPIL%29%2E+In+CEUR+Workshop+Proceedings%2E)
5. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. [find](https://scholar.google.com/scholar?q=Piotr+Bojanowski%2C+Edouard+Grave%2C+Armand+Joulin%2C+and+Tomas+Mikolov%2E+2017%2E+Enriching+word+vectors+with+subword+information%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+5%3A135%E2%80%93146%2E)
6. Ondrej Bojar, Vojtech Diatka, Pavel Rychlỳ, Pavel Stranák, Vít Suchomel, Ales Tamchyna, and Daniel Zeman. 2014. Hindencorp-hindi-english and hindi-only corpus for machine translation. In LREC, pages 3550–3555. [find](https://scholar.google.com/scholar?q=Ondrej+Bojar%2C+Vojtech+Diatka%2C+Pavel+Rychl%E1%BB%B3%2C+Pavel+Stran%C3%A1k%2C+V%C3%ADt+Suchomel%2C+Ales+Tamchyna%2C+and+Daniel+Zeman%2E+2014%2E+Hindencorp%2Dhindi%2Denglish+and+hindi%2Donly+corpus+for+machine+translation%2E+In+LREC%2C+pages+3)
7. Christian Buck, Kenneth Heafield, and Bas van Ooyen. 2014. N-gram counts and language models from the common crawl. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 3579–3584, Reykjavik, Iceland. European Language Resources Association (ELRA)… [find](https://scholar.google.com/scholar?q=Christian+Buck%2C+Kenneth+Heafield%2C+and+Bas+van+Ooyen%2E+2014%2E+N%2Dgram+counts+and+language+models+from+the+common+crawl%2E+In+Proceedings+of+the+Ninth+International+Conference+on+Language+Resources+and+Evalu) [our card](https://md.research.elamur.ai/free/en/papers/buck-2014-n-gram-counts-language-models/)
8. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jonathan+H%2E+Clark%2C+Eunsol+Choi%2C+Michael+Collins%2C+Dan+Garrette%2C+Tom+Kwiatkowski%2C+Vitaly+Nikolaev%2C+and+Jennimaria+Palomaki%2E+2020%2E+Tydi+qa%3A+A+benchmark+for+information%2Dseeking+question+answering+in+typol) [our card](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/)
9. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, E. Grave, Myle Ott, L. Zettlemoyer, and V. Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In ACL. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+E%2E+Grave%2C+Myle+Ott%2C+L%2E+Zettlemoyer%2C+and+V%2E+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dlingual+represent)
10. Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In Proceedings of the International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Guillaume+Lample%2C+Marc%E2%80%99Aurelio+Ranzato%2C+Ludovic+Denoyer%2C+and+Herv%C3%A9+J%C3%A9gou%2E+2018%2E+Word+translation+without+parallel+data%2E+In+Proceedings+of+the+International+Conference+on+Learning+Repre)
11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N)
12. Swapnil Dhanwal, Hritwik Dutta, Hitesh Nankani, Nilay Shrivastava, Yaman Kumar, Junyi Jessy Li, Debanjan Mahata, Rakesh Gosangi, Haimin Zhang, Rajiv Ratn Shah, and Amanda Stent. 2020. An annotated dataset of discourse modes in Hindi stories. In Proceedings of The 12th Language Resources and Evaluati… [find](https://scholar.google.com/scholar?q=Swapnil+Dhanwal%2C+Hritwik+Dutta%2C+Hitesh+Nankani%2C+Nilay+Shrivastava%2C+Yaman+Kumar%2C+Junyi+Jessy+Li%2C+Debanjan+Mahata%2C+Rakesh+Gosangi%2C+Haimin+Zhang%2C+Rajiv+Ratn+Shah%2C+and+Amanda+Stent%2E+2020%2E+An+annotated+dat)
13. Murray B Emeneau. 1956. India as a lingustic area. Language. [find](https://scholar.google.com/scholar?q=Murray+B+Emeneau%2E+1956%2E+India+as+a+lingustic+area%2E+Language%2E)
14. D. Goldhahn, T. Eckart, and U. Quasthoff. 2012. Building large monolingual dictionaries at the leipzig corpora collection: From 100 to 200 languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2012). [find](https://scholar.google.com/scholar?q=D%2E+Goldhahn%2C+T%2E+Eckart%2C+and+U%2E+Quasthoff%2E+2012%2E+Building+large+monolingual+dictionaries+at+the+leipzig+corpora+collection%3A+From+100+to+200+languages%2E+In+Proceedings+of+the+International+Conference+on+) [our card](https://md.research.elamur.ai/free/en/papers/goldhahn-2012-building-large-monolingual-dictionaries-leipzig/)
15. Andrew S. Gordon, Zornitsa Kozareva, and Melissa Roemmele. 2011. Semeval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In SemEval@NAACL-HLT. [find](https://scholar.google.com/scholar?q=Andrew+S%2E+Gordon%2C+Zornitsa+Kozareva%2C+and+Melissa+Roemmele%2E+2011%2E+Semeval%2D2012+task+7%3A+Choice+of+plausible+alternatives%3A+An+evaluation+of+commonsense+causal+reasoning%2E+In+SemEval%40NAACL%2DHLT%2E) [our card](https://md.research.elamur.ai/free/en/papers/gordon-2012-semeval-2012-task-7-choice/)
16. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018). [find](https://scholar.google.com/scholar?q=Edouard+Grave%2C+Piotr+Bojanowski%2C+Prakhar+Gupta%2C+Armand+Joulin%2C+and+Tomas+Mikolov%2E+2018%2E+Learning+word+vectors+for+157+languages%2E+In+Proceedings+of+the+International+Conference+on+Language+Resources+an) [our card](https://md.research.elamur.ai/free/en/papers/grave-2018-learning-word-vectors-157-languages/)
17. Deepak Gupta, Surabhi Kumari, Asif Ekbal, and Pushpak Bhattacharyya. 2018. MMQA: A Multi-domain Multi-lingual Question-Answering Framework for English and Hindi. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Langua… [find](https://scholar.google.com/scholar?q=Deepak+Gupta%2C+Surabhi+Kumari%2C+Asif+Ekbal%2C+and+Pushpak+Bhattacharyya%2E+2018%2E+MMQA%3A+A+Multi%2Ddomain+Multi%2Dlingual+Question%2DAnswering+Framework+for+English+and+Hindi%2E+In+Proceedings+of+the+Eleventh+Interna)
18. Bushra Jawaid, Amir Kamran, and Ondrej Bojar. 2014. A Tagged Corpus and a Tagger for Urdu. In LREC, pages 2938–2943. [find](https://scholar.google.com/scholar?q=Bushra+Jawaid%2C+Amir+Kamran%2C+and+Ondrej+Bojar%2E+2014%2E+A+Tagged+Corpus+and+a+Tagger+for+Urdu%2E+In+LREC%2C+pages+2938%E2%80%932943%2E)
19. Pratik Jawanpuria, Arjun Balgovind, Anoop Kunchukuttan, and Bamdev Mishra. 2019. Learning multilingual word embeddings in latent metric space: a geometric approach. Transaction of the Association for Computational Linguistics (TACL), 7:107–120. [find](https://scholar.google.com/scholar?q=Pratik+Jawanpuria%2C+Arjun+Balgovind%2C+Anoop+Kunchukuttan%2C+and+Bamdev+Mishra%2E+2019%2E+Learning+multilingual+word+embeddings+in+latent+metric+space%3A+a+geometric+approach%2E+Transaction+of+the+Association+for+)
20. Christian Kohlschütter, Péter Fankhauser, and Wolfgang Nejdl. 2010. Boilerplate detection using shallow text features. In WSDM. [find](https://scholar.google.com/scholar?q=Christian+Kohlsch%C3%BCtter%2C+P%C3%A9ter+Fankhauser%2C+and+Wolfgang+Nejdl%2E+2010%2E+Boilerplate+detection+using+shallow+text+features%2E+In+WSDM%2E)
21. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Associ… [find](https://scholar.google.com/scholar?q=Taku+Kudo+and+John+Richardson%2E+2018%2E+SentencePiece%3A+A+simple+and+language+independent+subword+tokenizer+and+detokenizer+for+neural+text+processing%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+M) [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/)
22. Anoop Kunchukuttan. 2020. The IndicNLP Library. [https://github.com/anoopkunchukuttan/indic_](https://github.com/anoopkunchukuttan/indic_) nlp_library/blob/master/docs/indicnlp.pdf . [↗](https://github.com/anoopkunchukuttan/indic_)
23. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. ArXiv, abs/1909.11942. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+2020%2E+Albert%3A+A+lite+bert+for+self%2Dsupervised+learning+of+language+representations%2E+ArXiv%2C+abs%2F1909%2E11942%2E)
24. Hang Le, Loic Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, B. Lecouteux, A. Allauzen, B. Crabbé, L. Besacier, and Didier Schwab. 2020. Flaubert: Unsupervised language model pre-training for french. In LREC. [find](https://scholar.google.com/scholar?q=Hang+Le%2C+Loic+Vial%2C+Jibril+Frej%2C+Vincent+Segonne%2C+Maximin+Coavoux%2C+B%2E+Lecouteux%2C+A%2E+Allauzen%2C+B%2E+Crabb%C3%A9%2C+L%2E+Besacier%2C+and+Didier+Schwab%2E+2020%2E+Flaubert%3A+Unsupervised+language+model+pre%2Dtraining+for+fr) [our card](https://md.research.elamur.ai/free/en/papers/le-2019-flaubert-unsupervised-language-model-pre/)
25. H. Levesque, E. Davis, and L. Morgenstern. 2011. The winograd schema challenge. In KR. [find](https://scholar.google.com/scholar?q=H%2E+Levesque%2C+E%2E+Davis%2C+and+L%2E+Morgenstern%2E+2011%2E+The+winograd+schema+challenge%2E+In+KR%2E)
26. Jindřich Libovický, Rudolf Rosa, and Alexander Fraser. 2019. How language-neutral is multilingual bert? [find](https://scholar.google.com/scholar?q=Jind%C5%99ich+Libovick%C3%BD%2C+Rudolf+Rosa%2C+and+Alexander+Fraser%2E+2019%2E+How+language%2Dneutral+is+multilingual+bert%3F)
27. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+RoBERTa%3A+A+robustly+optimized+BERT+pretraining+approach%2E+a)
28. Anthony McEnery, Paul Baker, Rob Gaizauskas, and Hamish Cunningham. 2000. Emille: Building a corpus of south asian languages. VIVEK-BOMBAY-, 13(3):22–28. [find](https://scholar.google.com/scholar?q=Anthony+McEnery%2C+Paul+Baker%2C+Rob+Gaizauskas%2C+and+Hamish+Cunningham%2E+2000%2E+Emille%3A+Building+a+corpus+of+south+asian+languages%2E+VIVEK%2DBOMBAY%2D%2C+13%283%29%3A22%E2%80%9328%2E)
29. Yu Meng, Jiaxin Huang, Guangyuan Wang, Chao Zhang, Honglei Zhuang, Lance Kaplan, and Jiawei Han. 2019. Spherical text embedding. In Advances in Neural Information Processing Systems, pages 8206–8215. [find](https://scholar.google.com/scholar?q=Yu+Meng%2C+Jiaxin+Huang%2C+Guangyuan+Wang%2C+Chao+Zhang%2C+Honglei+Zhuang%2C+Lance+Kaplan%2C+and+Jiawei+Han%2E+2019%2E+Spherical+text+embedding%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+8206%E2%80%938215%2E)
30. Tomas Mikolov, Kai Chen, G. S. Corrado, and J. Dean. 2013a. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Kai+Chen%2C+G%2E+S%2E+Corrado%2C+and+J%2E+Dean%2E+2013a%2E+Efficient+estimation+of+word+representations+in+vector+space%2E+CoRR%2C+abs%2F1301%2E3781%2E)
31. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Ilya+Sutskever%2C+Kai+Chen%2C+Greg+S+Corrado%2C+and+Jeff+Dean%2E+2013b%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+In+Advances+in+neural+information+processing)
32. Sandeep Sricharan Mukku and Radhika Mamidi. 2017. ACTSA: Annotated corpus for Telugu sentiment analysis. In Proceedings of the First Workshop on Building Linguistically Generalizable NLP Systems, pages 54–58, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sandeep+Sricharan+Mukku+and+Radhika+Mamidi%2E+2017%2E+ACTSA%3A+Annotated+corpus+for+Telugu+sentiment+analysis%2E+In+Proceedings+of+the+First+Workshop+on+Building+Linguistically+Generalizable+NLP+Systems%2C+page)
33. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7), Cardiff, United Kingdom. Leibniz-Institut für Deutsche Sprach… [find](https://scholar.google.com/scholar?q=Pedro+Javier+Ortiz+Su%C3%A1rez%2C+Beno%C3%AEt+Sagot%2C+and+Laurent+Romary%2E+2019%2E+Asynchronous+Pipeline+for+Processing+Huge+Corpora+on+Medium+to+Low+Resource+Infrastructures%2E+In+7th+Workshop+on+the+Challenges+in+the)
34. Pedro Javier Ortiz Suarez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. In Proceedings of the Workshop on Challenges in the Management of Large Corpora, pages 9 – 16, Mannheim. Leibniz-Institut für Deutsche Spra… [find](https://scholar.google.com/scholar?q=Pedro+Javier+Ortiz+Suarez%2C+Beno%C3%AEt+Sagot%2C+and+Laurent+Romary%2E+2019%2E+Asynchronous+pipelines+for+processing+huge+corpora+on+medium+to+low+resource+infrastructures%2E+In+Proceedings+of+the+Workshop+on+Chall)
35. Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958, Vancouver, Canada. A… [find](https://scholar.google.com/scholar?q=Xiaoman+Pan%2C+Boliang+Zhang%2C+Jonathan+May%2C+Joel+Nothman%2C+Kevin+Knight%2C+and+Heng+Ji%2E+2017%2E+Cross%2Dlingual+name+tagging+and+linking+for+282+languages%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Asso) [our card](https://md.research.elamur.ai/free/en/papers/pan-2017-cross-lingual-name-tagging-linking/)
36. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+Manning%2E+2014%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+Proceedings+of+the+2014+conference+on+empirical+methods+in+natural+language+process)
37. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… [find](https://scholar.google.com/scholar?q=Matthew+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+In+Proceedings+of+the+2018+Conference+of+t)
38. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural… [find](https://scholar.google.com/scholar?q=Fabio+Petroni%2C+Tim+Rockt%C3%A4schel%2C+Sebastian+Riedel%2C+Patrick+Lewis%2C+Anton+Bakhtin%2C+Yuxiang+Wu%2C+and+Alexander+Miller%2E+2019%2E+Language+models+as+knowledge+bases%3F+In+Proceedings+of+the+2019+Conference+on+Emp) [our card](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/)
39. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996– 5001, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Telmo+Pires%2C+Eva+Schlinger%2C+and+Dan+Garrette%2E+2019%2E+How+multilingual+is+multilingual+BERT%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+4996%E2%80%93+5001%2C) [our card](https://md.research.elamur.ai/free/en/papers/pires-2019-how-multilingual-is-multilingual-bert/)
40. Alec Radford. 2018. understanding by Improving language generative pre-training. [https://cdn.openai.com/research-covers/](https://cdn.openai.com/research-covers/) language-unsupervised/language_ understanding_paper.pdf . [↗](https://cdn.openai.com/research-covers/)
41. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [https://cdn.openai.com/better-language-models/language_models_are_](https://cdn.openai.com/better-language-models/language_models_are_) unsupervised_multitask_learners.pdf . [↗](https://cdn.openai.com/better-language-models/language_models_are_)
42. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J%2E+Liu%2E+2019%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtex)
43. Dataset N \# Examples Train Test Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (… [find](https://scholar.google.com/scholar?q=Dataset+N+%5C%23+Examples+Train+Test+Nils+Reimers+and+Iryna+Gurevych%2E+2019%2E+Sentence%2DBERT%3A+Sentence+embeddings+using+Siamese+BERT%2Dnetworks%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Na)
44. Sebastian Ruder, Anders Søgaard, and Ivan Vulić. 2019. Unsupervised cross-lingual representation learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 31–38, Florence, Italy. Association for Computational Linguistics. hi IITP+… [find](https://scholar.google.com/scholar?q=Sebastian+Ruder%2C+Anders+S%C3%B8gaard%2C+and+Ivan+Vuli%C4%87%2E+2019%2E+Unsupervised+cross%2Dlingual+representation+learning%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+T)
45. Shashank Siripragrada, Jerin Philip, Vinay P. Namboodiri, and C V Jawahar. 2020. A multilingual parallel corpora collection effort for Indian languages. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 3743–3751, Marseille, France. European Language Resources Associatio… [find](https://scholar.google.com/scholar?q=Shashank+Siripragrada%2C+Jerin+Philip%2C+Vinay+P%2E+Namboodiri%2C+and+C+V+Jawahar%2E+2020%2E+A+multilingual+parallel+corpora+collection+effort+for+Indian+languages%2E+In+Proceedings+of+The+12th+Language+Resources+a)
46. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,… [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+Bowman%2E+2018%2E+GLUE%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+In+Proceedings+of+the) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
47. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzman, Armand Joulin, and Edouard Grave. 2019. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359. [find](https://scholar.google.com/scholar?q=Guillaume+Wenzek%2C+Marie%2DAnne+Lachaux%2C+Alexis+Conneau%2C+Vishrav+Chaudhary%2C+Francisco+Guzman%2C+Armand+Joulin%2C+and+Edouard+Grave%2E+2019%2E+Ccnet%3A+Extracting+high+quality+monolingual+datasets+from+web+crawl+da) [our card](https://md.research.elamur.ai/free/en/papers/wenzek-2020-ccnet-extracting-high-quality-monolingual/)
48. Liang Xu, Xuanwei Zhang, Lu Li, Hai Hu, Chenjie Cao, Weitang Liu, Junyi Li, Yudong Li, Kai Sun, Yechen Xu, et al. 2020. Clue: A chinese language understanding evaluation benchmark. arXiv preprint arXiv:2004.05986. [find](https://scholar.google.com/scholar?q=Liang+Xu%2C+Xuanwei+Zhang%2C+Lu+Li%2C+Hai+Hu%2C+Chenjie+Cao%2C+Weitang+Liu%2C+Junyi+Li%2C+Yudong+Li%2C+Kai+Sun%2C+Yechen+Xu%2C+et+al%2E+2020%2E+Clue%3A+A+chinese+language+understanding+evaluation+benchmark%2E+arXiv+preprint+arXi) [our card](https://md.research.elamur.ai/free/en/papers/xu-2020-clue-chinese-language-understanding/)