---
title: "GLUECoS: An Evaluation Benchmark for — Khanuja 2020"
description: "Code-switching, the mixing of languages within one conversation or sentence, is common in multilingual communities, but labelled resources for it are…"
authors: ["Simran Khanuja", "Sandipan Dandapat", "Anirudh Srinivasan", "Sunayana Sitaram", "Monojit Choudhury"]
year: 2020
venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/2020.acl-main.329/"
doi: "10.18653/v1/2020.acl-main.329"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/khanuja-2020-gluecos-evaluation-benchmark-code-switched/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/khanuja-2020-gluecos-evaluation-benchmark-code-switched/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/) / GLUECoS: An Evaluation Benchmark for Code-Switched NLP

# GLUECoS: An Evaluation Benchmark for Code-Switched NLP

**Authors:** [Simran Khanuja](https://md.research.elamur.ai/free/en/authors/simran-khanuja/), [Sandipan Dandapat](https://md.research.elamur.ai/free/en/authors/sandipan-dandapat/), [Anirudh Srinivasan](https://md.research.elamur.ai/free/en/authors/anirudh-srinivasan/), [Sunayana Sitaram](https://md.research.elamur.ai/free/en/authors/sunayana-sitaram/), [Monojit Choudhury](https://md.research.elamur.ai/free/en/authors/monojit-choudhury/) · **Year:** 2020 · **Venue:** Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.acl-main.329/) · [PDF](https://aclanthology.org/2020.acl-main.329.pdf) · [DOI 10.18653/v1/2020.acl-main.329](https://doi.org/10.18653/v1/2020.acl-main.329) · [OpenAlex](https://openalex.org/W3019003205)

**Citations:** 128 citations (openalex-snapshot · 2026-10-14) · [94](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Code-switching, the mixing of languages within one conversation or sentence, is common in multilingual communities, but labelled resources for it are scarce. Earlier evaluations each targeted a single task, and claims that mBERT handles mixed-language text had been tested only on POS tagging. Inspired by GLUE, the authors built GLUECoS, a benchmark for English-Hindi and English-Spanish. It combines existing shared-task datasets for five tasks with a new code-switched Natural Language Inference dataset. On this benchmark they compared baselines, cross-lingual word embeddings and mBERT. They also fine-tuned mBERT on synthetic code-switched text generated with linguistic constraints, together with a little real mixed data. mBERT clearly beat all cross-lingual embeddings, and this modified mBERT usually did best of all. Even so, only LID looks close to solved: on NLI the best models score just above chance. The benchmark therefore exposes how far models remain from genuinely understanding code-switched language and gives the field a shared, multi-task testbed.

## Key points
- **Tasks and datasets:**
  - LID: FIRE 2013 (En-Hi) and EMNLP 2014 (En-Es).
  - POS: UD POS and FG POS (En-Hi), Bangor POS (En-Es).
  - NER: IIITH NER (En-Hi), CALCS NER (En-Es).
  - Sentiment: SAIL (En-Hi) and an En-Es sentiment corpus.
  - QA and the new NLI set: En-Hi only.
  - The En-Hi data includes Hindi in both Roman and Devanagari script.
- **How mixed the data is:** each corpus is described with CMI, switch-point average, M-index, I-index, burstiness, Language Entropy and Span Entropy.
- **Models compared:**
  - Cross-lingual embeddings: BiCVM, BiSkip, supervised and unsupervised MUSE, and GCM-based embeddings.
  - Multilingual models: mBERT and the modified mBERT.
- **Results on the hardest tasks:**
  - NLI (En-Hi, two-way, accuracy): mBERT 61.09, modified mBERT 63.1.
  - QA (En-Hi, F1): mBERT 71.96 versus 68.01 for the modified model. This is one of the few cases where fine-tuning on synthetic data did not help.
- **Language pairs:** En-Es scores were generally higher than En-Hi. The authors point to more similar word order, non-standard romanized Hindi spelling with transliteration errors, and domain mismatch in the En-Hi training corpora.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

94 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 91 not yet.

- [MEGA: Multilingual Evaluation of Generative AI](https://md.research.elamur.ai/free/en/papers/ahuja-2023-mega-multilingual-evaluation-generative-ai/)
- [Bridging Laughter Across Languages: Generation of Hindi-English Code-mixed Puns](https://md.research.elamur.ai/free/en/papers/asapu-2025-bridging-laughter-across-languages-generation/)
- [XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation](https://md.research.elamur.ai/free/en/papers/ruder-2021-xtreme-r-more-challenging-nuanced/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{khanuja2020gluecos,
  title = {{GLUECoS: An Evaluation Benchmark for Code-Switched NLP}},
  author = {Khanuja, Simran and Dandapat, Sandipan and Srinivasan, Anirudh and Sitaram, Sunayana and Choudhury, Monojit},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.329},
  url = {https://doi.org/10.18653/v1/2020.acl-main.329}
}
```

- [← previous GLUCOSE: GeneraLized and COntextualized Story Explanations](https://md.research.elamur.ai/free/en/papers/commonsense-2020-mostafazadeh-glucose-generalized-contextualized-story-explanations/)
- [next → Good-Enough Compositional Data Augmentation](https://md.research.elamur.ai/free/en/papers/andreas-2020-good-enough-compositional-data-augmentation/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/)

## References

37 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Gustavo Aguilar, Fahad AlGhamdi, Victor Soto, Mona Diab, Julia Hirschberg, and Thamar Solorio. 2019. Named entity recognition on code-switched data: Overview of the calcs 2018 shared task. arXiv preprint arXiv:1906.04138. [find](https://scholar.google.com/scholar?q=Gustavo+Aguilar%2C+Fahad+AlGhamdi%2C+Victor+Soto%2C+Mona+Diab%2C+Julia+Hirschberg%2C+and+Thamar+Solorio%2E+2019%2E+Named+entity+recognition+on+code%2Dswitched+data%3A+Overview+of+the+calcs+2018+shared+task%2E+arXiv+prepr)
2. Gustavo Aguilar, Fahad AlGhamdi, Victor Soto, Thamar Solorio, Mona Diab, and Julia Hirschberg. 2018. Proceedings of the third workshop on computational approaches to linguistic code-switching. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching. [find](https://scholar.google.com/scholar?q=Gustavo+Aguilar%2C+Fahad+AlGhamdi%2C+Victor+Soto%2C+Thamar+Solorio%2C+Mona+Diab%2C+and+Julia+Hirschberg%2E+2018%2E+Proceedings+of+the+third+workshop+on+computational+approaches+to+linguistic+code%2Dswitching%2E+In+Proc)
3. Fahad AlGhamdi, Giovanni Molina, Mona Diab, Thamar Solorio, Abdelati Hawwari, Victor Soto, and Julia Hirschberg. 2016. Part of speech tagging for code switched data. [find](https://scholar.google.com/scholar?q=Fahad+AlGhamdi%2C+Giovanni+Molina%2C+Mona+Diab%2C+Thamar+Solorio%2C+Abdelati+Hawwari%2C+Victor+Soto%2C+and+Julia+Hirschberg%2E+2016%2E+Part+of+speech+tagging+for+code+switched+data%2E)
4. Amitava Das Anupam Jamatia. 2016. Task report: Tool contest on pos tagging for code-mixed indian social media (facebook, twitter, and whatsapp) text @ icon 2016. Unpublished. [find](https://scholar.google.com/scholar?q=Amitava+Das+Anupam+Jamatia%2E+2016%2E+Task+report%3A+Tool+contest+on+pos+tagging+for+code%2Dmixed+indian+social+media+%28facebook%2C+twitter%2C+and+whatsapp%29+text+%40+icon+2016%2E+Unpublished%2E)
5. Somnath Banerjee, Kunal Chakma, Sudip Kumar Naskar, Amitava Das, Paolo Rosso, Sivaji Bandyopadhyay, and Monojit Choudhury. 2016. Overview of the mixed script information retrieval (msir) at fire-2016. In Forum for Information Retrieval Evaluation, pages 39–49. Springer. [find](https://scholar.google.com/scholar?q=Somnath+Banerjee%2C+Kunal+Chakma%2C+Sudip+Kumar+Naskar%2C+Amitava+Das%2C+Paolo+Rosso%2C+Sivaji+Bandyopadhyay%2C+and+Monojit+Choudhury%2E+2016%2E+Overview+of+the+mixed+script+information+retrieval+%28msir%29+at+fire%2D2016%2E)
6. Irshad Ahmad Bhat, Riyaz Ahmad Bhat, Manish Shrivastava, and Dipti Misra Sharma. 2018. Universal dependency parsing for hindi-english code-switching. arXiv preprint arXiv:1804.05868. [find](https://scholar.google.com/scholar?q=Irshad+Ahmad+Bhat%2C+Riyaz+Ahmad+Bhat%2C+Manish+Shrivastava%2C+and+Dipti+Misra+Sharma%2E+2018%2E+Universal+dependency+parsing+for+hindi%2Denglish+code%2Dswitching%2E+arXiv+preprint+arXiv%3A1804%2E05868%2E)
7. Khyathi Chandu, Ekaterina Loginova, Vishal Gupta, Josef van Genabith, Günter Neumann, Manoj Chinnakotla, Eric Nyberg, and Alan W Black. 2018a. Code-mixed question answering challenge: Crowdsourcing data and techniques. In Proceedings of the Third Workshop on Computational Approaches to Linguistic C… [find](https://scholar.google.com/scholar?q=Khyathi+Chandu%2C+Ekaterina+Loginova%2C+Vishal+Gupta%2C+Josef+van+Genabith%2C+Gu%CC%88nter+Neumann%2C+Manoj+Chinnakotla%2C+Eric+Nyberg%2C+and+Alan+W+Black%2E+2018a%2E+Code%2Dmixed+question+answering+challenge%3A+Crowdsourcing+d)
8. Khyathi Chandu, Thomas Manzini, Sumeet Singh, and Alan W. Black. 2018b. Language informed modeling of code-switched text. In Proceedings of the Third Workshop on Computational Approaches Mona Diab, Pascale Fung, Mahmoud Ghoneim, Julia Hirschberg, and Thamar Solorio. 2016. Proceedings of the second w… [find](https://scholar.google.com/scholar?q=Khyathi+Chandu%2C+Thomas+Manzini%2C+Sumeet+Singh%2C+and+Alan+W%2E+Black%2E+2018b%2E+Language+informed+modeling+of+code%2Dswitched+text%2E+In+Proceedings+of+the+Third+Workshop+on+Computational+Approaches+Mona+Diab%2C+Pa)
9. Mona Diab, Julia Hirschberg, Pascale Fung, and Thamar Solorio. 2014. Proceedings of the first workshop on computational approaches to code switching. In Proceedings of the First Workshop on Computational Approaches to Code Switching. [find](https://scholar.google.com/scholar?q=Mona+Diab%2C+Julia+Hirschberg%2C+Pascale+Fung%2C+and+Thamar+Solorio%2E+2014%2E+Proceedings+of+the+first+workshop+on+computational+approaches+to+code+switching%2E+In+Proceedings+of+the+First+Workshop+on+Computatio)
10. Björn Gambäck and Amitava Das. 2014. On measuring the complexity of code-mixing. In Proceedings of the 11th International Conference on Natural Language Processing, Goa, India, pages 1–7. Citeseer. [find](https://scholar.google.com/scholar?q=Bjo%CC%88rn+Gamba%CC%88ck+and+Amitava+Das%2E+2014%2E+On+measuring+the+complexity+of+code%2Dmixing%2E+In+Proceedings+of+the+11th+International+Conference+on+Natural+Language+Processing%2C+Goa%2C+India%2C+pages+1%E2%80%937%2E+Citeseer%2E)
11. Goran Glavas, Robert Litschko, Sebastian Ruder, and Ivan Vulic. 2019. How to (properly) evaluate cross-lingual word embeddings: On strong baselines, comparative analyses, and some misconceptions. arXiv preprint arXiv:1902.00508. [find](https://scholar.google.com/scholar?q=Goran+Glavas%2C+Robert+Litschko%2C+Sebastian+Ruder%2C+and+Ivan+Vulic%2E+2019%2E+How+to+%28properly%29+evaluate+cross%2Dlingual+word+embeddings%3A+On+strong+baselines%2C+comparative+analyses%2C+and+some+misconceptions%2E+arXi)
12. Gualberto A Guzmán, Joseph Ricard, Jacqueline Serigos, Barbara E Bullock, and Almeida Jacqueline Toribio. 2017. Metrics for modeling code-switching across corpora. In INTERSPEECH, pages 67–71. [find](https://scholar.google.com/scholar?q=Gualberto+A+Guzma%CC%81n%2C+Joseph+Ricard%2C+Jacqueline+Serigos%2C+Barbara+E+Bullock%2C+and+Almeida+Jacqueline+Toribio%2E+2017%2E+Metrics+for+modeling+code%2Dswitching+across+corpora%2E+In+INTERSPEECH%2C+pages+67%E2%80%9371%2E)
13. Karl Moritz Hermann and Phil Blunsom. 2014. Multilingual models for compositional distributed semantics. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 58–68, Baltimore, Maryland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Karl+Moritz+Hermann+and+Phil+Blunsom%2E+2014%2E+Multilingual+models+for+compositional+distributed+semantics%2E+In+Proceedings+of+the+52nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Vol) [our card](https://md.research.elamur.ai/free/en/papers/hermann-2014-multilingual-models-compositional-distributed-semantics/)
14. Anupam Jamatia, Björn Gambäck, and Amitava Das. 2016. Collecting and annotating indian social media code-mixed corpora. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 406–417. Springer. [find](https://scholar.google.com/scholar?q=Anupam+Jamatia%2C+Bjo%CC%88rn+Gamba%CC%88ck%2C+and+Amitava+Das%2E+2016%2E+Collecting+and+annotating+indian+social+media+code%2Dmixed+corpora%2E+In+International+Conference+on+Intelligent+Text+Processing+and+Computational+L)
15. Simran Khanuja, Sandipan Dandapat, Sunayana Sitaram, and Monojit Choudhury. 2020. A new dataset for natural language inference from code-mixed conversations. [find](https://scholar.google.com/scholar?q=Simran+Khanuja%2C+Sandipan+Dandapat%2C+Sunayana+Sitaram%2C+and+Monojit+Choudhury%2E+2020%2E+A+new+dataset+for+natural+language+inference+from+code%2Dmixed+conversations%2E)
16. Anoop Kunchukuttan, Pratik Mehta, and Pushpak Bhattacharyya. 2018. The iit bombay english-hindi parallel corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018). [find](https://scholar.google.com/scholar?q=Anoop+Kunchukuttan%2C+Pratik+Mehta%2C+and+Pushpak+Bhattacharyya%2E+2018%2E+The+iit+bombay+english%2Dhindi+parallel+corpus%2E+In+Proceedings+of+the+Eleventh+International+Conference+on+Language+Resources+and+Evalu) [our card](https://md.research.elamur.ai/free/en/papers/kunchukuttan-2018-iit-bombay-english-hindi-parallel/)
17. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360. [find](https://scholar.google.com/scholar?q=Guillaume+Lample%2C+Miguel+Ballesteros%2C+Sandeep+Subramanian%2C+Kazuya+Kawakami%2C+and+Chris+Dyer%2E+2016%2E+Neural+architectures+for+named+entity+recognition%2E+arXiv+preprint+arXiv%3A1603%2E01360%2E)
18. Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Bilingual word representations with monolingual quality in mind. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pages 151–159. [find](https://scholar.google.com/scholar?q=Thang+Luong%2C+Hieu+Pham%2C+and+Christopher+D+Manning%2E+2015%2E+Bilingual+word+representations+with+monolingual+quality+in+mind%2E+In+Proceedings+of+the+1st+Workshop+on+Vector+Space+Modeling+for+Natural+Langua) [our card](https://md.research.elamur.ai/free/en/papers/luong-2015-bilingual-word-representations-monolingual-quality/)
19. Rishiraj Saha Roy, Monojit Choudhury, Prasenjit Majumder, and Komal Agarwal. 2013. Overview of the fire 2013 track on transliterated search. In Post-Proceedings of the 4th and 5th Workshops of the Forum for Information Retrieval Evaluation, page 4. ACM. [find](https://scholar.google.com/scholar?q=Rishiraj+Saha+Roy%2C+Monojit+Choudhury%2C+Prasenjit+Majumder%2C+and+Komal+Agarwal%2E+2013%2E+Overview+of+the+fire+2013+track+on+transliterated+search%2E+In+Post%2DProceedings+of+the+4th+and+5th+Workshops+of+the+For)
20. Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2017. A survey of cross-lingual word embedding models. arXiv preprint arXiv:1706.04902. [find](https://scholar.google.com/scholar?q=Sebastian+Ruder%2C+Ivan+Vulic%CC%81%2C+and+Anders+S%C3%B8gaard%2E+2017%2E+A+survey+of+cross%2Dlingual+word+embedding+models%2E+arXiv+preprint+arXiv%3A1706%2E04902%2E)
21. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603. [find](https://scholar.google.com/scholar?q=Minjoon+Seo%2C+Aniruddha+Kembhavi%2C+Ali+Farhadi%2C+and+Hannaneh+Hajishirzi%2E+2016%2E+Bidirectional+attention+flow+for+machine+comprehension%2E+arXiv+preprint+arXiv%3A1611%2E01603%2E) [our card](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/)
22. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Ilya+Sutskever%2C+Kai+Chen%2C+Greg+S+Corrado%2C+and+Jeff+Dean%2E+2013%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+In+Advances+in+neural+information+processing+)
23. Royal Sequiera, Monojit Choudhury, Parth Gupta, Paolo Rosso, Shubham Kumar, Somnath Banerjee, Sudip Kumar Naskar, Sivaji Bandyopadhyay, Gokul Chittaranjan, Amitava Das, et al. 2015. Overview of fire-2015 shared task on mixed script information retrieval. In FIRE Workshops, volume 1587, pages 19–25. [find](https://scholar.google.com/scholar?q=Royal+Sequiera%2C+Monojit+Choudhury%2C+Parth+Gupta%2C+Paolo+Rosso%2C+Shubham+Kumar%2C+Somnath+Banerjee%2C+Sudip+Kumar+Naskar%2C+Sivaji+Bandyopadhyay%2C+Gokul+Chittaranjan%2C+Amitava+Das%2C+et+al%2E+2015%2E+Overview+of+fire%2D2)
24. Giovanni Molina, Nicolas Rey-Villamizar, Thamar Solorio, Fahad AlGhamdi, Mahmoud Ghoneim, Abdelati Hawwari, and Mona Diab. 2016. Overview for the second shared task on language identification in code-switched data. [find](https://scholar.google.com/scholar?q=Giovanni+Molina%2C+Nicolas+Rey%2DVillamizar%2C+Thamar+Solorio%2C+Fahad+AlGhamdi%2C+Mahmoud+Ghoneim%2C+Abdelati+Hawwari%2C+and+Mona+Diab%2E+2016%2E+Overview+for+the+second+shared+task+on+language+identification+in+code%2D)
25. Vinay Singh, Deepanshu Vijay, Syed Sarfaraz Akhtar, and Manish Shrivastava. 2018. Named entity recognition for hindi-english code-mixed social media text. In Proceedings of the Seventh Named Entities Workshop, pages 27–35. [find](https://scholar.google.com/scholar?q=Vinay+Singh%2C+Deepanshu+Vijay%2C+Syed+Sarfaraz+Akhtar%2C+and+Manish+Shrivastava%2E+2018%2E+Named+entity+recognition+for+hindi%2Denglish+code%2Dmixed+social+media+text%2E+In+Proceedings+of+the+Seventh+Named+Entities+)
26. Braja Gopal Patra, Dipankar Das, and Amitava Das. 2018. Sentiment analysis of code-mixed indian languages: An overview of sail code-mixed shared task@ icon-2017. arXiv preprint arXiv:1803.06745. — [arXiv:1803.06745](https://arxiv.org/abs/1803.06745) [find](https://scholar.google.com/scholar?q=Braja+Gopal+Patra%2C+Dipankar+Das%2C+and+Amitava+Das%2E+2018%2E+Sentiment+analysis+of+code%2Dmixed+indian+languages%3A+An+overview+of+sail+code%2Dmixed+shared+task%40+icon%2D2017%2E+arXiv+preprint+arXiv%3A1803%2E06745%2E+%E2%80%94+%5Bar)
27. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual bert? arXiv preprint arXiv:1906.01502. [find](https://scholar.google.com/scholar?q=Telmo+Pires%2C+Eva+Schlinger%2C+and+Dan+Garrette%2E+2019%2E+How+multilingual+is+multilingual+bert%3F+arXiv+preprint+arXiv%3A1906%2E01502%2E) [our card](https://md.research.elamur.ai/free/en/papers/pires-2019-how-multilingual-is-multilingual-bert/)
28. Adithya Pratapa, Gayatri Bhat, Monojit Choudhury, Sunayana Sitaram, Sandipan Dandapat, and Kalika Bali. 2018a. Language modeling for code-mixing: The role of linguistic theory based synthetic data. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1:… [find](https://scholar.google.com/scholar?q=Adithya+Pratapa%2C+Gayatri+Bhat%2C+Monojit+Choudhury%2C+Sunayana+Sitaram%2C+Sandipan+Dandapat%2C+and+Kalika+Bali%2E+2018a%2E+Language+modeling+for+code%2Dmixing%3A+The+role+of+linguistic+theory+based+synthetic+data%2E+In) [our card](https://md.research.elamur.ai/free/en/papers/pratapa-2018-language-modeling-code-mixing-role/)
29. Adithya Pratapa, Monojit Choudhury, and Sunayana Sitaram. 2018b. Word embeddings for code-mixed language processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3067–3072. [find](https://scholar.google.com/scholar?q=Adithya+Pratapa%2C+Monojit+Choudhury%2C+and+Sunayana+Sitaram%2E+2018b%2E+Word+embeddings+for+code%2Dmixed+language+processing%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Proc)
30. Pattabhi RK Rao and Sobha Lalitha Devi. 2016. Cmee-il: Code mix entity extraction in indian languages from social media text@ fire 2016-an overview. In FIRE (Working Notes), pages 289–295. [find](https://scholar.google.com/scholar?q=Pattabhi+RK+Rao+and+Sobha+Lalitha+Devi%2E+2016%2E+Cmee%2Dil%3A+Code+mix+entity+extraction+in+indian+languages+from+social+media+text%40+fire+2016%2Dan+overview%2E+In+FIRE+%28Working+Notes%29%2C+pages+289%E2%80%93295%2E)
31. Shruti Rijhwani, Royal Sequiera, Monojit Choudhury, Kalika Bali, and Chandra Shekhar Maddila. 2017. Estimating code-switching on twitter with a novel generalized word-level language detection technique. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volum… [find](https://scholar.google.com/scholar?q=Shruti+Rijhwani%2C+Royal+Sequiera%2C+Monojit+Choudhury%2C+Kalika+Bali%2C+and+Chandra+Shekhar+Maddila%2E+2017%2E+Estimating+code%2Dswitching+on+twitter+with+a+novel+generalized+word%2Dlevel+language+detection+techniqu)
32. Thamar Solorio, Elizabeth Blair, Suraj Maharjan, Steven Bethard, Mona Diab, Mahmoud Ghoneim, Abdelati Hawwari, Fahad AlGhamdi, Julia Hirschberg, Alison Chang, et al. 2014. Overview for the first shared task on language identification in code-switched data. In Proceedings of the First Workshop on Com… [find](https://scholar.google.com/scholar?q=Thamar+Solorio%2C+Elizabeth+Blair%2C+Suraj+Maharjan%2C+Steven+Bethard%2C+Mona+Diab%2C+Mahmoud+Ghoneim%2C+Abdelati+Hawwari%2C+Fahad+AlGhamdi%2C+Julia+Hirschberg%2C+Alison+Chang%2C+et+al%2E+2014%2E+Overview+for+the+first+share) [our card](https://md.research.elamur.ai/free/en/papers/solorio-2014-overview-first-shared-task-language/)
33. Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to fine-tune bert for text classification? arXiv preprint arXiv:1905.05583. [find](https://scholar.google.com/scholar?q=Chi+Sun%2C+Xipeng+Qiu%2C+Yige+Xu%2C+and+Xuanjing+Huang%2E+2019%2E+How+to+fine%2Dtune+bert+for+text+classification%3F+arXiv+preprint+arXiv%3A1905%2E05583%2E) [our card](https://md.research.elamur.ai/free/en/papers/sun-2019-fine-tune-bert-text-classification/)
34. David Vilares, Miguel A Alonso, and Carlos Gómez-Rodrı́guez. 2016. En-es-cs: An english-spanish code-switching twitter corpus for multilingual sentiment analysis. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4149–4153. [find](https://scholar.google.com/scholar?q=David+Vilares%2C+Miguel+A+Alonso%2C+and+Carlos+Go%CC%81mez%2DRodr%C4%B1%CC%81guez%2E+2016%2E+En%2Des%2Dcs%3A+An+english%2Dspanish+code%2Dswitching+twitter+corpus+for+multilingual+sentiment+analysis%2E+In+Proceedings+of+the+Tenth+Internat)
35. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Yada+Pruksachatkun%2C+Nikita+Nangia%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R+Bowman%2E+2019%2E+Superglue%3A+A+stickier+benchmark+for+general%2Dpurpose+language+understand) [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
36. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R+Bowman%2E+2018%2E+Glue%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+arXiv+preprint+arXi) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
37. Genta Indra Winata, Zhaojiang Lin, Jamin Shin, Zihan Liu, and Pascale Fung. 2019. Hierarchical meta-embeddings for code-switching named entity recognition. arXiv preprint arXiv:1909.08504. [find](https://scholar.google.com/scholar?q=Genta+Indra+Winata%2C+Zhaojiang+Lin%2C+Jamin+Shin%2C+Zihan+Liu%2C+and+Pascale+Fung%2E+2019%2E+Hierarchical+meta%2Dembeddings+for+code%2Dswitching+named+entity+recognition%2E+arXiv+preprint+arXiv%3A1909%2E08504%2E)