---
title: "LinCE: A Centralized Benchmark for Linguistic — Aguilar 2020"
description: "Research on code-switching, where speakers mix two languages inside one sentence or post, has been fragmented."
authors: ["Gustavo Aguilar", "Sudipta Kar", "Thamar Solorio"]
year: 2020
venue: "Proceedings of the Twelfth Language Resources and Evaluation Conference"
source: "https://aclanthology.org/2020.lrec-1.223/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/aguilar-2020-lince-centralized-benchmark-linguistic-code/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/aguilar-2020-lince-centralized-benchmark-linguistic-code/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) / LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation

# LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation

**Authors:** [Gustavo Aguilar](https://md.research.elamur.ai/free/en/authors/gustavo-aguilar/), [Sudipta Kar](https://md.research.elamur.ai/free/en/authors/sudipta-kar/), [Thamar Solorio](https://md.research.elamur.ai/free/en/authors/thamar-solorio/) · **Year:** 2020 · **Venue:** Proceedings of the Twelfth Language Resources and Evaluation Conference · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.lrec-1.223/) · [PDF](https://aclanthology.org/2020.lrec-1.223.pdf) · [OpenAlex](https://openalex.org/W3023437581)

**Citations:** 4 citations (openalex-snapshot · 2026-10-14) · [92](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Research on code-switching, where speakers mix two languages inside one sentence or post, has been fragmented. Each method tends to be tested on its own language pair, domain and dataset. As a result, it is unclear whether a model carries over to another pair or task, and comparing work usually means re-implementing it.

LinCE addresses this by gathering ten existing social-media corpora into one benchmark. It covers four language pairs (Spanish-English, Nepali-English, Hindi-English, Modern Standard Arabic-Egyptian Arabic) and four tasks: token-level language identification, POS tagging, NER and sentiment analysis. The authors measure how mixed each corpus is with the code-mixing index (CMI). They also audit the original releases, document five recurring problems, and re-stratify nine of the ten datasets into new, fairer splits. On top of this, they report baseline scores for LSTM, ELMo and multilingual BERT and host a public leaderboard at ritual.uh.edu/lince that scores submissions online.

The value is a shared yardstick that pushes methods to be judged across many pairs and tasks, not on one convenient corpus.

## Key points
- Ten datasets in total: four LID, two POS, three NER and one SA. SA is covered only for SPA-ENG, using the corpus of Patwa et al. (2020) from SemEval-2020 Task 9.
- Corpora differ strongly in how much they mix languages:
  - The MSA-EA LID corpus has the lowest overall CMI (2.82), and only 1,326 of its 11,243 posts are code-switched.
  - NEP-ENG LID reaches 19.85, and SPA-ENG POS reaches 24.19.
- Corpus size varies widely, from the SPA-ENG NER corpus of Aguilar et al. (2018) at 808,663 tokens down to HIN-ENG POS at 1,489 posts and 33,010 tokens.
- Label schemes reuse existing standards:
  - **LID:** CALCS labels (lang1, lang2, mixed, ambiguous, fw, ne, unk, other).
  - **POS:** the UPOS tagset of Petrov et al. (2012), extended with PRON_WH and UNK.
  - **NER:** BIO tags with CALCS entity types (person, organization, location, group, product, title, event, time, other).
- Pair-specific difficulties:
  - Spanish and English share many Latin/French-derived words, which confuses character-based LID.
  - English verbs conjugated by Spanish rules (Spanglish) raise the OOV rate.
  - Romanized Hindi and Nepali have no standard transliteration.

## Interesting ideas
- Each corpus reports its mixing index twice, once over all posts and once over only the code-switched ones, so difficulty is measured separately from size and from the share of easy items. — For Mova (learn): evaluation sets could carry a difficulty measure (e.g. share of non-projective arcs or rare constructions), with scores given for the whole set and for the hard subset, so gains on hard sentences are not masked by easy ones.
- Rather than trusting the original releases, the authors audited them for flaws and re-drew train/dev/test splits with stratification before benchmarking. — For Mova (tree): the treebanks behind the perceptron parser and gparse could be audited for duplicates, near-duplicates and genre imbalance across splits, and re-split where needed, so reported LAS reflects real generalization.
- Sources are harmonized by adopting a universal tagset and adding only a few clearly documented extensions (PRON_WH, UNK), instead of inventing a fresh scheme. — For Mova (tree): UD dialect conversion could keep the UD core intact and record each local extension label with an explicit mapping back to plain UD, so converted data stays comparable with other UD resources.
- The LID scheme gives explicit "cannot decide" labels (ambiguous, mixed, unk) instead of forcing every token into one class. — For Mova (tokens): the tagger and morphology could emit an explicit ambiguous or unknown state with its candidate readings, keeping the decision explainable and letting later stages resolve it from context.
- A single held-out test set per task, scored centrally across all tasks and pairs, stops systems from reporting only their best-looking subset. — For Mova (answer): all reader evaluations (school tests, maths word problems, fable QA) could feed one fixed scoreboard with hidden answers, so a change that helps one reader but hurts another shows up immediately.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [Named entity recognition](https://md.research.elamur.ai/free/en/topics/named-entity-recognition/) · [Part-of-speech tagging](https://md.research.elamur.ai/free/en/topics/part-of-speech-tagging/)

## Cited by

Cited by

92 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 89 not yet.

- [GlotLID: Language Identification for Low-Resource Languages](https://md.research.elamur.ai/free/en/papers/kargaran-2023-glotlid-language-identification-low-resource/)
- [Parsing the Switch: LLM-Based UD Annotation for Complex Code-Switched and Low-Resource Languages](https://md.research.elamur.ai/free/en/papers/kellert-2025-parsing-switch-llm-based-ud/)
- [Detecting Latin in Historical Books with Large Language Models: A Multimodal Benchmark](https://md.research.elamur.ai/free/en/papers/wu-2026-detecting-latin-historical-books-large/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{aguilar2020lince,
  title = {{LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation}},
  author = {Aguilar, Gustavo and Kar, Sudipta and Solorio, Thamar},
  booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference},
  year = {2020},
  url = {https://aclanthology.org/2020.lrec-1.223/}
}
```

- [← previous Lemmatization and POS-tagging process by using joint learning approach. Experimental results on Classical Armenian, Old Georgian, and Syriac](https://md.research.elamur.ai/free/en/papers/aclbulk-2020-gorene-lemmatization-pos-tagging-process-joint/)
- [next → Linear Neural Parsing and Hybrid Enhancement for Enhanced Universal Dependencies](https://md.research.elamur.ai/free/en/papers/attardi-2020-linear-neural-parsing-hybrid-enhancement/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/)

## References

53 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aguilar, G. and Solorio, T. (2019). From English to Code-Switching: Transfer Learning with Strong Morphological Clues. arXiv preprint arXiv:1909.05158. [↗](https://arxiv.org/abs/1909.05158)
2. Aguilar, G., AlGhamdi, F., Soto, V., Diab, M., Hirschberg, J., and Solorio, T. (2018). Named Entity Recognition on Code-Switched Data: Overview of the CALCS 2018 Shared Task. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, pages 138–147, Melbourne, Australia, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Aguilar%2C+G%2E%2C+AlGhamdi%2C+F%2E%2C+Soto%2C+V%2E%2C+Diab%2C+M%2E%2C+Hirschberg%2C+J%2E%2C+and+Solorio%2C+T%2E+%282018%29%2E+Named+Entity+Recognition+on+Code%2DSwitched+Data%3A+Overview+of+the+CALCS+2018+Shared+Task%2E+In+Proceedings+of+the+Thi)
3. Al-Badrashiny, M. and Diab, M. (2016). LILI: A Simple Language Independent Approach for Language Identification. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1211–1219, Osaka, Japan, December. [find](https://scholar.google.com/scholar?q=Al%2DBadrashiny%2C+M%2E+and+Diab%2C+M%2E+%282016%29%2E+LILI%3A+A+Simple+Language+Independent+Approach+for+Language+Identification%2E+In+Proceedings+of+COLING+2016%2C+the+26th+International+Conference+on+Computational+Lingu)
4. The COLING 2016 Organizing Committee. [find](https://scholar.google.com/scholar?q=The+COLING+2016+Organizing+Committee%2E)
5. AlGhamdi, F., Molina, G., Diab, M., Solorio, T., Hawwari, A., Soto, V., and Hirschberg, J. (2019). Part-of-Speech Tagging for Code-Switched Data. arXiv preprint arXiv:1909.13006. [↗](https://arxiv.org/abs/1909.13006)
6. Bali, K., Sharma, J., Choudhury, M., and Vyas, Y. (2014). “I am borrowing ya mixing ?” An Analysis of English-Hindi Code Mixing in Facebook. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 116–126, Doha, Qatar, October. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bali%2C+K%2E%2C+Sharma%2C+J%2E%2C+Choudhury%2C+M%2E%2C+and+Vyas%2C+Y%2E+%282014%29%2E+%E2%80%9CI+am+borrowing+ya+mixing+%3F%E2%80%9D+An+Analysis+of+English%2DHindi+Code+Mixing+in+Facebook%2E+In+Proceedings+of+the+First+Workshop+on+Computational+Appro)
7. Chandu, K., Loginova, E., Gupta, V., Genabith, J. v., Neumann, G., Chinnakotla, M., Nyberg, E., and Black, A. W. (2018). Code-Mixed Question Answering Challenge: Crowd-sourcing Data and Techniques. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, pages 29–38, Melbourne, Australia, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chandu%2C+K%2E%2C+Loginova%2C+E%2E%2C+Gupta%2C+V%2E%2C+Genabith%2C+J%2E+v%2E%2C+Neumann%2C+G%2E%2C+Chinnakotla%2C+M%2E%2C+Nyberg%2C+E%2E%2C+and+Black%2C+A%2E+W%2E+%282018%29%2E+Code%2DMixed+Question+Answering+Challenge%3A+Crowd%2Dsourcing+Data+and+Techniques%2E+In)
8. Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., and Robinson, T. (2013). One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling. Technical report, Google. [find](https://scholar.google.com/scholar?q=Chelba%2C+C%2E%2C+Mikolov%2C+T%2E%2C+Schuster%2C+M%2E%2C+Ge%2C+Q%2E%2C+Brants%2C+T%2E%2C+Koehn%2C+P%2E%2C+and+Robinson%2C+T%2E+%282013%29%2E+One+Billion+Word+Benchmark+for+Measuring+Progress+in+Statistical+Language+Modeling%2E+Technical+report%2C+Goo) [our card](https://md.research.elamur.ai/free/en/papers/chelba-2013-one-billion-word-benchmark-measuring/)
9. Choudhury, M., Chittaranjan, G., Gupta, P., and Das, A. (2014). Overview of FIRE 2014 Track on Transliterated Search. Proceedings of FIRE, pages 68–89. [find](https://scholar.google.com/scholar?q=Choudhury%2C+M%2E%2C+Chittaranjan%2C+G%2E%2C+Gupta%2C+P%2E%2C+and+Das%2C+A%2E+%282014%29%2E+Overview+of+FIRE+2014+Track+on+Transliterated+Search%2E+Proceedings+of+FIRE%2C+pages+68%E2%80%9389%2E)
10. Das, A. (2016). Tool contest on POS tagging for code-mixed Indian social media (Facebook, Twitter, and Whatsapp) text. retrieved 05-10-2019. [find](https://scholar.google.com/scholar?q=Das%2C+A%2E+%282016%29%2E+Tool+contest+on+POS+tagging+for+code%2Dmixed+Indian+social+media+%28Facebook%2C+Twitter%2C+and+Whatsapp%29+text%2E+retrieved+05%2D10%2D2019%2E)
11. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805. [↗](https://arxiv.org/abs/1810.04805)
12. Ferguson, C. A. (1959). Diglossia. word, 15(2):325–340. [find](https://scholar.google.com/scholar?q=Ferguson%2C+C%2E+A%2E+%281959%29%2E+Diglossia%2E+word%2C+15%282%29%3A325%E2%80%93340%2E)
13. Gambäck, B. and Das, A. (2014). On Measuring the Complexity of Code-mixing. In Proceedings of the 11th International Conference on Natural Language Processing, Goa, India, pages 1–7. Citeseer. [find](https://scholar.google.com/scholar?q=Gamba%CC%88ck%2C+B%2E+and+Das%2C+A%2E+%282014%29%2E+On+Measuring+the+Complexity+of+Code%2Dmixing%2E+In+Proceedings+of+the+11th+International+Conference+on+Natural+Language+Processing%2C+Goa%2C+India%2C+pages+1%E2%80%937%2E+Citeseer%2E)
14. Gardner, M., Grus, J., Neumann, M., Tafjord, O., Dasigi, P., Liu, N., Peters, M., Schmitz, M., and Zettlemoyer, L. (2018). AllenNLP: A Deep Semantic Natural Language Processing Platform. arXiv preprint arXiv:1803.07640. [↗](https://arxiv.org/abs/1803.07640) [our card](https://md.research.elamur.ai/free/en/papers/gardner-2018-allennlp-deep-semantic-natural-language/)
15. Geetha, P., Chandu, K., and Black, A. W. (2018). Tackling Code-Switched NER: Participation of CMU. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, pages 126–131, Melbourne, Australia, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Geetha%2C+P%2E%2C+Chandu%2C+K%2E%2C+and+Black%2C+A%2E+W%2E+%282018%29%2E+Tackling+Code%2DSwitched+NER%3A+Participation+of+CMU%2E+In+Proceedings+of+the+Third+Workshop+on+Computational+Approaches+to+Linguistic+Code%2DSwitching%2C+pages+)
16. Hamed, I., Elmahdy, M., and Abdennadher, S. (2018). Collection and Analysis of Code-switch Egyptian Arabic-English Speech Corpus. In Nicoletta Calzolari (Conference chair), et al., editors, Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 7-12, 2018. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Hamed%2C+I%2E%2C+Elmahdy%2C+M%2E%2C+and+Abdennadher%2C+S%2E+%282018%29%2E+Collection+and+Analysis+of+Code%2Dswitch+Egyptian+Arabic%2DEnglish+Speech+Corpus%2E+In+Nicoletta+Calzolari+%28Conference+chair%29%2C+et+al%2E%2C+editors%2C+Proceeding)
17. Hochreiter, S. and Schmidhuber, J. (1997). Long Short-Term Memory. Neural computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Hochreiter%2C+S%2E+and+Schmidhuber%2C+J%2E+%281997%29%2E+Long+Short%2DTerm+Memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2E)
18. Jamatia, A., Gambäck, B., and Das, A. (2015). Part-of-speech tagging for code-mixed English-Hindi twitter and Facebook chat messages. In Proceedings of the International Conference Recent Advances in Natural Language Processing, pages 239–248, Hissar, Bulgaria, September. INCOMA Ltd. Shoumen, BULGARIA. [find](https://scholar.google.com/scholar?q=Jamatia%2C+A%2E%2C+Gamba%CC%88ck%2C+B%2E%2C+and+Das%2C+A%2E+%282015%29%2E+Part%2Dof%2Dspeech+tagging+for+code%2Dmixed+English%2DHindi+twitter+and+Facebook+chat+messages%2E+In+Proceedings+of+the+International+Conference+Recent+Advances+in)
19. Khandelwal, A., Swami, S., Akhtar, S. S., and Shrivastava, M. (2018). Humor detection in English-Hindi code-mixed social media content : Corpus and baseline system. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Khandelwal%2C+A%2E%2C+Swami%2C+S%2E%2C+Akhtar%2C+S%2E+S%2E%2C+and+Shrivastava%2C+M%2E+%282018%29%2E+Humor+detection+in+English%2DHindi+code%2Dmixed+social+media+content+%3A+Corpus+and+baseline+system%2E+In+Proceedings+of+the+Eleventh+Inte)
20. Loshchilov, I. and Hutter, F. (2019). Decoupled weight decay regularization. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Loshchilov%2C+I%2E+and+Hutter%2C+F%2E+%282019%29%2E+Decoupled+weight+decay+regularization%2E+In+International+Conference+on+Learning+Representations%2E)
21. Mager, M., Çetinog̃lu, Ö., and Kann, K. (2019). Subword-level language identification for intra-word code-switching. Proceedings of the 2019 Conference of the North. [find](https://scholar.google.com/scholar?q=Mager%2C+M%2E%2C+C%CC%A7etinog%CC%83lu%2C+O%CC%88%2E%2C+and+Kann%2C+K%2E+%282019%29%2E+Subword%2Dlevel+language+identification+for+intra%2Dword+code%2Dswitching%2E+Proceedings+of+the+2019+Conference+of+the+North%2E)
22. Mave, D., Maharjan, S., and Solorio, T. (2018). Language identification and analysis of code-switched social media text. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, pages 51–61, Melbourne, Australia, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mave%2C+D%2E%2C+Maharjan%2C+S%2E%2C+and+Solorio%2C+T%2E+%282018%29%2E+Language+identification+and+analysis+of+code%2Dswitched+social+media+text%2E+In+Proceedings+of+the+Third+Workshop+on+Computational+Approaches+to+Linguistic+)
23. Molina, G., AlGhamdi, F., Ghoneim, M., Hawwari, A., Rey-Villamizar, N., Diab, M., and Solorio, T. (2016). Overview for the Second Shared Task on Language Identification in Code-Switched Data. In Proceedings of the Second Workshop on Computational Approaches to Code Switching, pages 40–49, Austin, Texas, November. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Molina%2C+G%2E%2C+AlGhamdi%2C+F%2E%2C+Ghoneim%2C+M%2E%2C+Hawwari%2C+A%2E%2C+Rey%2DVillamizar%2C+N%2E%2C+Diab%2C+M%2E%2C+and+Solorio%2C+T%2E+%282016%29%2E+Overview+for+the+Second+Shared+Task+on+Language+Identification+in+Code%2DSwitched+Data%2E+In+Proce)
24. Partanen, N., Lim, K., Rießler, M., and Poibeau, T. (2018). Dependency parsing of code-switching data with cross-lingual feature representations. In Proceedings of the Fourth International Workshop on Computational Linguistics of Uralic Languages, pages 1–17, Helsinki, Finland, January. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Partanen%2C+N%2E%2C+Lim%2C+K%2E%2C+Rie%C3%9Fler%2C+M%2E%2C+and+Poibeau%2C+T%2E+%282018%29%2E+Dependency+parsing+of+code%2Dswitching+data+with+cross%2Dlingual+feature+representations%2E+In+Proceedings+of+the+Fourth+International+Workshop+on)
25. Patwa, P., Aguilar, G., Kar, S., Pandey, S., PYKL, S., Garrette, D., Gambäck, B., Chakraborty, T., Solorio, T., and Das, A. (2020). SemEval-2020 Sentimix Task 9: Overview of SENTIment Analysis of Code-MIXed Tweets. In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain, September. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Patwa%2C+P%2E%2C+Aguilar%2C+G%2E%2C+Kar%2C+S%2E%2C+Pandey%2C+S%2E%2C+PYKL%2C+S%2E%2C+Garrette%2C+D%2E%2C+Gamba%CC%88ck%2C+B%2E%2C+Chakraborty%2C+T%2E%2C+Solorio%2C+T%2E%2C+and+Das%2C+A%2E+%282020%29%2E+SemEval%2D2020+Sentimix+Task+9%3A+Overview+of+SENTIment+Analysis+of+Cod)
26. Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana, June. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Peters%2C+M%2E%2C+Neumann%2C+M%2E%2C+Iyyer%2C+M%2E%2C+Gardner%2C+M%2E%2C+Clark%2C+C%2E%2C+Lee%2C+K%2E%2C+and+Zettlemoyer%2C+L%2E+%282018%29%2E+Deep+Contextualized+Word+Representations%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+C)
27. Petrov, S., Das, D., and McDonald, R. (2012). A universal part-of-speech tagset. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2089–2096, Istanbul, Turkey, May. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Petrov%2C+S%2E%2C+Das%2C+D%2E%2C+and+McDonald%2C+R%2E+%282012%29%2E+A+universal+part%2Dof%2Dspeech+tagset%2E+In+Proceedings+of+the+Eighth+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%E2%80%9912%29%2C+pages+2089%E2%80%932096%2C+)
28. Pratapa, A., Choudhury, M., and Sitaram, S. (2018). Word embeddings for code-mixed language processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3067–3072, Brussels, Belgium, October-November. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Pratapa%2C+A%2E%2C+Choudhury%2C+M%2E%2C+and+Sitaram%2C+S%2E+%282018%29%2E+Word+embeddings+for+code%2Dmixed+language+processing%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages)
29. Raghavi, K. C., Chinnakotla, M. K., and Shrivastava, M. (2015). Answer ka type kya he?: Learning to classify questions in code-mixed language. In Proceedings of the 24th International Conference on World Wide Web, pages 853–858. ACM. [find](https://scholar.google.com/scholar?q=Raghavi%2C+K%2E+C%2E%2C+Chinnakotla%2C+M%2E+K%2E%2C+and+Shrivastava%2C+M%2E+%282015%29%2E+Answer+ka+type+kya+he%3F%3A+Learning+to+classify+questions+in+code%2Dmixed+language%2E+In+Proceedings+of+the+24th+International+Conference+on+Wo)
30. Rothman, J. and Rell, A. B. (2007). A linguistic analysis of spanglish: Relating language to identity. Linguistics and the Human Sciences, 1(3):515–536. [find](https://scholar.google.com/scholar?q=Rothman%2C+J%2E+and+Rell%2C+A%2E+B%2E+%282007%29%2E+A+linguistic+analysis+of+spanglish%3A+Relating+language+to+identity%2E+Linguistics+and+the+Human+Sciences%2C+1%283%29%3A515%E2%80%93536%2E)
31. Roy, R. S., Choudhury, M., Majumder, P., and Agarwal, K. (2013). Overview of the FIRE 2013 Track on Transliterated Search. In Post-Proceedings of the 4th and 5th Workshops of the Forum for Information Retrieval Evaluation, page 4. ACM. [find](https://scholar.google.com/scholar?q=Roy%2C+R%2E+S%2E%2C+Choudhury%2C+M%2E%2C+Majumder%2C+P%2E%2C+and+Agarwal%2C+K%2E+%282013%29%2E+Overview+of+the+FIRE+2013+Track+on+Transliterated+Search%2E+In+Post%2DProceedings+of+the+4th+and+5th+Workshops+of+the+Forum+for+Information)
32. Samih, Y., Maharjan, S., Attia, M., Kallmeyer, L., and Solorio, T. (2016). Multilingual code-switching identification via LSTM recurrent neural networks. In Proceedings of the Second Workshop on Computational Approaches to Code Switching, pages 50–59, Austin, Texas, November. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Samih%2C+Y%2E%2C+Maharjan%2C+S%2E%2C+Attia%2C+M%2E%2C+Kallmeyer%2C+L%2E%2C+and+Solorio%2C+T%2E+%282016%29%2E+Multilingual+code%2Dswitching+identification+via+LSTM+recurrent+neural+networks%2E+In+Proceedings+of+the+Second+Workshop+on+Compu)
33. Sechidis, K., Tsoumakas, G., and Vlahavas, I. (2011). On the stratification of Multi-label Data. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 145–158. Springer. [find](https://scholar.google.com/scholar?q=Sechidis%2C+K%2E%2C+Tsoumakas%2C+G%2E%2C+and+Vlahavas%2C+I%2E+%282011%29%2E+On+the+stratification+of+Multi%2Dlabel+Data%2E+In+Joint+European+Conference+on+Machine+Learning+and+Knowledge+Discovery+in+Databases%2C+pages+145%E2%80%93158%2E+S)
34. Sequiera, R., Choudhury, M., and Bali, K. (2015a). POS Tagging of Hindi-English Code Mixed Text from Social Media: Some Machine Learning Experiments. In 2015 Proceedings of International Conference on NLP. NL-PAI, December. [find](https://scholar.google.com/scholar?q=Sequiera%2C+R%2E%2C+Choudhury%2C+M%2E%2C+and+Bali%2C+K%2E+%282015a%29%2E+POS+Tagging+of+Hindi%2DEnglish+Code+Mixed+Text+from+Social+Media%3A+Some+Machine+Learning+Experiments%2E+In+2015+Proceedings+of+International+Conference+on)
35. Sequiera, R., Choudhury, M., Gupta, P., Rosso, P., Kumar, S., Banerjee, S., Naskar, S. K., Bandyopadhyay, S., Chittaranjan, G., Das, A., et al. (2015b). Overview of FIRE-2015 Shared Task on Mixed Script Information Retrieval. In FIRE Workshops, volume 1587, pages 19– 25. [find](https://scholar.google.com/scholar?q=Sequiera%2C+R%2E%2C+Choudhury%2C+M%2E%2C+Gupta%2C+P%2E%2C+Rosso%2C+P%2E%2C+Kumar%2C+S%2E%2C+Banerjee%2C+S%2E%2C+Naskar%2C+S%2E+K%2E%2C+Bandyopadhyay%2C+S%2E%2C+Chittaranjan%2C+G%2E%2C+Das%2C+A%2E%2C+et+al%2E+%282015b%29%2E+Overview+of+FIRE%2D2015+Shared+Task+on+Mixed+Scri)
36. Singh, K., Sen, I., and Kumaraguru, P. (2018a). Language identification and named entity recognition in Hinglish code mixed tweets. In Proceedings of ACL 2018, Student Research Workshop, pages 52–58, Melbourne, Australia, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Singh%2C+K%2E%2C+Sen%2C+I%2E%2C+and+Kumaraguru%2C+P%2E+%282018a%29%2E+Language+identification+and+named+entity+recognition+in+Hinglish+code+mixed+tweets%2E+In+Proceedings+of+ACL+2018%2C+Student+Research+Workshop%2C+pages+52%E2%80%9358%2C+)
37. Singh, K., Sen, I., and Kumaraguru, P. (2018b). A twitter corpus for Hindi-English code mixed POS tagging. In Proceedings of the Sixth International Workshop on Natural Language Processing for Social Media, pages 12– 17, Melbourne, Australia, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Singh%2C+K%2E%2C+Sen%2C+I%2E%2C+and+Kumaraguru%2C+P%2E+%282018b%29%2E+A+twitter+corpus+for+Hindi%2DEnglish+code+mixed+POS+tagging%2E+In+Proceedings+of+the+Sixth+International+Workshop+on+Natural+Language+Processing+for+Social+)
38. Sitaram, S., Chandu, K. R., Rallabandi, S. K., and Black, A. W. (2019). A survey of code-switched speech and language processing. arXiv preprint arXiv:1904.00784. [↗](https://arxiv.org/abs/1904.00784)
39. Solorio, T. and Liu, Y. (2008). Part-of-speech tagging for english-spanish code-switched text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’08, pages 1051–1060, Stroudsburg, PA, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Solorio%2C+T%2E+and+Liu%2C+Y%2E+%282008%29%2E+Part%2Dof%2Dspeech+tagging+for+english%2Dspanish+code%2Dswitched+text%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+EMNLP+%E2%80%9908%2C+pages+10)
40. Solorio, T., Blair, E., Maharjan, S., Bethard, S., Diab, M., Ghoneim, M., Hawwari, A., AlGhamdi, F., Hirschberg, J., Chang, A., and Fung, P. (2014). Overview for the first shared task on language identification in code-switched data. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 62–72, Doha, Qatar, October. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Solorio%2C+T%2E%2C+Blair%2C+E%2E%2C+Maharjan%2C+S%2E%2C+Bethard%2C+S%2E%2C+Diab%2C+M%2E%2C+Ghoneim%2C+M%2E%2C+Hawwari%2C+A%2E%2C+AlGhamdi%2C+F%2E%2C+Hirschberg%2C+J%2E%2C+Chang%2C+A%2E%2C+and+Fung%2C+P%2E+%282014%29%2E+Overview+for+the+first+shared+task+on+language+iden) [our card](https://md.research.elamur.ai/free/en/papers/solorio-2014-overview-first-shared-task-language/)
41. Soto, V. and Hirschberg, J. (2017). Crowdsourcing universal part-of-speech tags for code-switching. Interspeech 2017, Aug. [find](https://scholar.google.com/scholar?q=Soto%2C+V%2E+and+Hirschberg%2C+J%2E+%282017%29%2E+Crowdsourcing+universal+part%2Dof%2Dspeech+tags+for+code%2Dswitching%2E+Interspeech+2017%2C+Aug%2E)
42. Soto, V. and Hirschberg, J. (2018). Joint part-of-speech and language ID tagging for code-switched data. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, pages 1–10, Melbourne, Australia, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Soto%2C+V%2E+and+Hirschberg%2C+J%2E+%282018%29%2E+Joint+part%2Dof%2Dspeech+and+language+ID+tagging+for+code%2Dswitched+data%2E+In+Proceedings+of+the+Third+Workshop+on+Computational+Approaches+to+Linguistic+Code%2DSwitching%2C+)
43. Tristram, H. L. C. (1999). How Celtic is Standard English? Nauka, St Petersburg, Russia. [find](https://scholar.google.com/scholar?q=Tristram%2C+H%2E+L%2E+C%2E+%281999%29%2E+How+Celtic+is+Standard+English%3F+Nauka%2C+St+Petersburg%2C+Russia%2E)
44. Trivedi, S., Rangwani, H., and Kumar Singh, A. (2018). IIT (BHU) submission for the ACL shared task on named entity recognition on code-switched data. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, pages 148–153, Melbourne, Australia, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Trivedi%2C+S%2E%2C+Rangwani%2C+H%2E%2C+and+Kumar+Singh%2C+A%2E+%282018%29%2E+IIT+%28BHU%29+submission+for+the+ACL+shared+task+on+named+entity+recognition+on+code%2Dswitched+data%2E+In+Proceedings+of+the+Third+Workshop+on+Computati)
45. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. (2017). Attention is all you need. In I. Guyon, et al., editors, Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Vaswani%2C+A%2E%2C+Shazeer%2C+N%2E%2C+Parmar%2C+N%2E%2C+Uszkoreit%2C+J%2E%2C+Jones%2C+L%2E%2C+Gomez%2C+A%2E+N%2E%2C+Kaiser%2C+L%2E+u%2E%2C+and+Polosukhin%2C+I%2E+%282017%29%2E+Attention+is+all+you+need%2E+In+I%2E+Guyon%2C+et+al%2E%2C+editors%2C+Advances+in+Neural+Info)
46. Vilares, D., Alonso, M. A., and Gómez-Rodrı́guez, C. (2015). Sentiment analysis on monolingual, multilingual and code-switching twitter corpora. In Proceedings of the 6th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 2– 8. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Vilares%2C+D%2E%2C+Alonso%2C+M%2E+A%2E%2C+and+Go%CC%81mez%2DRodr%C4%B1%CC%81guez%2C+C%2E+%282015%29%2E+Sentiment+analysis+on+monolingual%2C+multilingual+and+code%2Dswitching+twitter+corpora%2E+In+Proceedings+of+the+6th+Workshop+on+Computational+Ap)
47. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. (2018a). GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium, November. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wang%2C+A%2E%2C+Singh%2C+A%2E%2C+Michael%2C+J%2E%2C+Hill%2C+F%2E%2C+Levy%2C+O%2E%2C+and+Bowman%2C+S%2E+%282018a%29%2E+GLUE%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+In+Proceedings+of+the+2018+EMNLP+Wo) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
48. Wang, C., Cho, K., and Kiela, D. (2018b). Code-Switched Named Entity Recognition with Embedding Attention. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, pages 154– 158, Melbourne, Australia, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wang%2C+C%2E%2C+Cho%2C+K%2E%2C+and+Kiela%2C+D%2E+%282018b%29%2E+Code%2DSwitched+Named+Entity+Recognition+with+Embedding+Attention%2E+In+Proceedings+of+the+Third+Workshop+on+Computational+Approaches+to+Linguistic+Code%2DSwitching)
49. Winata, G. I., Wu, C.-S., Madotto, A., and Fung, P. (2018). Bilingual character representation for efficiently addressing out-of-vocabulary words in code-switching named entity recognition. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, pages 110–114, Melbourne, Australia, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Winata%2C+G%2E+I%2E%2C+Wu%2C+C%2E%2DS%2E%2C+Madotto%2C+A%2E%2C+and+Fung%2C+P%2E+%282018%29%2E+Bilingual+character+representation+for+efficiently+addressing+out%2Dof%2Dvocabulary+words+in+code%2Dswitching+named+entity+recognition%2E+In+Proceed)
50. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, J. (2019). Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. [↗](https://arxiv.org/abs/1910.03771) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
51. Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, S. (2015). Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Conference on Computer Vision (ICCV), December. Appendix for “LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation” A. Label Schemes A.1. LID Label Scheme We use the CALCS label scheme for language identification, which contains the following labels: • lang1 represents one of the code-switched languages. For English-paired lanaguages, this label is used for English tokens; for MSA-EA, this label is used for Modern Standard Arabic. • lang2 represents the other code-switched language. For LinCE, this label could be Spanish, Hindi, Nepali, or Egyptian Arabic, depending on the dataset. • mixed represents a word that is partly in lang1 and partly in lang2. • ambiguous is used for words whether it is unclear if they belong to one or the other language. • fw means foreign word, and it is used for words that are in a language different than lang1 or lang2. • ne represents the named entity tokens. • unk means unknown, and it is used for tokens whose language is not recognized. • other captures symbols, punctuation, and emoticons. A.2. POS Label Scheme We use the universal part-of-speech (UPOS) tagset proposed by Petrov et al. (2012) with the addition of PRON WH proposed by Singh et al. (2018b) and UNK proposed by Soto and Hirschberg (2017). • ADJ is used for adjectives. • ADP is used for prepositions and postpositions. • ADV is used for adverbs. • AUX is used for auxiliaries. • CONJ is used for coordinating conjunctions. This is represented by ‘CCONJ’ in the univeral POS tagset. • DET is used for determiners and articles. • INTJ is used for interjections. • NOUN is used for nouns. • NUM is used for numerals. • PART is used for particles. • PRON is used for pronouns. • PROPN is used for proper nouns. • PUNCT is used for punctuation marks. • SCONJ is used for subordinating conjunctions. • VERB is used for verbs. • X for all other categories such as abbreviations or foreign words. [find](https://scholar.google.com/scholar?q=Zhu%2C+Y%2E%2C+Kiros%2C+R%2E%2C+Zemel%2C+R%2E%2C+Salakhutdinov%2C+R%2E%2C+Urtasun%2C+R%2E%2C+Torralba%2C+A%2E%2C+and+Fidler%2C+S%2E+%282015%29%2E+Aligning+books+and+movies%3A+Towards+story%2Dlike+visual+explanations+by+watching+movies+and+reading+boo) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/)
52. PRON WH is used for interrogative pronouns (like where, why, etc.). This extension is employed by Singh et al. (2018b). [find](https://scholar.google.com/scholar?q=PRON+WH+is+used+for+interrogative+pronouns+%28like+where%2C+why%2C+etc%2E%29%2E+This+extension+is+employed+by+Singh+et+al%2E+%282018b%29%2E)
53. UNK is used when it is not possible to determine the syntactic category. This extension is employed by Soto and Hirschberg (2017). A.3. NER Label Scheme We use the CALCS label scheme for named entity recognition (Aguilar et al., 2018). These labels use the BIO scheme and contain the following entity types: • person for proper nouns or nicknames. • organization for institutions, companies, organizations, or corporations. Not to confuse with products when they have the same name as an organization. • location for physical places that people can visit and that have a unique name. Addresses, facilities, and touristic places are examples of this. • group for sports teams, music bands, duets, etc. Not to confuse with organization. • product for articles that have been manufactured or refine for sale, like devices, medicine, food, well-defined services. [find](https://scholar.google.com/scholar?q=UNK+is+used+when+it+is+not+possible+to+determine+the+syntactic+category%2E+This+extension+is+employed+by+Soto+and+Hirschberg+%282017%29%2E+A%2E3%2E+NER+Label+Scheme+We+use+the+CALCS+label+scheme+for+named+entity+)