---
title: "Language Model Tokenizers Introduce Unfairness Between Languages — Petrov 2023"
description: "To compare languages, they propose the notion of tokenizer parity. 112 references, cited by 149 papers here."
authors: ["Aleksandar Petrov", "Emanuele La Malfa", "Philip H. S. Torr", "Adel Bibi"]
year: 2023
venue: "arXiv:2305.15425"
source: "https://arxiv.org/abs/2305.15425"
doi: "10.48550/arxiv.2305.15425"
arxiv: "2305.15425"
licence: "CC BY-NC-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-nc-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/petrov-2023-language-model-tokenizers-introduce-unfairness/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/petrov-2023-language-model-tokenizers-introduce-unfairness/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/) / Language Model Tokenizers Introduce Unfairness Between Languages

# Language Model Tokenizers Introduce Unfairness Between Languages

**Authors:** Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr and Adel Bibi · **Year:** 2023 · **Venue:** arXiv:2305.15425 · **Licence:** [CC BY-NC-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2305.15425) · [PDF](https://arxiv.org/pdf/2305.15425) · [DOI 10.48550/arxiv.2305.15425](https://doi.org/10.48550/arxiv.2305.15425) · [arXiv 2305.15425](https://arxiv.org/abs/2305.15425)

**Citations:** [149](https://md.research.elamur.ai/free/en/papers/petrov-2023-language-model-tokenizers-introduce-unfairness/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary

The authors examine whether language model tokenizers treat different languages equally. They show that inequality arises even before the model receives the text: already at the stage where the text is split into tokens. To compare languages, they propose the notion of tokenizer parity. The same sentence is taken in different languages, and one computes how many times longer its tokenization is in one language than in another. The authors call this ratio the "premium". The data come from the parallel corpus FLORES-200: 2000 sentences from Wikipedia that were translated by humans into 200 languages. The study covers English-centric tokenizers (GPT-2, RoBERTa, cl100k_base in ChatGPT and GPT-4, FlanT5), models built for individual languages, multilingual tokenizers, as well as character-level and byte-level encodings. None of the approaches comes close to parity. The consequences are quite practical: more expensive commercial APIs, longer processing, and less useful text in the context window for speakers of "disadvantaged" languages. The authors call for training future models on multilingually fair tokenizers and propose how to build them. The work also concerns Cyrillic-script languages: Bulgarian is among the noticeably "more expensive" ones.

## Key points

- In cl100k_base (ChatGPT, GPT-4), the same text takes approximately 1.6 times more tokens in Italian than in English, 2.6 times more in Bulgarian, three times more in Arabic, and up to 15 times more in Shan. Even Portuguese, the closest to English, requires approximately 50% more tokens.
- Byte-level encoding is also unequal: for Burmese and Tibetan, the text comes out more than 4 times longer than for Chinese. In tokenizers trained specifically for many languages, the gap reaches 13 times.
- FlanT5 turns more than 10% of characters into UNK for 42% of languages. MuRIL, created for 16 Indian languages, keeps the premium relative to English within 1.0–1.75.
- According to the authors' estimate, because of per-token or per-character pricing, users of some languages face a cost at least 2.5 times higher for an identical task. Processing can also take twice as long.
- The authors propose building a tokenizer in two steps. First, train a separate tokenizer for each language. Then merge them, starting with 256 byte tokens and each time adding the most frequent token of the language with the highest premium. The losses for English are small: if the cl100k_base vocabulary is reduced to a third, English sequences become only approximately 10% longer.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

💡 Concepts (1): [✂️ Subword tokenization](https://md.research.elamur.ai/free/en/concepts/subword-tokenization/) · 30 works

📚 Shelves (2): [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 7,418 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 7,357 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,079 works · [✂️ Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/) · 68 works

## Cite

Cite as: Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr and Adel Bibi. 2023. Language Model Tokenizers Introduce Unfairness Between Languages. arXiv:2305.15425.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{petrov2023language,
  title = {{Language Model Tokenizers Introduce Unfairness Between Languages}},
  author = {Aleksandar Petrov, Emanuele {La Malfa}, Philip H. S. Torr and Adel Bibi},
  year = {2023},
  doi = {10.48550/arxiv.2305.15425},
  eprint = {2305.15425},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.15425}
}
```

- [← previous Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation](https://md.research.elamur.ai/free/en/papers/mundler-2023-self-contradictory-hallucinations-large-language/)
- [next → MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages](https://md.research.elamur.ai/free/en/papers/zhang-2023-miracl-multilingual-retrieval-dataset-covering/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/)

## References

112 references, 40 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ahmed Abdelali, Francisco Guzman, Hassan Sajjad, and Stephan Vogel. 2014. The AMARA corpus: Building parallel language resources for the educational domain. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14). European Language Resources Association (E… [our card](https://md.research.elamur.ai/free/en/papers/abdelali-2014-amara-corpus-building-parallel-language/)
2. Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R. Mortensen, Noah A. Smith, and Yulia Tsvetkov. 2023. Do all languages cost the same? Tokenization in the era of commercial language models. arXiv preprint arXiv:2305.13707. [our card](https://md.research.elamur.ai/free/en/papers/ahia-2023-do-all-languages-cost-same/)
3. Kabir Ahuja, Rishav Hada, Millicent Ochieng, Prachi Jain, Harshita Diddee, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram. 2023. MEGA: Multilingual evaluation of generative AI. arXiv preprint arXiv:2303.12528. [our card](https://md.research.elamur.ai/free/en/papers/ahuja-2023-mega-multilingual-evaluation-generative-ai/)
4. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966. [our card](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/)
5. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity. arXiv preprint arX… [our card](https://md.research.elamur.ai/free/en/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/)
6. Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, et al. 2023. SeamlessM4T – massively multilingual & multimodal machine translation. arXiv preprint arXiv:2308.11596. [our card](https://md.research.elamur.ai/free/en/papers/communication-2023-seamlessm4t-massively-multilingual-multimodal-machine/)
7. Reem Bassiouney. 2009. Arabic Sociolinguistics. Edinburgh University Press. [find](https://scholar.google.com/scholar?q=Reem+Bassiouney%2E+2009%2E+Arabic+Sociolinguistics%2E+Edinburgh+University+Press%2E)
8. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. [find](https://scholar.google.com/scholar?q=Emily+M%2E+Bender%2C+Timnit+Gebru%2C+Angelina+McMillan%2DMajor%2C+and+Shmargaret+Shmitchell%2E+2021%2E+On+the+dangers+of+stochastic+parrots%3A+Can+language+models+be+too+big%3F+In+Proceedings+of+the+2021+ACM+Conference)
9. Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. 2000. A neural probabilistic language model. Advances in Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Yoshua+Bengio%2C+R%C3%A9jean+Ducharme%2C+and+Pascal+Vincent%2E+2000%2E+A+neural+probabilistic+language+model%2E+Advances+in+Neural+Information+Processing+Systems%2E)
10. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Advances in Neural Information Processing Systems. [our card](https://md.research.elamur.ai/free/en/papers/bolukbasi-2016-man-computer-programmer-woman-homemaker/)
11. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, et al. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Eval… [find](https://scholar.google.com/scholar?q=Houda+Bouamor%2C+Nizar+Habash%2C+Mohammad+Salameh%2C+Wajdi+Zaghouani%2C+Owen+Rambow%2C+Dana+Abdulrahim%2C+Ossama+Obeid%2C+Salam+Khalifa%2C+Fadhl+Eryani%2C+Alexander+Erdmann%2C+et+al%2E+2018%2E+The+MADAR+Arabic+dialect+corpus)
12. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
13. Monojit Choudhury and Amit Deshpande. 2021. How linguistically fair are multilingual pre-trained language models? In Proceedings of the AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Monojit+Choudhury+and+Amit+Deshpande%2E+2021%2E+How+linguistically+fair+are+multilingual+pre%2Dtrained+language+models%3F+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2E)
14. Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, and Jason Riesa. 2020. Improving multilingual models with language-clustered vocabularies. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Dan+Garrette%2C+Kiat+Chuan+Tan%2C+and+Jason+Riesa%2E+2020%2E+Improving+multilingual+models+with+language%2Dclustered+vocabularies%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+)
15. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. [our card](https://md.research.elamur.ai/free/en/papers/chung-2022-scaling-instruction-finetuned-language-models/)
16. Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. 2016. A character-level decoder without explicit segmentation for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). [our card](https://md.research.elamur.ai/free/en/papers/chung-2016-character-level-decoder-without-explicit/)
17. Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. 2022. Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation. Transactions of the Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/clark-2022-canine-pre-training-efficient-tokenization/)
18. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Annual Meeting of the Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+%C3%89douard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dlingu)
19. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. [find](https://scholar.google.com/scholar?q=Marta+R+Costa%2Djuss%C3%A0%2C+James+Cross%2C+Onur+%C3%87elebi%2C+Maha+Elbayad%2C+Kenneth+Heafield%2C+Kevin+Heffernan%2C+Elahe+Kalbassi%2C+Janice+Lam%2C+Daniel+Licht%2C+Jean+Maillard%2C+et+al%2E+2022%2E+No+language+left+behind%3A+Scaling+h)
20. Raj Dabre, Chenhui Chu, and Anoop Kunchukuttan. 2020. A survey of multilingual neural machine translation. ACM Computing Surveys. [find](https://scholar.google.com/scholar?q=Raj+Dabre%2C+Chenhui+Chu%2C+and+Anoop+Kunchukuttan%2E+2020%2E+A+survey+of+multilingual+neural+machine+translation%2E+ACM+Computing+Surveys%2E)
21. Raj Dabre and Aneerav Sukhoo. 2022. MorisienMT: A dataset for Mauritian Creole machine translation. arXiv preprint arXiv:2206.02421. — [arXiv:2206.02421](https://arxiv.org/abs/2206.02421) [find](https://scholar.google.com/scholar?q=Raj+Dabre+and+Aneerav+Sukhoo%2E+2022%2E+MorisienMT%3A+A+dataset+for+Mauritian+Creole+machine+translation%2E+arXiv+preprint+arXiv%3A2206%2E02421%2E+%E2%80%94+%5BarXiv%3A2206%2E02421%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2206%2E02421%29)
22. Michel DeGraff. 2007. Kreyòl Ayisyen, or Haitian Creole (Creole French). [find](https://scholar.google.com/scholar?q=Michel+DeGraff%2E+2007%2E+Krey%C3%B2l+Ayisyen%2C+or+Haitian+Creole+%28Creole+French%29%2E)
23. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
24. Pelin Dogan-Schönberger, Julian Mäder, and Thomas Hofmann. 2021. SwissDial: Parallel multidialectal corpus of spoken Swiss German. arXiv preprint arXiv:2103.11401. [find](https://scholar.google.com/scholar?q=Pelin+Dogan%2DSch%C3%B6nberger%2C+Julian+M%C3%A4der%2C+and+Thomas+Hofmann%2E+2021%2E+SwissDial%3A+Parallel+multidialectal+corpus+of+spoken+Swiss+German%2E+arXiv+preprint+arXiv%3A2103%2E11401%2E)
25. Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. 2023. GPTs are GPTs: An early look at the labor market impact potential of large language models. arXiv preprint arXiv:2303.10130. — [arXiv:2303.10130](https://arxiv.org/abs/2303.10130) [find](https://scholar.google.com/scholar?q=Tyna+Eloundou%2C+Sam+Manning%2C+Pamela+Mishkin%2C+and+Daniel+Rock%2E+2023%2E+GPTs+are+GPTs%3A+An+early+look+at+the+labor+market+impact+potential+of+large+language+models%2E+arXiv+preprint+arXiv%3A2303%2E10130%2E+%E2%80%94+%5BarXiv)
26. Facebook. 2021. Sri Lanka human rights impact assessment. Accessed on April 11, 2023. [find](https://scholar.google.com/scholar?q=Facebook%2E+2021%2E+Sri+Lanka+human+rights+impact+assessment%2E+Accessed+on+April+11%2C+2023%2E)
27. Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021. Beyond English-centric multilingual machine translation. The Journal of Machine Learning Research. [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Shruti+Bhosale%2C+Holger+Schwenk%2C+Zhiyi+Ma%2C+Ahmed+El%2DKishky%2C+Siddharth+Goyal%2C+Mandeep+Baines%2C+Onur+Celebi%2C+Guillaume+Wenzek%2C+Vishrav+Chaudhary%2C+et+al%2E+2021%2E+Beyond+English%2Dcentric+multilingu)
28. Philip Gage. 1994. A new algorithm for data compression. C Users Journal. [find](https://scholar.google.com/scholar?q=Philip+Gage%2E+1994%2E+A+new+algorithm+for+data+compression%2E+C+Users+Journal%2E)
29. Yingqiang Gao, Nikola I. Nikolov, Yuhuang Hu, and Richard H.R. Hahnloser. 2020. Character-level translation with self-attention. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yingqiang+Gao%2C+Nikola+I%2E+Nikolov%2C+Yuhuang+Hu%2C+and+Richard+H%2ER%2E+Hahnloser%2E+2020%2E+Character%2Dlevel+translation+with+self%2Dattention%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Comput)
30. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/gehman-2020-realtoxicityprompts-evaluating-neural-toxic-degeneration/)
31. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2021. The FLORES-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computat… [find](https://scholar.google.com/scholar?q=Naman+Goyal%2C+Cynthia+Gao%2C+Vishrav+Chaudhary%2C+Peng%2DJen+Chen%2C+Guillaume+Wenzek%2C+Da+Ju%2C+Sanjana+Krishnan%2C+Marc%E2%80%99Aurelio+Ranzato%2C+Francisco+Guzm%C3%A1n%2C+and+Angela+Fan%2E+2021%2E+The+FLORES%2D101+evaluation+benchmark)
32. Suchin Gururangan, Dallas Card, Sarah K. Dreier, Emily K. Gade, Leroy Z. Wang, Zeyu Wang, Luke Zettlemoyer, and Noah A. Smith. 2022. Whose language counts as high quality? Measuring language ideologies in text data selection. arXiv preprint arXiv:2201.10474. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Dallas+Card%2C+Sarah+K%2E+Dreier%2C+Emily+K%2E+Gade%2C+Leroy+Z%2E+Wang%2C+Zeyu+Wang%2C+Luke+Zettlemoyer%2C+and+Noah+A%2E+Smith%2E+2022%2E+Whose+language+counts+as+high+quality%3F+Measuring+language+ideologie)
33. Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. 2019. Two new evaluation datasets for low-resource machine translation: Nepali-English and Sinhala-English. In Proceedings of the 2019 Conference on Empirical Methods i… [find](https://scholar.google.com/scholar?q=Francisco+Guzm%C3%A1n%2C+Peng%2DJen+Chen%2C+Myle+Ott%2C+Juan+Pino%2C+Guillaume+Lample%2C+Philipp+Koehn%2C+Vishrav+Chaudhary%2C+and+Marc%E2%80%99Aurelio+Ranzato%2E+2019%2E+Two+new+evaluation+datasets+for+low%2Dresource+machine+translati)
34. Shiro Hattori. 1973. Japanese dialects. In Diachronic, areal, and typological linguistics. [find](https://scholar.google.com/scholar?q=Shiro+Hattori%2E+1973%2E+Japanese+dialects%2E+In+Diachronic%2C+areal%2C+and+typological+linguistics%2E)
35. Valentin Hofmann, Janet Pierrehumbert, and Hinrich Schütze. 2021. Superbizarre is not superb: Derivational morphology improves BERT’s interpretation of complex words. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conferenc… [our card](https://md.research.elamur.ai/free/en/papers/hofmann-2021-superbizarre-derivational/)
36. Valentin Hofmann, Hinrich Schuetze, and Janet Pierrehumbert. 2022. An embarrassingly simple method to mitigate undesirable properties of pretrained language model tokenizers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). [find](https://scholar.google.com/scholar?q=Valentin+Hofmann%2C+Hinrich+Schuetze%2C+and+Janet+Pierrehumbert%2E+2022%2E+An+embarrassingly+simple+method+to+mitigate+undesirable+properties+of+pretrained+language+model+tokenizers%2E+In+Proceedings+of+the+60t)
37. Michael A. Hogg, Nicholas Joyce, and Dominic Abrams. 1984. Diglossia in Switzerland? A social identity analysis of speaker evaluations. Journal of Language and Social Psychology. [find](https://scholar.google.com/scholar?q=Michael+A%2E+Hogg%2C+Nicholas+Joyce%2C+and+Dominic+Abrams%2E+1984%2E+Diglossia+in+Switzerland%3F+A+social+identity+analysis+of+speaker+evaluations%2E+Journal+of+Language+and+Social+Psychology%2E)
38. Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is ChatGPT a good translator? Yes with GPT-4 as the engine. arXiv preprint arXiv:2301.08745. [our card](https://md.research.elamur.ai/free/en/papers/jiao-2023-chatgpt-good-translator-yes-gpt/)
39. Johnson. 2023. ChatGPT is a marvel of multilingualism. The Economist. [find](https://scholar.google.com/scholar?q=Johnson%2E+2023%2E+ChatGPT+is+a+marvel+of+multilingualism%2E+The+Economist%2E)
40. Alan S. Kaye. 2001. Diglossia: The state of the art. International Journal of the Sociology of Language. [find](https://scholar.google.com/scholar?q=Alan+S%2E+Kaye%2E+2001%2E+Diglossia%3A+The+state+of+the+art%2E+International+Journal+of+the+Sociology+of+Language%2E)
41. Feyza Duman Keles, Pruthuvi Mahesakya Wijewardena, and Chinmay Hegde. 2023. On the computational complexity of self-attention. In International Conference on Algorithmic Learning Theory. [find](https://scholar.google.com/scholar?q=Feyza+Duman+Keles%2C+Pruthuvi+Mahesakya+Wijewardena%2C+and+Chinmay+Hegde%2E+2023%2E+On+the+computational+complexity+of+self%2Dattention%2E+In+International+Conference+on+Algorithmic+Learning+Theory%2E)
42. Simran Khanuja, Diksha Bansal, Sarvesh Mehtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Margam, Pooja Aggarwal, Rajiv Teja Nagipogu, Shachi Dave, Shruti Gupta, Subhash Chandra Bose Gali, Vish Subramanian, and Partha Talukdar. 2021. MuRIL: Multilingual representations for Indian langua… [our card](https://md.research.elamur.ai/free/en/papers/khanuja-2021-muril-multilingual-representations-indian-languages/)
43. Wojciech Kryściński, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Languag… [our card](https://md.research.elamur.ai/free/en/papers/kryscinski-2019-neural-text-summarization-critical-evaluation/)
44. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-subword-regularization-improving-neural-network/)
45. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/)
46. Mohammad Amin Kuhail, Nazik Alturki, Salwa Alramlawi, and Kholood Alhejori. 2023. Interacting with educational chatbots: A systematic review. Education and Information Technologies. [find](https://scholar.google.com/scholar?q=Mohammad+Amin+Kuhail%2C+Nazik+Alturki%2C+Salwa+Alramlawi%2C+and+Kholood+Alhejori%2E+2023%2E+Interacting+with+educational+chatbots%3A+A+systematic+review%2E+Education+and+Information+Technologies%2E)
47. Siddique Latif, Aun Zaidi, Heriberto Cuayahuitl, Fahad Shamshad, Moazzam Shoukat, and Junaid Qadir. 2023. Transformers in speech processing: A survey. arXiv preprint arXiv:2303.11607. — [arXiv:2303.11607](https://arxiv.org/abs/2303.11607) [find](https://scholar.google.com/scholar?q=Siddique+Latif%2C+Aun+Zaidi%2C+Heriberto+Cuayahuitl%2C+Fahad+Shamshad%2C+Moazzam+Shoukat%2C+and+Junaid+Qadir%2E+2023%2E+Transformers+in+speech+processing%3A+A+survey%2E+arXiv+preprint+arXiv%3A2303%2E11607%2E+%E2%80%94+%5BarXiv%3A2303%2E11)
48. Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2017. Fully character-level neural machine translation without explicit segmentation. Transactions of the Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/lee-2017-fully-character-level-neural-machine/)
49. Peter Lee, Sebastien Bubeck, and Joseph Petro. 2023. Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine. New England Journal of Medicine. [find](https://scholar.google.com/scholar?q=Peter+Lee%2C+Sebastien+Bubeck%2C+and+Joseph+Petro%2E+2023%2E+Benefits%2C+limits%2C+and+risks+of+GPT%2D4+as+an+AI+chatbot+for+medicine%2E+New+England+Journal+of+Medicine%2E)
50. Heather Lent, Emanuele Bugliarello, Miryam de Lhoneux, Chen Qiu, and Anders Søgaard. 2021. On language models for creoles. In Proceedings of the 25th Conference on Computational Natural Language Learning. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Heather+Lent%2C+Emanuele+Bugliarello%2C+Miryam+de+Lhoneux%2C+Chen+Qiu%2C+and+Anders+S%C3%B8gaard%2E+2021%2E+On+language+models+for+creoles%2E+In+Proceedings+of+the+25th+Conference+on+Computational+Natural+Language+Learn)
51. Heather Lent, Kelechi Ogueji, Miryam de Lhoneux, Orevaoghene Ahia, and Anders Søgaard. 2022. What a creole wants, what a creole needs. In Proceedings of the Thirteenth Language Resources and Evaluation Conference. [find](https://scholar.google.com/scholar?q=Heather+Lent%2C+Kelechi+Ogueji%2C+Miryam+de+Lhoneux%2C+Orevaoghene+Ahia%2C+and+Anders+S%C3%B8gaard%2E+2022%2E+What+a+creole+wants%2C+what+a+creole+needs%2E+In+Proceedings+of+the+Thirteenth+Language+Resources+and+Evaluatio)
52. Janny Leung. 2022. Shortcuts and shortfalls in Meta’ s content moderation practices: A glimpse from its oversight board’ s first year of operation. Comparative Law and Language. [find](https://scholar.google.com/scholar?q=Janny+Leung%2E+2022%2E+Shortcuts+and+shortfalls+in+Meta%E2%80%99+s+content+moderation+practices%3A+A+glimpse+from+its+oversight+board%E2%80%99+s+first+year+of+operation%2E+Comparative+Law+and+Language%2E)
53. Sharon Levy, Neha Anna John, Ling Liu, Yogarshi Vyas, Jie Ma, Yoshinari Fujinuma, Miguel Ballesteros, Vittorio Castelli, and Dan Roth. 2023. Comparing biases and the impact of multilingual training across multiple languages. arXiv preprint arXiv:2305.11242. [find](https://scholar.google.com/scholar?q=Sharon+Levy%2C+Neha+Anna+John%2C+Ling+Liu%2C+Yogarshi+Vyas%2C+Jie+Ma%2C+Yoshinari+Fujinuma%2C+Miguel+Ballesteros%2C+Vittorio+Castelli%2C+and+Dan+Roth%2E+2023%2E+Comparing+biases+and+the+impact+of+multilingual+training+ac)
54. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172. [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/)
55. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Jiatao+Gu%2C+Naman+Goyal%2C+Xian+Li%2C+Sergey+Edunov%2C+Marjan+Ghazvininejad%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+2020%2E+Multilingual+denoising+pre%2Dtraining+for+neural+machine+translation%2E+Transactio)
56. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+RoBERTa%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
57. Georges Lüdi. 2007. The Swiss model of plurilingual communication. Receptive multilingualism: Linguistic analyses, language policies and didactic concepts. [find](https://scholar.google.com/scholar?q=Georges+L%C3%BCdi%2E+2007%2E+The+Swiss+model+of+plurilingual+communication%2E+Receptive+multilingualism%3A+Linguistic+analyses%2C+language+policies+and+didactic+concepts%2E)
58. Sophie Lythreatis, Sanjay Kumar Singh, and Abdul-Nasser El-Kassar. 2022. The digital divide: A review and future research agenda. Technological Forecasting and Social Change. [find](https://scholar.google.com/scholar?q=Sophie+Lythreatis%2C+Sanjay+Kumar+Singh%2C+and+Abdul%2DNasser+El%2DKassar%2E+2022%2E+The+digital+divide%3A+A+review+and+future+research+agenda%2E+Technological+Forecasting+and+Social+Change%2E)
59. Liam Magee, Lida Ghahremanlou, Karen Soldatic, and Shanthi Robertson. 2021. Intersectional bias in causal language models. arXiv preprint arXiv:2107.07691. — [arXiv:2107.07691](https://arxiv.org/abs/2107.07691) [find](https://scholar.google.com/scholar?q=Liam+Magee%2C+Lida+Ghahremanlou%2C+Karen+Soldatic%2C+and+Shanthi+Robertson%2E+2021%2E+Intersectional+bias+in+causal+language+models%2E+arXiv+preprint+arXiv%3A2107%2E07691%2E+%E2%80%94+%5BarXiv%3A2107%2E07691%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
60. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de La Clergerie, Djamé Seddah, and Benoît Sagot. 2020. CamemBERT: A tasty French language model. In Annual Meeting of the Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/martin-2019-camembert-tasty-french-language-model/)
61. Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and… [our card](https://md.research.elamur.ai/free/en/papers/may-2019-measuring-social-biases-sentence-encoders/)
62. Sabrina J Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y Lee, Benoît Sagot, et al. 2021. Between words and characters: A brief history of open-vocabulary modeling and tokenization in NLP. arXiv preprint arXiv:2112.10508. [our card](https://md.research.elamur.ai/free/en/papers/mielke-2021-between-words-characters-brief-history/)
63. Rob Miles and Sean Riley. 2023. Glitch tokens – Computerphile. Accessed on April 11, 2023. [find](https://scholar.google.com/scholar?q=Rob+Miles+and+Sean+Riley%2E+2023%2E+Glitch+tokens+%E2%80%93+Computerphile%2E+Accessed+on+April+11%2C+2023%2E)
64. Robert Munro. 2010. Crowdsourced translation for emergency response in Haiti: the global collaboration of local knowledge. In Proceedings of the Workshop on Collaborative Translation: technology, crowdsourcing, and the translator perspective. Association for Machine Translation in the Americas. [find](https://scholar.google.com/scholar?q=Robert+Munro%2E+2010%2E+Crowdsourced+translation+for+emergency+response+in+Haiti%3A+the+global+collaboration+of+local+knowledge%2E+In+Proceedings+of+the+Workshop+on+Collaborative+Translation%3A+technology%2C+crow)
65. Pieter Muysken and Norval Smith. 1994. The study of pidgin and creole languages. In Pidgins and creoles: An introduction. [find](https://scholar.google.com/scholar?q=Pieter+Muysken+and+Norval+Smith%2E+1994%2E+The+study+of+pidgin+and+creole+languages%2E+In+Pidgins+and+creoles%3A+An+introduction%2E)
66. Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long… [our card](https://md.research.elamur.ai/free/en/papers/nadeem-2021-stereoset-measuring-stereotypical-bias-pretrained/)
67. Dat Quoc Nguyen and Anh-Tuan Nguyen. 2020. PhoBERT: Pre-trained language models for Vietnamese. In Findings of the Association for Computational Linguistics: EMNLP. [our card](https://md.research.elamur.ai/free/en/papers/nguyen-2020-phobert-pre-trained-language-models/)
68. Akintunde Oladipo, Odunayo Ogundepo, Kelechi Ogueji, and Jimmy Lin. 2022. An exploration of vocabulary size and transfer effects in multilingual language models for African languages. In 3rd Workshop on African Natural Language Processing. [find](https://scholar.google.com/scholar?q=Akintunde+Oladipo%2C+Odunayo+Ogundepo%2C+Kelechi+Ogueji%2C+and+Jimmy+Lin%2E+2022%2E+An+exploration+of+vocabulary+size+and+transfer+effects+in+multilingual+language+models+for+African+languages%2E+In+3rd+Workshop+)
69. OpenAI. 2022. Introducing ChatGPT. Accessed on April 11, 2023. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2022%2E+Introducing+ChatGPT%2E+Accessed+on+April+11%2C+2023%2E)
70. OpenAI. 2022. tiktoken. Git commit: 82facf9. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2022%2E+tiktoken%2E+Git+commit%3A+82facf9%2E)
71. OpenAI. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774. [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
72. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems. [our card](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
73. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. BBQ: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022. [our card](https://md.research.elamur.ai/free/en/papers/parrish-2022-bbq-hand-built-bias-benchmark/)
74. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2021. UNKs everywhere: Adapting multilingual language models to new scripts. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. [our card](https://md.research.elamur.ai/free/en/papers/pfeiffer-2021-unks-everywhere-adapting-multilingual-language/)
75. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
76. Jenalea Rajab. 2022. Effect of tokenisation strategies for low-resourced Southern African languages. In 3rd Workshop on African Natural Language Processing. [find](https://scholar.google.com/scholar?q=Jenalea+Rajab%2E+2022%2E+Effect+of+tokenisation+strategies+for+low%2Dresourced+Southern+African+languages%2E+In+3rd+Workshop+on+African+Natural+Language+Processing%2E)
77. Krithika Ramesh, Sunayana Sitaram, and Monojit Choudhury. 2023. Fairness in language models beyond English: Gaps and challenges. In Findings of the Association for Computational Linguistics: EACL 2023. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Krithika+Ramesh%2C+Sunayana+Sitaram%2C+and+Monojit+Choudhury%2E+2023%2E+Fairness+in+language+models+beyond+English%3A+Gaps+and+challenges%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EACL+2023)
78. Surangika Ranathunga, En-Shiun Annie Lee, Marjana Prifti Skenduli, Ravi Shekhar, Mehreen Alam, and Rishemjit Kaur. 2023. Neural machine translation for low-resource languages: A survey. ACM Computing Surveys. [find](https://scholar.google.com/scholar?q=Surangika+Ranathunga%2C+En%2DShiun+Annie+Lee%2C+Marjana+Prifti+Skenduli%2C+Ravi+Shekhar%2C+Mehreen+Alam%2C+and+Rishemjit+Kaur%2E+2023%2E+Neural+machine+translation+for+low%2Dresource+languages%3A+A+survey%2E+ACM+Computing+)
79. Jessica Rumbelow and Matthew Watkins. 2023a. SolidGoldMagikarp III: Glitch token archaelogy. Accessed on April 11, 2023. [find](https://scholar.google.com/scholar?q=Jessica+Rumbelow+and+Matthew+Watkins%2E+2023a%2E+SolidGoldMagikarp+III%3A+Glitch+token+archaelogy%2E+Accessed+on+April+11%2C+2023%2E)
80. Jessica Rumbelow and Matthew Watkins. 2023b. SolidGoldMagikarp (plus, prompt generation). Accessed on April 11, 2023. [find](https://scholar.google.com/scholar?q=Jessica+Rumbelow+and+Matthew+Watkins%2E+2023b%2E+SolidGoldMagikarp+%28plus%2C+prompt+generation%29%2E+Accessed+on+April+11%2C+2023%2E)
81. Charles Russ. 1990. The Dialects of Modern German: A Linguistic Survey. [find](https://scholar.google.com/scholar?q=Charles+Russ%2E+1990%2E+The+Dialects+of+Modern+German%3A+A+Linguistic+Survey%2E)
82. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? On the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint… [our card](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/)
83. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. KUISAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation. [our card](https://md.research.elamur.ai/free/en/papers/safaya-2020-kuisail-semeval-2020-task-12/)
84. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. BLOOM: A 176B-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ili%C4%87%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+Matthias+Gall%C3%A9%2C+et+al%2E+2022%2E+BLOOM%3A+A+176B%2Dparameter+)
85. Raphael Scheible, Fabian Thomczyk, Patric Tippmann, Victor Jaravine, and Martin Boeker. 2020. GottBERT: A pure German language model. arXiv preprint arXiv:2012.02110. — [arXiv:2012.02110](https://arxiv.org/abs/2012.02110) [find](https://scholar.google.com/scholar?q=Raphael+Scheible%2C+Fabian+Thomczyk%2C+Patric+Tippmann%2C+Victor+Jaravine%2C+and+Martin+Boeker%2E+2020%2E+GottBERT%3A+A+pure+German+language+model%2E+arXiv+preprint+arXiv%3A2012%2E02110%2E+%E2%80%94+%5BarXiv%3A2012%2E02110%5D%28https%3A%2F%2Farxi)
86. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and Korean voice search. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). [find](https://scholar.google.com/scholar?q=Mike+Schuster+and+Kaisuke+Nakajima%2E+2012%2E+Japanese+and+Korean+voice+search%2E+In+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2E)
87. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). [our card](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/)
88. Pieter A. M. Seuren. 1995. Notes on the history and the syntax of Mauritian Creole. Linguistics. [find](https://scholar.google.com/scholar?q=Pieter+A%2E+M%2E+Seuren%2E+1995%2E+Notes+on+the+history+and+the+syntax+of+Mauritian+Creole%2E+Linguistics%2E)
89. Yan Shao, Christian Hardmeier, and Joakim Nivre. 2018. Universal word segmentation: Implementation and interpretation. Transactions of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yan+Shao%2C+Christian+Hardmeier%2C+and+Joakim+Nivre%2E+2018%2E+Universal+word+segmentation%3A+Implementation+and+interpretation%2E+Transactions+of+the+Association+for+Computational+Linguistics%2E)
90. Peter Sieber and Horst Sitta. 1987. Deutsch in der Schweiz. Zeitschrift für Germanistik. [find](https://scholar.google.com/scholar?q=Peter+Sieber+and+Horst+Sitta%2E+1987%2E+Deutsch+in+der+Schweiz%2E+Zeitschrift+f%C3%BCr+Germanistik%2E)
91. Felix Stahlberg. 2020. Neural machine translation: A review. Journal of Artificial Intelligence Research. [find](https://scholar.google.com/scholar?q=Felix+Stahlberg%2E+2020%2E+Neural+machine+translation%3A+A+review%2E+Journal+of+Artificial+Intelligence+Research%2E)
92. Steve Stecklow. 2018. Hatebook. Reuters. Accessed on April 11, 2023. [find](https://scholar.google.com/scholar?q=Steve+Stecklow%2E+2018%2E+Hatebook%2E+Reuters%2E+Accessed+on+April+11%2C+2023%2E)
93. Hui Su, Weiwei Shi, Xiaoyu Shen, Zhou Xiao, Tuo Ji, Jiarui Fang, and Jie Zhou. 2022. RoCbert: Robust Chinese BERT with multimodal contrastive pretraining. In Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Hui+Su%2C+Weiwei+Shi%2C+Xiaoyu+Shen%2C+Zhou+Xiao%2C+Tuo+Ji%2C+Jiarui+Fang%2C+and+Jie+Zhou%2E+2022%2E+RoCbert%3A+Robust+Chinese+BERT+with+multimodal+contrastive+pretraining%2E+In+Annual+Meeting+of+the+Association+for+Comp)
94. Lichao Sun, Kazuma Hashimoto, Wenpeng Yin, Akari Asai, Jia Li, Philip Yu, and Caiming Xiong. 2020. Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERT. arXiv preprint arXiv:2003.04985. — [arXiv:2003.04985](https://arxiv.org/abs/2003.04985) [find](https://scholar.google.com/scholar?q=Lichao+Sun%2C+Kazuma+Hashimoto%2C+Wenpeng+Yin%2C+Akari+Asai%2C+Jia+Li%2C+Philip+Yu%2C+and+Caiming+Xiong%2E+2020%2E+Adv%2DBERT%3A+BERT+is+not+robust+on+misspellings%21+Generating+nature+adversarial+samples+on+BERT%2E+arXiv+pr)
95. Shinnosuke Takamichi and Hiroshi Saruwatari. 2018. CPJD corpus: Crowdsourced parallel speech corpus of Japanese dialects. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). [find](https://scholar.google.com/scholar?q=Shinnosuke+Takamichi+and+Hiroshi+Saruwatari%2E+2018%2E+CPJD+corpus%3A+Crowdsourced+parallel+speech+corpus+of+Japanese+dialects%2E+In+Proceedings+of+the+Eleventh+International+Conference+on+Language+Resources+)
96. Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. Multilingual translation with extensible multilingual pretraining and finetuning. arXiv preprint arXiv:2008.00401. [find](https://scholar.google.com/scholar?q=Yuqing+Tang%2C+Chau+Tran%2C+Xian+Li%2C+Peng%2DJen+Chen%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Jiatao+Gu%2C+and+Angela+Fan%2E+2020%2E+Multilingual+translation+with+extensible+multilingual+pretraining+and+finetuning%2E+arXiv)
97. Timm Teubner, Christoph M Flath, Christof Weinhardt, Wil van der Aalst, and Oliver Hinz. 2023. Welcome to the era of ChatGPT et al: The prospects of large language models. Business & Information Systems Engineering. [find](https://scholar.google.com/scholar?q=Timm+Teubner%2C+Christoph+M+Flath%2C+Christof+Weinhardt%2C+Wil+van+der+Aalst%2C+and+Oliver+Hinz%2E+2023%2E+Welcome+to+the+era+of+ChatGPT+et+al%3A+The+prospects+of+large+language+models%2E+Business+%26+Information+Syste)
98. The Unicode Consortium. 2022. The Unicode standard, Version 15.0.0. [find](https://scholar.google.com/scholar?q=The+Unicode+Consortium%2E+2022%2E+The+Unicode+standard%2C+Version+15%2E0%2E0%2E)
99. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12). [find](https://scholar.google.com/scholar?q=J%C3%B6rg+Tiedemann%2E+2012%2E+Parallel+data%2C+tools+and+interfaces+in+OPUS%2E+In+Proceedings+of+the+Eight+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%E2%80%9912%29%2E)
100. Tohoku NLP Group. 2019. BERT models for Japanese NLP. [find](https://scholar.google.com/scholar?q=Tohoku+NLP+Group%2E+2019%2E+BERT+models+for+Japanese+NLP%2E)
101. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. [our card](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/)
102. Emma Trentman and Sonia Shiri. 2020. The mutual intelligibility of Arabic dialects: Implications for the language classroom. Critical Multilingualism Studies. [our card](https://md.research.elamur.ai/free/en/papers/trentman-2020-mutual-intelligibility-arabic-dialects-implications/)
103. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems. [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
104. Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: BERT for Finnish. arXiv preprint arXiv:1912.07076. [our card](https://md.research.elamur.ai/free/en/papers/virtanen-2019-multilingual-enough-bert-finnish/)
105. Jonathan J. Webster and Chunyu Kit. 1992. Tokenization as the initial phase in NLP. In The International Conference on Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/webster-1992-tokenization-as-initial-phase-nlp/)
106. Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Discourse-aware neural extractive text summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/xu-2019-discourse-aware-neural-extractive-text/)
107. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Linting+Xue%2C+Aditya+Barua%2C+Noah+Constant%2C+Rami+Al%2DRfou%2C+Sharan+Narang%2C+Mihir+Kale%2C+Adam+Roberts%2C+and+Colin+Raffel%2E+2022%2E+ByT5%3A+Towards+a+token%2Dfree+future+with+pre%2Dtrained+byte%2Dto%2Dbyte+models%2E+Transac)
108. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mT5: A massively multilingual pre-trained text-to-text transformer. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language… [our card](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/)
109. Joseph K. Yamagiwa. 1967. On dialect intelligibility in Japan. Anthropological Linguistics. [find](https://scholar.google.com/scholar?q=Joseph+K%2E+Yamagiwa%2E+1967%2E+On+dialect+intelligibility+in+Japan%2E+Anthropological+Linguistics%2E)
110. Shiyue Zhang, Vishrav Chaudhary, Naman Goyal, James Cross, Guillaume Wenzek, Mohit Bansal, and Francisco Guzman. 2022. How robust is neural machine translation to language imbalance in multilingual tokenizer training? In Proceedings of the 15th Biennial Conference of the Association for Machine Tran… [find](https://scholar.google.com/scholar?q=Shiyue+Zhang%2C+Vishrav+Chaudhary%2C+Naman+Goyal%2C+James+Cross%2C+Guillaume+Wenzek%2C+Mohit+Bansal%2C+and+Francisco+Guzman%2E+2022%2E+How+robust+is+neural+machine+translation+to+language+imbalance+in+multilingual+to)
111. Judit Ács. 2019. Exploring BERT’s vocabulary. Accessed on April 11, 2023. [find](https://scholar.google.com/scholar?q=Judit+%C3%81cs%2E+2019%2E+Exploring+BERT%E2%80%99s+vocabulary%2E+Accessed+on+April+11%2C+2023%2E)
112. Slavomír Čéplö, Ján Bátora, Adam Benkato, Jiří Milička, Christophe Pereira, and Petr Zemánek. 2016. Mutual intelligibility of spoken Maltese, Libyan Arabic, and Tunisian Arabic functionally tested: A pilot study. Folia Linguistica. [find](https://scholar.google.com/scholar?q=Slavom%C3%ADr+%C4%8C%C3%A9pl%C3%B6%2C+J%C3%A1n+B%C3%A1tora%2C+Adam+Benkato%2C+Ji%C5%99%C3%AD+Mili%C4%8Dka%2C+Christophe+Pereira%2C+and+Petr+Zem%C3%A1nek%2E+2016%2E+Mutual+intelligibility+of+spoken+Maltese%2C+Libyan+Arabic%2C+and+Tunisian+Arabic+functionally+tested%3A+A+)

*Updated 23 October 2026*