---
title: "Between words and characters — Mielke 2021"
description: "The paper offers no new data and no method of its own. 198 references, cited by 89 papers here."
authors: ["Sabrina J. Mielke", "Zaid Alyafeai", "Elizabeth Salesky", "Colin Raffel", "Manan Dey", "Matthias Gallé", "Arun Raja", "Chenglei Si", "Wilson Y. Lee", "Benoît Sagot", "Samson Tan"]
year: 2021
venue: "arXiv:2112.10508"
source: "https://arxiv.org/abs/2112.10508"
doi: "10.48550/arxiv.2112.10508"
arxiv: "2112.10508"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mielke-2021-between-words-characters-brief-history/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mielke-2021-between-words-characters-brief-history/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP

# Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP

**Authors:** [Sabrina J. Mielke](https://md.research.elamur.ai/free/en/authors/sabrina-j-mielke/), [Zaid Alyafeai](https://md.research.elamur.ai/free/en/authors/zaid-alyafeai/), [Elizabeth Salesky](https://md.research.elamur.ai/free/en/authors/elizabeth-salesky/), [Colin Raffel](https://md.research.elamur.ai/free/en/authors/colin-raffel/), [Manan Dey](https://md.research.elamur.ai/free/en/authors/manan-dey/), Matthias Gallé, [Arun Raja](https://md.research.elamur.ai/free/en/authors/arun-raja/), [Chenglei Si](https://md.research.elamur.ai/free/en/authors/chenglei-si/), Wilson Y. Lee, [Benoît Sagot](https://md.research.elamur.ai/free/en/authors/benoit-sagot/) and [Samson Tan](https://md.research.elamur.ai/free/en/authors/samson-tan/) · **Year:** 2021 · **Venue:** arXiv:2112.10508 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2112.10508) · [PDF](https://arxiv.org/pdf/2112.10508) · [DOI 10.48550/arxiv.2112.10508](https://doi.org/10.48550/arxiv.2112.10508) · [arXiv 2112.10508](https://arxiv.org/abs/2112.10508)

**Citations:** [89](https://md.research.elamur.ai/free/en/papers/mielke-2021-between-words-characters-brief-history/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
This is a survey paper by the BigScience Workshop tokenization working group. The authors examine which units of text NLP models should operate on: words, subwords, characters, bytes, or even pixels. The paper offers no new data and no method of its own. Instead, it connects research from the pre-neural and neural eras into a single narrative. First, the authors disentangle the concepts of typographic tokens, word forms, "pre-tokens", and subwords. They then discuss hybrid models that augment words with character-level information, as well as unsupervised discovery of word boundaries. Separate sections cover subword algorithms (BPE, WordPiece, Unigram LM), the competition among languages for a shared vocabulary in multilingual models, and the maximal decomposition of text into characters, bytes, or images. The main conclusion is that there is no universal solution, and most likely there never will be. Every approach has both strengths and weaknesses, and some requirements directly conflict with one another, for example robustness to noise versus computational efficiency that is fair across languages. Tokenization therefore continues to require deliberate decisions, even now that byte-level models are available.

## Key points
- The authors propose a taxonomy of segmentation algorithms along two dimensions: from linguistically motivated to learned from data, and from simple to complex. It includes the Moses tokenizer, Morfessor, Linguistica, BPE, WordPiece, Unigram LM, SentencePiece, as well as characters, bytes, and pixels.
- A terminological shift: what used to be called "tokenization" is now called pre-tokenization, while tokens have become subwords, which need not carry any linguistic meaning. The correspondence between tokens and word forms is not one-to-one. Examples: don't = do + n't in the Penn Treebank, "multiword tokens" in UD.
- Most tokenizers are irreversible: because of normalization and whitespace merging, the original text cannot be exactly recovered from their output.
- Byte- and character-level models (CANINE, ByT5, Charformer) are more robust to noise and do not suffer from the OOV problem, but because of longer sequences they are slower to train and to run.
- Visual models that render sentences as images (Salesky et al., 2021) are competitive in machine translation across different scripts and noticeably more robust to synthetic noise.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [✂️ Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/)

## Cite

Cite as: Sabrina J. Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y. Lee, Benoît Sagot and Samson Tan. 2021. Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP. arXiv:2112.10508.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{mielke2021between,
  title = {{Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP}},
  author = {Sabrina J. Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y. Lee, Benoît Sagot and Samson Tan},
  year = {2021},
  doi = {10.48550/arxiv.2112.10508},
  eprint = {2112.10508},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2112.10508}
}
```

- [← previous Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes](https://md.research.elamur.ai/free/en/papers/kim-2021-perspective-taking-pragmatics-generating-empathetic/)
- [next → Probing for idiomaticity in vector space models](https://md.research.elamur.ai/free/en/papers/garcia-2021-probing-idiomaticity-vector-space-models/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

198 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Judit Ács. 2019. Exploring BERT’s vocabulary. [find](https://scholar.google.com/scholar?q=Judit+%C3%81cs%2E+2019%2E+Exploring+BERT%E2%80%99s+vocabulary%2E)
2. Gustavo Aguilar, Bryan McCann, Tong Niu, Nazneen Rajani, Nitish Keskar, and Thamar Solorio. 2021. Char2subword: Extending the subword embedding space using robust character compositionality. [find](https://scholar.google.com/scholar?q=Gustavo+Aguilar%2C+Bryan+McCann%2C+Tong+Niu%2C+Nazneen+Rajani%2C+Nitish+Keskar%2C+and+Thamar+Solorio%2E+2021%2E+Char2subword%3A+Extending+the+subword+embedding+space+using+robust+character+compositionality%2E)
3. Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. 2019. Character-level language modeling with deeper self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3159–3166. [find](https://scholar.google.com/scholar?q=Rami+Al%2DRfou%2C+Dokook+Choe%2C+Noah+Constant%2C+Mandy+Guo%2C+and+Llion+Jones%2E+2019%2E+Character%2Dlevel+language+modeling+with+deeper+self%2Dattention%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligen)
4. David Aldón Mínguez, Marta Ruiz Costa-Jussà, and José Adrián Rodríguez Fonollosa. 2016. Neural machine translation using bitmap fonts. In Proceedings of the EAMT 2016 Fifth Workshop on Hybrid Approaches to Translation (Hy-Tra), pages 1–9. [find](https://scholar.google.com/scholar?q=David+Ald%C3%B3n+M%C3%ADnguez%2C+Marta+Ruiz+Costa%2DJuss%C3%A0%2C+and+Jos%C3%A9+Adri%C3%A1n+Rodr%C3%ADguez+Fonollosa%2E+2016%2E+Neural+machine+translation+using+bitmap+fonts%2E+In+Proceedings+of+the+EAMT+2016+Fifth+Workshop+on+Hybrid+Approach)
5. Zaid Alyafeai, Maged S. Al-shaibani, Mustafa Ghaleb, and Irfan Ahmad. 2021. Evaluating various tokenizers for arabic text classification. ArXiv, abs/2106.07540. [our card](https://md.research.elamur.ai/free/en/papers/alyafeai-2021-evaluating-various-tokenizers-arabic-text/)
6. Chantal Amrhein and Rico Sennrich. 2021. How suitable are subword segmentation strategies for translating non-concatenative morphology? In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 689– 705, Punta Cana, Dominican Republic. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chantal+Amrhein+and+Rico+Sennrich%2E+2021%2E+How+suitable+are+subword+segmentation+strategies+for+translating+non%2Dconcatenative+morphology%3F+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EM)
7. Duygu Ataman, Wilker Aziz, and Alexandra Birch. 2020. A latent morphology model for open-vocabulary neural machine translation. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Duygu+Ataman%2C+Wilker+Aziz%2C+and+Alexandra+Birch%2E+2020%2E+A+latent+morphology+model+for+open%2Dvocabulary+neural+machine+translation%2E+In+International+Conference+on+Learning+Representations%2E)
8. Duygu Ataman and Marcello Federico. 2018a. Compositional representation of morphologically-rich input for neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 305–311, Melbourne, Australia. Association… [find](https://scholar.google.com/scholar?q=Duygu+Ataman+and+Marcello+Federico%2E+2018a%2E+Compositional+representation+of+morphologically%2Drich+input+for+neural+machine+translation%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+C)
9. Duygu Ataman and Marcello Federico. 2018b. An evaluation of two vocabulary reduction methods for neural machine translation. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 97–110. [find](https://scholar.google.com/scholar?q=Duygu+Ataman+and+Marcello+Federico%2E+2018b%2E+An+evaluation+of+two+vocabulary+reduction+methods+for+neural+machine+translation%2E+In+Proceedings+of+the+13th+Conference+of+the+Association+for+Machine+Transl)
10. Duygu Ataman, Orhan Firat, Mattia A. Di Gangi, Marcello Federico, and Alexandra Birch. 2019. On the importance of word boundaries in character-level neural machine translation. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 187–193, Hong Kong. Association for Computat… [find](https://scholar.google.com/scholar?q=Duygu+Ataman%2C+Orhan+Firat%2C+Mattia+A%2E+Di+Gangi%2C+Marcello+Federico%2C+and+Alexandra+Birch%2E+2019%2E+On+the+importance+of+word+boundaries+in+character%2Dlevel+neural+machine+translation%2E+In+Proceedings+of+the+3)
11. R.H. Baayen, R Piepenbrock, and L Gulikers. 1995. CELEX2. In LDC96L14. Linguistic Data Consortium, University of Pennsylvania. [find](https://scholar.google.com/scholar?q=R%2EH%2E+Baayen%2C+R+Piepenbrock%2C+and+L+Gulikers%2E+1995%2E+CELEX2%2E+In+LDC96L14%2E+Linguistic+Data+Consortium%2C+University+of+Pennsylvania%2E)
12. Tamali Banerjee and Pushpak Bhattacharyya. 2018. Meaningless yet meaningful: Morphology grounded subword-level nmt. In Proceedings of the second workshop on subword/character level models, pages 55–60. [find](https://scholar.google.com/scholar?q=Tamali+Banerjee+and+Pushpak+Bhattacharyya%2E+2018%2E+Meaningless+yet+meaningful%3A+Morphology+grounded+subword%2Dlevel+nmt%2E+In+Proceedings+of+the+second+workshop+on+subword%2Fcharacter+level+models%2C+pages+55%E2%80%9360)
13. Marco Baroni. 2000. Distributional Cues in Morpheme Discovery: A Computational Model and Empirical Evidence. Ph.D. thesis, University of California Los Angeles. [find](https://scholar.google.com/scholar?q=Marco+Baroni%2E+2000%2E+Distributional+Cues+in+Morpheme+Discovery%3A+A+Computational+Model+and+Empirical+Evidence%2E+Ph%2ED%2E+thesis%2C+University+of+California+Los+Angeles%2E)
14. Sarah Beemer, Zak Boston, April Bukoski, Daniel Chen, Princess Dickens, Andrew Gerlach, Torin Hopkins, Parth Anand Jawale, Chris Koski, Akanksha Malhotra, Piyush Mishra, Saliha Muradoglu, Lan Sang, Tyler Short, Sagarika Shreevastava, Elizabeth Spaulding, Testumichi Umada, Beilei Xiang, Changbing Yan… [find](https://scholar.google.com/scholar?q=Sarah+Beemer%2C+Zak+Boston%2C+April+Bukoski%2C+Daniel+Chen%2C+Princess+Dickens%2C+Andrew+Gerlach%2C+Torin+Hopkins%2C+Parth+Anand+Jawale%2C+Chris+Koski%2C+Akanksha+Malhotra%2C+Piyush+Mishra%2C+Saliha+Muradoglu%2C+Lan+Sang%2C+Ty)
15. Kenneth R Beesley and Lauri Karttunen. 2003. Finite-state morphology: Xerox tools and techniques. CSLI, Stanford. [find](https://scholar.google.com/scholar?q=Kenneth+R+Beesley+and+Lauri+Karttunen%2E+2003%2E+Finite%2Dstate+morphology%3A+Xerox+tools+and+techniques%2E+CSLI%2C+Stanford%2E)
16. Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. 2001. A neural probabilistic language model. In Advances in Neural Information Processing Systems, volume 13. MIT Press. [our card](https://md.research.elamur.ai/free/en/papers/bengio-2003-neural-probabilistic-language-model/)
17. Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432. [find](https://scholar.google.com/scholar?q=Yoshua+Bengio%2C+Nicholas+L%C3%A9onard%2C+and+Aaron+C%2E+Courville%2E+2013%2E+Estimating+or+propagating+gradients+through+stochastic+neurons+for+conditional+computation%2E+CoRR%2C+abs%2F1308%2E3432%2E)
18. Taylor Berg-Kirkpatrick, Alexandre Bouchard-Côté, John DeNero, and Dan Klein. 2010. Painless unsupervised learning with features. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 582–590, Los Angeles, Cal… [find](https://scholar.google.com/scholar?q=Taylor+Berg%2DKirkpatrick%2C+Alexandre+Bouchard%2DC%C3%B4t%C3%A9%2C+John+DeNero%2C+and+Dan+Klein%2E+2010%2E+Painless+unsupervised+learning+with+features%2E+In+Human+Language+Technologies%3A+The+2010+Annual+Conference+of+the+Nort)
19. Toms Bergmanis and Sharon Goldwater. 2017. From segmentation to analyses: a probabilistic model for unsupervised morphology induction. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 337–346, Valencia, Spain… [find](https://scholar.google.com/scholar?q=Toms+Bergmanis+and+Sharon+Goldwater%2E+2017%2E+From+segmentation+to+analyses%3A+a+probabilistic+model+for+unsupervised+morphology+induction%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+)
20. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5(0):135–146. [our card](https://md.research.elamur.ai/free/en/papers/bojanowski-2017-enriching-word-vectors-subword-information/)
21. Kaj Bostrom and Greg Durrett. 2020. Byte pair encoding is suboptimal for language model pre-training. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4617–4624, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kaj+Bostrom+and+Greg+Durrett%2E+2020%2E+Byte+pair+encoding+is+suboptimal+for+language+model+pre%2Dtraining%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2020%2C+pages+4617%E2%80%934624%2C+Online%2E)
22. Martin Braschler and Bärbel Ripplinger. 2004. How effective is stemming and decompounding for german text retrieval? Information Retrieval, 7(3):291–316. [find](https://scholar.google.com/scholar?q=Martin+Braschler+and+B%C3%A4rbel+Ripplinger%2E+2004%2E+How+effective+is+stemming+and+decompounding+for+german+text+retrieval%3F+Information+Retrieval%2C+7%283%29%3A291%E2%80%93316%2E)
23. Michael R Brent and Timothy A Cartwright. 1996. Distributional regularity and phonotactic constraints are useful for segmentation. Cognition, 61(1-2):93–125. [find](https://scholar.google.com/scholar?q=Michael+R+Brent+and+Timothy+A+Cartwright%2E+1996%2E+Distributional+regularity+and+phonotactic+constraints+are+useful+for+segmentation%2E+Cognition%2C+61%281%2D2%29%3A93%E2%80%93125%2E)
24. Michael R Brent, Sreerama K Murthy, and Andrew Lundberg. 1995. Discovering morphemic suffixes: a case study in MDL induction. In In Fifth International Workshop on AI and Statistics, Ft. Citeseer. [find](https://scholar.google.com/scholar?q=Michael+R+Brent%2C+Sreerama+K+Murthy%2C+and+Andrew+Lundberg%2E+1995%2E+Discovering+morphemic+suffixes%3A+a+case+study+in+MDL+induction%2E+In+In+Fifth+International+Workshop+on+AI+and+Statistics%2C+Ft%2E+Citeseer%2E)
25. Samuel Broscheit. 2018. Learning distributional token representations from visual features. In Proceedings of The Third Workshop on Representation Learning for NLP, pages 187–194, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Samuel+Broscheit%2E+2018%2E+Learning+distributional+token+representations+from+visual+features%2E+In+Proceedings+of+The+Third+Workshop+on+Representation+Learning+for+NLP%2C+pages+187%E2%80%93194%2C+Melbourne%2C+Australia)
26. Jacob Buckman and Graham Neubig. 2018. Neural lattice language models. Transactions of the Association for Computational Linguistics (TACL), 6. [find](https://scholar.google.com/scholar?q=Jacob+Buckman+and+Graham+Neubig%2E+2018%2E+Neural+lattice+language+models%2E+Transactions+of+the+Association+for+Computational+Linguistics+%28TACL%29%2C+6%2E)
27. Kris Cao and Laura Rimell. 2021. You should evaluate your language model on marginal likelihood over tokenisations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2104–2114, Online and Punta Cana, Dominican Republic. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Kris+Cao+and+Laura+Rimell%2E+2021%2E+You+should+evaluate+your+language+model+on+marginal+likelihood+over+tokenisations%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Proce)
28. William Chan, Yu Zhang, Quoc Le, and Navdeep Jaitly. 2017. Latent sequence decompositions. In International Conference on Learning Representations 2017. [find](https://scholar.google.com/scholar?q=William+Chan%2C+Yu+Zhang%2C+Quoc+Le%2C+and+Navdeep+Jaitly%2E+2017%2E+Latent+sequence+decompositions%2E+In+International+Conference+on+Learning+Representations+2017%2E)
29. Ethan C. Chau, Lucy H. Lin, and Noah A. Smith. 2020. Parsing with multilingual BERT, a small corpus, and a small treebank. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1324–1334, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ethan+C%2E+Chau%2C+Lucy+H%2E+Lin%2C+and+Noah+A%2E+Smith%2E+2020%2E+Parsing+with+multilingual+BERT%2C+a+small+corpus%2C+and+a+small+treebank%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2020%2C+pag)
30. Stanley F. Chen and Joshua Goodman. 1999. An empirical study of smoothing techniques for language modeling. Computer Speech & Language, 13(4):359–394. [find](https://scholar.google.com/scholar?q=Stanley+F%2E+Chen+and+Joshua+Goodman%2E+1999%2E+An+empirical+study+of+smoothing+techniques+for+language+modeling%2E+Computer+Speech+%26+Language%2C+13%284%29%3A359%E2%80%93394%2E)
31. Colin Cherry, George Foster, Ankur Bapna, Orhan Firat, and Wolfgang Macherey. 2018. Revisiting character-based neural machine translation with capacity and compression. In EMNLP. [find](https://scholar.google.com/scholar?q=Colin+Cherry%2C+George+Foster%2C+Ankur+Bapna%2C+Orhan+Firat%2C+and+Wolfgang+Macherey%2E+2018%2E+Revisiting+character%2Dbased+neural+machine+translation+with+capacity+and+compression%2E+In+EMNLP%2E)
32. Dokook Choe, Rami Al-Rfou, Mandy Guo, Heeyoung Lee, and Noah Constant. 2019. Bridging the gap for tokenizer-free language models. [find](https://scholar.google.com/scholar?q=Dokook+Choe%2C+Rami+Al%2DRfou%2C+Mandy+Guo%2C+Heeyoung+Lee%2C+and+Noah+Constant%2E+2019%2E+Bridging+the+gap+for+tokenizer%2Dfree+language+models%2E)
33. Grzegorz Chrupała. 2013. Text segmentation with character-level text embeddings. Workshop on Deep Learning for Audio, Speech and Language Processing, ICML 2013; Workshop on Deep Learning for Audio, Speech and Language Processing, ICML 2013 ; Conference date: 16-06-2013. [find](https://scholar.google.com/scholar?q=Grzegorz+Chrupa%C5%82a%2E+2013%2E+Text+segmentation+with+character%2Dlevel+text+embeddings%2E+Workshop+on+Deep+Learning+for+Audio%2C+Speech+and+Language+Processing%2C+ICML+2013%3B+Workshop+on+Deep+Learning+for+Audio%2C+Sp)
34. Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, and Jason Riesa. 2020. Improving multilingual models with language-clustered vocabularies. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4536–4546, Online. Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Dan+Garrette%2C+Kiat+Chuan+Tan%2C+and+Jason+Riesa%2E+2020%2E+Improving+multilingual+models+with+language%2Dclustered+vocabularies%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+)
35. Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. 2017. Hierarchical multiscale recurrent neural networks. In International Conference on Learning Representations 2017. [find](https://scholar.google.com/scholar?q=Junyoung+Chung%2C+Sungjin+Ahn%2C+and+Yoshua+Bengio%2E+2017%2E+Hierarchical+multiscale+recurrent+neural+networks%2E+In+International+Conference+on+Learning+Representations+2017%2E)
36. Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. 2016. A character-level decoder without explicit segmentation for neural machine translation. arXiv preprint arXiv:1603.06147. [our card](https://md.research.elamur.ai/free/en/papers/chung-2016-character-level-decoder-without-explicit/)
37. Kenneth W. Church. 2000. Empirical estimates of adaptation: The chance of two Noriegas is closer to 𝑝/2 than 𝑝 2 . In Proceedings of the 18th Conference on Computational Linguistics - Volume 1, pages 180–186. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kenneth+W%2E+Church%2E+2000%2E+Empirical+estimates+of+adaptation%3A+The+chance+of+two+Noriegas+is+closer+to+%F0%9D%91%9D%2F2+than+%F0%9D%91%9D+2+%2E+In+Proceedings+of+the+18th+Conference+on+Computational+Linguistics+%2D+Volume+1%2C+pages+)
38. Jonathan H Clark, Dan Garrette, Iulia Turc, and John Wieting. 2021. Canine: Pre-training an efficient tokenization-free encoder for language representation. arXiv preprint arXiv:2103.06874. [our card](https://md.research.elamur.ai/free/en/papers/clark-2022-canine-pre-training-efficient-tokenization/)
39. Lionel Clément, Eric De la Clergerie, and Lionel Net. 2005. Maf: a morphosyntactic annotation framework. [find](https://scholar.google.com/scholar?q=Lionel+Cl%C3%A9ment%2C+Eric+De+la+Clergerie%2C+and+Lionel+Net%2E+2005%2E+Maf%3A+a+morphosyntactic+annotation+framework%2E)
40. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning, pages 160–167. [find](https://scholar.google.com/scholar?q=Ronan+Collobert+and+Jason+Weston%2E+2008%2E+A+unified+architecture+for+natural+language+processing%3A+Deep+neural+networks+with+multitask+learning%2E+In+Proceedings+of+the+25th+International+Conference+on+Mac)
41. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dlingu)
42. Alexis CONNEAU and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Alexis+CONNEAU+and+Guillaume+Lample%2E+2019%2E+Cross%2Dlingual+language+model+pretraining%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+32%2E+Curran+Associates%2C+Inc%2E)
43. Alexis Conneau, Holger Schwenk, Loïc Barrault, and Yann Lecun. 2017. Very deep convolutional networks for text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1107–1116, Valencia, Spain. Asso… [our card](https://md.research.elamur.ai/free/en/papers/conneau-2017-very-deep-convolutional-networks-text/)
44. Marta R. Costa-jussà, David Aldón, and José A. R. Fonollosa. 2017. Chinese–spanish neural machine translation enhanced with character and word bitmap fonts. Machine Translation, 31(1):35–47. [find](https://scholar.google.com/scholar?q=Marta+R%2E+Costa%2Djuss%C3%A0%2C+David+Ald%C3%B3n%2C+and+Jos%C3%A9+A%2E+R%2E+Fonollosa%2E+2017%2E+Chinese%E2%80%93spanish+neural+machine+translation+enhanced+with+character+and+word+bitmap+fonts%2E+Machine+Translation%2C+31%281%29%3A35%E2%80%9347%2E)
45. Ryan Cotterell, Sabrina J. Mielke, Jason Eisner, and Brian Roark. 2018. Are all languages equally hard to language-model? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages… [find](https://scholar.google.com/scholar?q=Ryan+Cotterell%2C+Sabrina+J%2E+Mielke%2C+Jason+Eisner%2C+and+Brian+Roark%2E+2018%2E+Are+all+languages+equally+hard+to+language%2Dmodel%3F+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Ass)
46. Mathias Creutz and Krista Lagus. 2002. Unsupervised discovery of morphemes. In Proceedings of the ACL-02 Workshop on Morphological and Phonological Learning, pages 21–30. [find](https://scholar.google.com/scholar?q=Mathias+Creutz+and+Krista+Lagus%2E+2002%2E+Unsupervised+discovery+of+morphemes%2E+In+Proceedings+of+the+ACL%2D02+Workshop+on+Morphological+and+Phonological+Learning%2C+pages+21%E2%80%9330%2E)
47. Mathias Creutz and Krista Lagus. 2005. Inducing the morphological lexicon of a natural language from unannotated text. In Proceedings of the International and Interdisciplinary Conference on Adaptive Knowledge Representation and Reasoning (AKRR’05), volume 1, pages 51–59. [find](https://scholar.google.com/scholar?q=Mathias+Creutz+and+Krista+Lagus%2E+2005%2E+Inducing+the+morphological+lexicon+of+a+natural+language+from+unannotated+text%2E+In+Proceedings+of+the+International+and+Interdisciplinary+Conference+on+Adaptive+)
48. Mathias Creutz and Krista Lagus. 2007. Unsupervised models for morpheme segmentation and morphology learning. ACM Trans. Speech Lang. Process., 4(1). [find](https://scholar.google.com/scholar?q=Mathias+Creutz+and+Krista+Lagus%2E+2007%2E+Unsupervised+models+for+morpheme+segmentation+and+morphology+learning%2E+ACM+Trans%2E+Speech+Lang%2E+Process%2E%2C+4%281%29%2E)
49. Falcon Dai and Zheng Cai. 2017. Glyph-aware embedding of Chinese characters. In Proceedings of the First Workshop on Subword and Character Level Models in NLP, pages 64–69, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Falcon+Dai+and+Zheng+Cai%2E+2017%2E+Glyph%2Daware+embedding+of+Chinese+characters%2E+In+Proceedings+of+the+First+Workshop+on+Subword+and+Character+Level+Models+in+NLP%2C+pages+64%E2%80%9369%2C+Copenhagen%2C+Denmark%2E+Associ)
50. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
51. Shuoyang Ding, Adithya Renduchintala, and Kevin Duh. 2019. A call for prudent choice of subword merge operations in neural machine translation. In Proceedings of Machine Translation Summit XVII Volume 1: Research Track, pages 204–213, Dublin, Ireland. European Association for Machine Translation. [find](https://scholar.google.com/scholar?q=Shuoyang+Ding%2C+Adithya+Renduchintala%2C+and+Kevin+Duh%2E+2019%2E+A+call+for+prudent+choice+of+subword+merge+operations+in+neural+machine+translation%2E+In+Proceedings+of+Machine+Translation+Summit+XVII+Volume)
52. M. Domingo, M. Garcıa-Martınez, A. Helle, F. Casacuberta, and M. Herranz. 2018. How Much Does Tokenization Affect Neural Machine Translation? arXiv e-prints. [find](https://scholar.google.com/scholar?q=M%2E+Domingo%2C+M%2E+Garc%C4%B1a%2DMart%C4%B1nez%2C+A%2E+Helle%2C+F%2E+Casacuberta%2C+and+M%2E+Herranz%2E+2018%2E+How+Much+Does+Tokenization+Affect+Neural+Machine+Translation%3F+arXiv+e%2Dprints%2E)
53. Miguel Domingo, Mercedes Garcıa-Martınez, Alexandre Helle, Francisco Casacuberta, and Manuel Herranz. 2018. How much does tokenization affect neural machine translation? arXiv preprint arXiv:1812.08621. — [arXiv:1812.08621](https://arxiv.org/abs/1812.08621) [find](https://scholar.google.com/scholar?q=Miguel+Domingo%2C+Mercedes+Garc%C4%B1a%2DMart%C4%B1nez%2C+Alexandre+Helle%2C+Francisco+Casacuberta%2C+and+Manuel+Herranz%2E+2018%2E+How+much+does+tokenization+affect+neural+machine+translation%3F+arXiv+preprint+arXiv%3A1812%2E0862)
54. Bonaventure F. P. Dossou and Chris C. Emezue. 2021. Crowdsourced phrase-based tokenization for low-resourced neural machine translation: The case of fon language. [find](https://scholar.google.com/scholar?q=Bonaventure+F%2E+P%2E+Dossou+and+Chris+C%2E+Emezue%2E+2021%2E+Crowdsourced+phrase%2Dbased+tokenization+for+low%2Dresourced+neural+machine+translation%3A+The+case+of+fon+language%2E)
55. Yerai Doval and Carlos Gómez-Rodríguez. 2019. Comparing neural- and n-gram-based language models for word segmentation. Journal of the Association for Information Science and Technology, 70(2):187–197. [find](https://scholar.google.com/scholar?q=Yerai+Doval+and+Carlos+G%C3%B3mez%2DRodr%C3%ADguez%2E+2019%2E+Comparing+neural%2D+and+n%2Dgram%2Dbased+language+models+for+word+segmentation%2E+Journal+of+the+Association+for+Information+Science+and+Technology%2C+70%282%29%3A187%E2%80%93197)
56. C. M. Downey, Fei Xia, Gina-Anne Levow, and Shane Steinert-Threlkeld. 2021. A masked segmental language model for unsupervised natural language segmentation. [find](https://scholar.google.com/scholar?q=C%2E+M%2E+Downey%2C+Fei+Xia%2C+Gina%2DAnne+Levow%2C+and+Shane+Steinert%2DThrelkeld%2E+2021%2E+A+masked+segmental+language+model+for+unsupervised+natural+language+segmentation%2E)
57. Abteen Ebrahimi and Katharina Kann. 2021. How to adapt your pretrained multilingual model to 1600 languages. [find](https://scholar.google.com/scholar?q=Abteen+Ebrahimi+and+Katharina+Kann%2E+2021%2E+How+to+adapt+your+pretrained+multilingual+model+to+1600+languages%2E)
58. Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, and Jun’ichi Tsujii. 2020. CharacterBERT: Reconciling ELMo and BERT for word-level open-vocabulary representations from characters. In Proceedings of the 28th International Conference on Computational Linguistics… [find](https://scholar.google.com/scholar?q=Hicham+El+Boukkouri%2C+Olivier+Ferret%2C+Thomas+Lavergne%2C+Hiroshi+Noji%2C+Pierre+Zweigenbaum%2C+and+Jun%E2%80%99ichi+Tsujii%2E+2020%2E+CharacterBERT%3A+Reconciling+ELMo+and+BERT+for+word%2Dlevel+open%2Dvocabulary+representatio)
59. Salah El Hihi and Yoshua Bengio. 1995. Hierarchical recurrent neural networks for long-term dependencies. In Proceedings of the 8th International Conference on Neural Information Processing Systems, NIPS’95, page 493–499, Cambridge, MA, USA. MIT Press. [find](https://scholar.google.com/scholar?q=Salah+El+Hihi+and+Yoshua+Bengio%2E+1995%2E+Hierarchical+recurrent+neural+networks+for+long%2Dterm+dependencies%2E+In+Proceedings+of+the+8th+International+Conference+on+Neural+Information+Processing+Systems%2C+N)
60. Jeffrey L. Elman. 1990. Finding structure in time. Cognitive Science, 14(2):179–211. [find](https://scholar.google.com/scholar?q=Jeffrey+L%2E+Elman%2E+1990%2E+Finding+structure+in+time%2E+Cognitive+Science%2C+14%282%29%3A179%E2%80%93211%2E)
61. Micha Elsner, Sharon Goldwater, Naomi Feldman, and Frank Wood. 2013. A joint learning model of word segmentation, lexical acquisition, and phonetic variability. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 42–54, Seattle, Washington, USA. Associati… [find](https://scholar.google.com/scholar?q=Micha+Elsner%2C+Sharon+Goldwater%2C+Naomi+Feldman%2C+and+Frank+Wood%2E+2013%2E+A+joint+learning+model+of+word+segmentation%2C+lexical+acquisition%2C+and+phonetic+variability%2E+In+Proceedings+of+the+2013+Conference+o)
62. Philip Gage. 1994. A new algorithm for data compression. C Users J., 12(2):23–38. [find](https://scholar.google.com/scholar?q=Philip+Gage%2E+1994%2E+A+new+algorithm+for+data+compression%2E+C+Users+J%2E%2C+12%282%29%3A23%E2%80%9338%2E)
63. Marco Gaido, Beatrice Savoldi, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2021. How to split: the effect of word segmentation on gender bias in speech translation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3576–3589, Online. Association for Computati… [find](https://scholar.google.com/scholar?q=Marco+Gaido%2C+Beatrice+Savoldi%2C+Luisa+Bentivogli%2C+Matteo+Negri%2C+and+Marco+Turchi%2E+2021%2E+How+to+split%3A+the+effect+of+word+segmentation+on+gender+bias+in+speech+translation%2E+In+Findings+of+the+Associatio)
64. Matthias Gallé. 2019. Investigating the effectiveness of BPE: The power of shorter sequences. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1375–1381, Hong Kong… [find](https://scholar.google.com/scholar?q=Matthias+Gall%C3%A9%2E+2019%2E+Investigating+the+effectiveness+of+BPE%3A+The+power+of+shorter+sequences%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+9th+Inte)
65. Yingqiang Gao, Nikola I Nikolov, Yuhuang Hu, and Richard HR Hahnloser. 2020. Character-level translation with self-attention. arXiv preprint arXiv:2004.14788. [find](https://scholar.google.com/scholar?q=Yingqiang+Gao%2C+Nikola+I+Nikolov%2C+Yuhuang+Hu%2C+and+Richard+HR+Hahnloser%2E+2020%2E+Character%2Dlevel+translation+with+self%2Dattention%2E+arXiv+preprint+arXiv%3A2004%2E14788%2E)
66. John Goldsmith. 2001. Unsupervised learning of the morphology of a natural language. Computational linguistics, 27(2):153–198. [find](https://scholar.google.com/scholar?q=John+Goldsmith%2E+2001%2E+Unsupervised+learning+of+the+morphology+of+a+natural+language%2E+Computational+linguistics%2C+27%282%29%3A153%E2%80%93198%2E)
67. S. Goldwater, T. L. Griffiths, and M. Johnson. 2006a. Contextual dependencies in unsupervised word segmentation. In Proc. of COLING-ACL. [find](https://scholar.google.com/scholar?q=S%2E+Goldwater%2C+T%2E+L%2E+Griffiths%2C+and+M%2E+Johnson%2E+2006a%2E+Contextual+dependencies+in+unsupervised+word+segmentation%2E+In+Proc%2E+of+COLING%2DACL%2E)
68. Sharon Goldwater, Thomas L. Griffiths, and Mark Johnson. 2009. A Bayesian framework for word segmentation: Exploring the effects of context. Cognition, 112(1):21–54. [find](https://scholar.google.com/scholar?q=Sharon+Goldwater%2C+Thomas+L%2E+Griffiths%2C+and+Mark+Johnson%2E+2009%2E+A+Bayesian+framework+for+word+segmentation%3A+Exploring+the+effects+of+context%2E+Cognition%2C+112%281%29%3A21%E2%80%9354%2E)
69. Sharon Goldwater, Thomas L. Griffiths, and Mark Johnson. 2011. Producing power-law distributions and damping word frequencies with two-stage language models. Journal of Machine Learning Research, 12(Jul):2335–2382. [find](https://scholar.google.com/scholar?q=Sharon+Goldwater%2C+Thomas+L%2E+Griffiths%2C+and+Mark+Johnson%2E+2011%2E+Producing+power%2Dlaw+distributions+and+damping+word+frequencies+with+two%2Dstage+language+models%2E+Journal+of+Machine+Learning+Research%2C+12%28J)
70. Sharon Goldwater, Mark Johnson, and Thomas Griffiths. 2006b. Interpolating between types and tokens by estimating power-law generators. In Advances in Neural Information Processing Systems, volume 18. MIT Press. [find](https://scholar.google.com/scholar?q=Sharon+Goldwater%2C+Mark+Johnson%2C+and+Thomas+Griffiths%2E+2006b%2E+Interpolating+between+types+and+tokens+by+estimating+power%2Dlaw+generators%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+18%2E)
71. Thamme Gowda and Jonathan May. 2020. Finding the optimal vocabulary size for neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3955–3964, Online. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/gowda-2020-finding-optimal-vocabulary-size-neural/)
72. Edouard Grave, Armand Joulin, and Nicolas Usunier. 2016. Improving neural language models with a continuous cache. In International Conference on Learning Representations 2016. [find](https://scholar.google.com/scholar?q=Edouard+Grave%2C+Armand+Joulin%2C+and+Nicolas+Usunier%2E+2016%2E+Improving+neural+language+models+with+a+continuous+cache%2E+In+International+Conference+on+Learning+Representations+2016%2E)
73. Edouard Grave, Sainbayar Sukhbaatar, Piotr Bojanowski, and Armand Joulin. 2019. Training hybrid language models by marginalizing over segmentations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1477–1482, Florence, Italy. Association for Computati… [find](https://scholar.google.com/scholar?q=Edouard+Grave%2C+Sainbayar+Sukhbaatar%2C+Piotr+Bojanowski%2C+and+Armand+Joulin%2E+2019%2E+Training+hybrid+language+models+by+marginalizing+over+segmentations%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+As)
74. Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850. [find](https://scholar.google.com/scholar?q=Alex+Graves%2E+2013%2E+Generating+sequences+with+recurrent+neural+networks%2E+arXiv+preprint+arXiv%3A1308%2E0850%2E)
75. Stig-Arne Grönroos, Sami Virpioja, and Mikko Kurimo. 2020. Morfessor EM+Prune: Improved subword segmentation with expectation maximization and pruning. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 3944–3953, Marseille, France. European Language Resources Association… [find](https://scholar.google.com/scholar?q=Stig%2DArne+Gr%C3%B6nroos%2C+Sami+Virpioja%2C+and+Mikko+Kurimo%2E+2020%2E+Morfessor+EM%2BPrune%3A+Improved+subword+segmentation+with+expectation+maximization+and+pruning%2E+In+Proceedings+of+the+12th+Language+Resources+an)
76. Stig-Arne Grönroos, Sami Virpioja, Peter Smit, and Mikko Kurimo. 2014. Morfessor FlatCat: An HMM-based method for unsupervised and semi-supervised learning of morphology. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1177–1185. [find](https://scholar.google.com/scholar?q=Stig%2DArne+Gr%C3%B6nroos%2C+Sami+Virpioja%2C+Peter+Smit%2C+and+Mikko+Kurimo%2E+2014%2E+Morfessor+FlatCat%3A+An+HMM%2Dbased+method+for+unsupervised+and+semi%2Dsupervised+learning+of+morphology%2E+In+Proceedings+of+COLING+2014)
77. Rohit Gupta, Laurent Besacier, Marc Dymetman, and Matthias Gallé. 2019. Character-based nmt with transformer. arXiv preprint arXiv:1911.04997. — [arXiv:1911.04997](https://arxiv.org/abs/1911.04997) [find](https://scholar.google.com/scholar?q=Rohit+Gupta%2C+Laurent+Besacier%2C+Marc+Dymetman%2C+and+Matthias+Gall%C3%A9%2E+2019%2E+Character%2Dbased+nmt+with+transformer%2E+arXiv+preprint+arXiv%3A1911%2E04997%2E+%E2%80%94+%5BarXiv%3A1911%2E04997%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1911%2E04997%29)
78. Ximena Gutierrez-Vasques, Christian Bentz, Olga Sozinova, and Tanja Samardzic. 2021. From characters to words: the turning point of BPE merges. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3454–3468, Online. Associ… [find](https://scholar.google.com/scholar?q=Ximena+Gutierrez%2DVasques%2C+Christian+Bentz%2C+Olga+Sozinova%2C+and+Tanja+Samardzic%2E+2021%2E+From+characters+to+words%3A+the+turning+point+of+BPE+merges%2E+In+Proceedings+of+the+16th+Conference+of+the+European+Ch)
79. Xuanli He, Gholamreza Haffari, and Mohammad Norouzi. 2020. Dynamic programming encoding for subword segmentation in neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3042–3051, Online. Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Xuanli+He%2C+Gholamreza+Haffari%2C+and+Mohammad+Norouzi%2E+2020%2E+Dynamic+programming+encoding+for+subword+segmentation+in+neural+machine+translation%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Associa)
80. James Henderson. 2020. The unstoppable rise of computational linguistics in deep learning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6294–6306, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=James+Henderson%2E+2020%2E+The+unstoppable+rise+of+computational+linguistics+in+deep+learning%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+6294%E2%80%936306%2C+)
81. Tatsuya Hiraoka, Hiroyuki Shindo, and Yuji Matsumoto. 2019. Stochastic tokenization with a language model for neural text classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1620–1629, Florence, Italy. Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Tatsuya+Hiraoka%2C+Hiroyuki+Shindo%2C+and+Yuji+Matsumoto%2E+2019%2E+Stochastic+tokenization+with+a+language+model+for+neural+text+classification%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+f)
82. Tatsuya Hiraoka, Sho Takase, Kei Uchiumi, Atsushi Keyaki, and Naoaki Okazaki. 2020. Optimizing word segmentation for downstream task. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1341–1351, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tatsuya+Hiraoka%2C+Sho+Takase%2C+Kei+Uchiumi%2C+Atsushi+Keyaki%2C+and+Naoaki+Okazaki%2E+2020%2E+Optimizing+word+segmentation+for+downstream+task%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNL)
83. Tatsuya Hiraoka, Sho Takase, Kei Uchiumi, Atsushi Keyaki, and Naoaki Okazaki. 2021. Joint optimization of tokenization and downstream model. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 244–255, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tatsuya+Hiraoka%2C+Sho+Takase%2C+Kei+Uchiumi%2C+Atsushi+Keyaki%2C+and+Naoaki+Okazaki%2E+2021%2E+Joint+optimization+of+tokenization+and+downstream+model%2E+In+Findings+of+the+Association+for+Computational+Linguistic)
84. Valentin Hofmann, Janet Pierrehumbert, and Hinrich Schütze. 2021. Superbizarre is not superb: Derivational morphology improves BERT’s interpretation of complex words. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conferenc… [our card](https://md.research.elamur.ai/free/en/papers/hofmann-2021-superbizarre-derivational/)
85. Po-Sen Huang, Chong Wang, Sitao Huang, Dengyong Zhou, and Li Deng. 2018. Towards neural phrase-based machine translation. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Po%2DSen+Huang%2C+Chong+Wang%2C+Sitao+Huang%2C+Dengyong+Zhou%2C+and+Li+Deng%2E+2018%2E+Towards+neural+phrase%2Dbased+machine+translation%2E+In+International+Conference+on+Learning+Representations%2E)
86. Matthias Huck, Simon Riess, and Alexander Fraser. 2017. Target-side word segmentation strategies for neural machine translation. In Proceedings of the Second Conference on Machine Translation, pages 56–67. [find](https://scholar.google.com/scholar?q=Matthias+Huck%2C+Simon+Riess%2C+and+Alexander+Fraser%2E+2017%2E+Target%2Dside+word+segmentation+strategies+for+neural+machine+translation%2E+In+Proceedings+of+the+Second+Conference+on+Machine+Translation%2C+pages+5)
87. Gérard Huet. 2003. Lexicon-directed segmentation and tagging of sanskrit. In in « XIIth World Sanskrit Conference, pages 307–325. [find](https://scholar.google.com/scholar?q=G%C3%A9rard+Huet%2E+2003%2E+Lexicon%2Ddirected+segmentation+and+tagging+of+sanskrit%2E+In+in+%C2%AB+XIIth+World+Sanskrit+Conference%2C+pages+307%E2%80%93325%2E)
88. Kyuyeon Hwang and Wonyong Sung. 2017. Character-level language modeling with hierarchical recurrent neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on, pages 5720– 5724. IEEE. [find](https://scholar.google.com/scholar?q=Kyuyeon+Hwang+and+Wonyong+Sung%2E+2017%2E+Character%2Dlevel+language+modeling+with+hierarchical+recurrent+neural+networks%2E+In+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2C+2017+IEEE+International+Confer)
89. Itay Itzhak and Omer Levy. 2021. Models in a spelling bee: Language models implicitly learn the character composition of tokens. [find](https://scholar.google.com/scholar?q=Itay+Itzhak+and+Omer+Levy%2E+2021%2E+Models+in+a+spelling+bee%3A+Language+models+implicitly+learn+the+character+composition+of+tokens%2E)
90. Miguel Ángel Jiménez-Montaño. 1984. On the syntactic structure of protein sequences and the concept of grammar complexity. Bulletin of Mathematical Biology, 46(4):641–659. [find](https://scholar.google.com/scholar?q=Miguel+%C3%81ngel+Jim%C3%A9nez%2DMonta%C3%B1o%2E+1984%2E+On+the+syntactic+structure+of+protein+sequences+and+the+concept+of+grammar+complexity%2E+Bulletin+of+Mathematical+Biology%2C+46%284%29%3A641%E2%80%93659%2E)
91. Mark Johnson and Sharon Goldwater. 2009. Improving nonparameteric Bayesian inference: experiments on unsupervised word segmentation with adaptor grammars. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Mark+Johnson+and+Sharon+Goldwater%2E+2009%2E+Improving+nonparameteric+Bayesian+inference%3A+experiments+on+unsupervised+word+segmentation+with+adaptor+grammars%2E+In+Proceedings+of+Human+Language+Technologies)
92. Mark Johnson, Thomas L. Griffiths, and Sharon Goldwater. 2007. Adaptor grammars: A framework for specifying compositional nonparametric Bayesian models. In Advances in Neural Information Processing Systems 19, pages 641– 648. MIT Press. [find](https://scholar.google.com/scholar?q=Mark+Johnson%2C+Thomas+L%2E+Griffiths%2C+and+Sharon+Goldwater%2E+2007%2E+Adaptor+grammars%3A+A+framework+for+specifying+compositional+nonparametric+Bayesian+models%2E+In+Advances+in+Neural+Information+Processing+Sy)
93. Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Ass… [find](https://scholar.google.com/scholar?q=Melvin+Johnson%2C+Mike+Schuster%2C+Quoc+V%2E+Le%2C+Maxim+Krikun%2C+Yonghui+Wu%2C+Zhifeng+Chen%2C+Nikhil+Thorat%2C+Fernanda+Vi%C3%A9gas%2C+Martin+Wattenberg%2C+Greg+Corrado%2C+Macduff+Hughes%2C+and+Jeffrey+Dean%2E+2017%2E+Google%E2%80%99s+mul)
94. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. [find](https://scholar.google.com/scholar?q=Rafal+Jozefowicz%2C+Oriol+Vinyals%2C+Mike+Schuster%2C+Noam+Shazeer%2C+and+Yonghui+Wu%2E+2016%2E+Exploring+the+limits+of+language+modeling%2E)
95. Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual bert: An empirical study. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Karthikeyan+K%2C+Zihan+Wang%2C+Stephen+Mayhew%2C+and+Dan+Roth%2E+2020%2E+Cross%2Dlingual+ability+of+multilingual+bert%3A+An+empirical+study%2E+In+International+Conference+on+Learning+Representations%2E)
96. Ákos Kádár, Marc-Alexandre Côté, Grzegorz Chrupała, and Afra Alishahi. 2018. Revisiting the hierarchical multiscale lstm. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3215–3227. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=%C3%81kos+K%C3%A1d%C3%A1r%2C+Marc%2DAlexandre+C%C3%B4t%C3%A9%2C+Grzegorz+Chrupa%C5%82a%2C+and+Afra+Alishahi%2E+2018%2E+Revisiting+the+hierarchical+multiscale+lstm%2E+In+Proceedings+of+the+27th+International+Conference+on+Computational+Linguisti)
97. Kazuya Kawakami, Chris Dyer, and Phil Blunsom. 2017. Learning to create and reuse words in open-vocabulary neural language modeling. In Proceedings of the 55th Annual Meeting of the ACL (Volume 1: Long Papers), pages 1492– 1502, Vancouver, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kazuya+Kawakami%2C+Chris+Dyer%2C+and+Phil+Blunsom%2E+2017%2E+Learning+to+create+and+reuse+words+in+open%2Dvocabulary+neural+language+modeling%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+ACL+%28Volume+1%3A+Lon)
98. Kazuya Kawakami, Chris Dyer, and Phil Blunsom. 2019. Learning to discover, ground and use words with segmental neural language models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6429–6441, Florence, Italy. Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Kazuya+Kawakami%2C+Chris+Dyer%2C+and+Phil+Blunsom%2E+2019%2E+Learning+to+discover%2C+ground+and+use+words+with+segmental+neural+language+models%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+)
99. Eugene Kharitonov, Marco Baroni, and Dieuwke Hupkes. 2021. How bpe affects memorization in transformers. [find](https://scholar.google.com/scholar?q=Eugene+Kharitonov%2C+Marco+Baroni%2C+and+Dieuwke+Hupkes%2E+2021%2E+How+bpe+affects+memorization+in+transformers%2E)
100. Yoon Kim, Yacine Jernite, David Sontag, and Alexander Rush. 2016. Character-aware neural language models. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1). [our card](https://md.research.elamur.ai/free/en/papers/kim-2015-character-aware-neural-language-models/)
101. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Pro… [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2C+Hieu+Hoang%2C+Alexandra+Birch%2C+Chris+Callison%2DBurch%2C+Marcello+Federico%2C+Nicola+Bertoldi%2C+Brooke+Cowan%2C+Wade+Shen%2C+Christine+Moran%2C+Richard+Zens%2C+Chris+Dyer%2C+Ondr%CC%8Cej+Bojar%2C+Alexandra+Const)
102. Philipp Koehn and Kevin Knight. 2003. Empirical methods for compound splitting. In 10th Conference of the European Chapter of the Association for Computational Linguistics, Budapest, Hungary. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/koehn-2003-empirical-methods-compound-splitting/)
103. Lingpeng Kong, Chris Dyer, and Noah A. Smith. 2016. Segmental recurrent neural networks. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Lingpeng+Kong%2C+Chris+Dyer%2C+and+Noah+A%2E+Smith%2E+2016%2E+Segmental+recurrent+neural+networks%2E+In+International+Conference+on+Learning+Representations%2E)
104. Jan Koutnik, Klaus Greff, Faustino Gomez, and Juergen Schmidhuber. 2014. A clockwork rnn. In Proceedings of Machine Learning Research, volume 32, pages 1863–1871, Bejing, China. PMLR. [find](https://scholar.google.com/scholar?q=Jan+Koutnik%2C+Klaus+Greff%2C+Faustino+Gomez%2C+and+Juergen+Schmidhuber%2E+2014%2E+A+clockwork+rnn%2E+In+Proceedings+of+Machine+Learning+Research%2C+volume+32%2C+pages+1863%E2%80%931871%2C+Bejing%2C+China%2E+PMLR%2E)
105. Julia Kreutzer and Artem Sokolov. 2018. Learning to Segment Inputs for NMT Favors Character-Level Processing. In Proceedings of the 15th International Workshop on Spoken Language Translation (IWSLT). [find](https://scholar.google.com/scholar?q=Julia+Kreutzer+and+Artem+Sokolov%2E+2018%2E+Learning+to+Segment+Inputs+for+NMT+Favors+Character%2DLevel+Processing%2E+In+Proceedings+of+the+15th+International+Workshop+on+Spoken+Language+Translation+%28IWSLT%29%2E)
106. Amrith Krishna, Pavan Kumar Satuluri, and Pawan Goyal. 2017. A dataset for Sanskrit word segmentation. In Proceedings of the Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 105–114, Vancouver, Canada. Association for Computa… [find](https://scholar.google.com/scholar?q=Amrith+Krishna%2C+Pavan+Kumar+Satuluri%2C+and+Pawan+Goyal%2E+2017%2E+A+dataset+for+Sanskrit+word+segmentation%2E+In+Proceedings+of+the+Joint+SIGHUM+Workshop+on+Computational+Linguistics+for+Cultural+Heritage%2C+S)
107. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia. Association for Computational… [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-subword-regularization-improving-neural-network/)
108. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Associ… [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/)
109. Mikko Kurimo, Sami Virpioja, Ville Turunen, and Krista Lagus. 2010. Morpho Challenge 2005-2010: Evaluations and Results. In Proceedings of the 11th Meeting of the ACL Special Interest Group on Computational Morphology and Phonology, pages 87–95, Uppsala, Sweden. Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Mikko+Kurimo%2C+Sami+Virpioja%2C+Ville+Turunen%2C+and+Krista+Lagus%2E+2010%2E+Morpho+Challenge+2005%2D2010%3A+Evaluations+and+Results%2E+In+Proceedings+of+the+11th+Meeting+of+the+ACL+Special+Interest+Group+on+Computa)
110. N Jesper Larsson and Alistair Moffat. 2000. Offline dictionary-based compression. Proceedings of the IEEE, 88(11):1722–1732. [find](https://scholar.google.com/scholar?q=N+Jesper+Larsson+and+Alistair+Moffat%2E+2000%2E+Offline+dictionary%2Dbased+compression%2E+Proceedings+of+the+IEEE%2C+88%2811%29%3A1722%E2%80%931732%2E)
111. Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2017. Fully character-level neural machine translation without explicit segmentation. Transactions of the Association for Computational Linguistics, 5(0):365–378. [our card](https://md.research.elamur.ai/free/en/papers/lee-2017-fully-character-level-neural-machine/)
112. Jindřich Libovický and Alexander Fraser. 2020. Towards reasonably-sized character-level transformer NMT by finetuning subword systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2572–2579, Online. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Jindr%CC%8Cich+Libovick%C3%BD+and+Alexander+Fraser%2E+2020%2E+Towards+reasonably%2Dsized+character%2Dlevel+transformer+NMT+by+finetuning+subword+systems%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Na)
113. Jindřich Libovický, Helmut Schmid, and Alexander Fraser. 2021. Why don’t people use character-level machine translation? [find](https://scholar.google.com/scholar?q=Jindr%CC%8Cich+Libovick%C3%BD%2C+Helmut+Schmid%2C+and+Alexander+Fraser%2E+2021%2E+Why+don%E2%80%99t+people+use+character%2Dlevel+machine+translation%3F)
114. Constantine Lignos. 2010. Learning from unseen data. In Proceedings of the Morpho Challenge 2010 Workshop, pages 35–38, Helsinki, Finland. Aalto University School of Science and Technology. [find](https://scholar.google.com/scholar?q=Constantine+Lignos%2E+2010%2E+Learning+from+unseen+data%2E+In+Proceedings+of+the+Morpho+Challenge+2010+Workshop%2C+pages+35%E2%80%9338%2C+Helsinki%2C+Finland%2E+Aalto+University+School+of+Science+and+Technology%2E)
115. Wang Ling, Chris Dyer, Alan W. Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Pro… [our card](https://md.research.elamur.ai/free/en/papers/ling-2015-finding-function-form-compositional-character/)
116. Frederick Liu, Han Lu, Chieh Lo, and Graham Neubig. 2017. Learning character-level compositionality with visual features. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2059–2068, Vancouver, Canada. Association for Computatio… [find](https://scholar.google.com/scholar?q=Frederick+Liu%2C+Han+Lu%2C+Chieh+Lo%2C+and+Graham+Neubig%2E+2017%2E+Learning+character%2Dlevel+compositionality+with+visual+features%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computational)
117. Xin Liu, Baosong Yang, Dayiheng Liu, Haibo Zhang, Weihua Luo, Min Zhang, Haiying Zhang, and Jinsong Su. 2021. Bridging subword gaps in pretrain-finetune paradigm for natural language generation. [find](https://scholar.google.com/scholar?q=Xin+Liu%2C+Baosong+Yang%2C+Dayiheng+Liu%2C+Haibo+Zhang%2C+Weihua+Luo%2C+Min+Zhang%2C+Haiying+Zhang%2C+and+Jinsong+Su%2E+2021%2E+Bridging+subword+gaps+in+pretrain%2Dfinetune+paradigm+for+natural+language+generation%2E)
118. Zhaoxin Luo and Michael Zhu. 2021. Recurrent neural networks with mixed hierarchical structures for natural language processing. CoRR, abs/2106.02562. — [arXiv:2106.02562](https://arxiv.org/abs/2106.02562) [find](https://scholar.google.com/scholar?q=Zhaoxin+Luo+and+Michael+Zhu%2E+2021%2E+Recurrent+neural+networks+with+mixed+hierarchical+structures+for+natural+language+processing%2E+CoRR%2C+abs%2F2106%2E02562%2E+%E2%80%94+%5BarXiv%3A2106%2E02562%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2106%2E0)
119. Minh-Thang Luong and Christopher D. Manning. 2016. Achieving open vocabulary neural machine translation with hybrid word-character models. In Proceedings of the 54th Annual Meeting of the ACL (Volume 1: Long Papers), pages 1054–1063, Berlin, Germany. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/luong-2016-achieving-open-vocabulary-neural-machine/)
120. Wentao Ma, Yiming Cui, Chenglei Si, Ting Liu, Shijin Wang, and Guoping Hu. 2020. Char-BERT: Character-aware Pre-trained Language Model. In COLING. [find](https://scholar.google.com/scholar?q=Wentao+Ma%2C+Yiming+Cui%2C+Chenglei+Si%2C+Ting+Liu%2C+Shijin+Wang%2C+and+Guoping+Hu%2E+2020%2E+Char%2DBERT%3A+Character%2Daware+Pre%2Dtrained+Language+Model%2E+In+COLING%2E)
121. Dominik Macháček, Jonáš Vidra, and Ondřej Bojar. 2018. Morphological and language-agnostic word segmentation for nmt. In International Conference on Text, Speech, and Dialogue, pages 277–284. Springer. [find](https://scholar.google.com/scholar?q=Dominik+Mach%C3%A1c%CC%8Cek%2C+Jon%C3%A1%C5%A1+Vidra%2C+and+Ondr%CC%8Cej+Bojar%2E+2018%2E+Morphological+and+language%2Dagnostic+word+segmentation+for+nmt%2E+In+International+Conference+on+Text%2C+Speech%2C+and+Dialogue%2C+pages+277%E2%80%93284%2E+Spring)
122. Klaus Macherey, Andrew Dai, David Talbot, Ashok Popat, and Franz Och. 2011. Language-independent compound splitting with morphological operations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1395–1404, Portland, Orego… [find](https://scholar.google.com/scholar?q=Klaus+Macherey%2C+Andrew+Dai%2C+David+Talbot%2C+Ashok+Popat%2C+and+Franz+Och%2E+2011%2E+Language%2Dindependent+compound+splitting+with+morphological+operations%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Asso)
123. David J. C. MacKay and Linda C. Bauman Peto. 1995. A hierarchical Dirichlet language model. Journal of Natural Language Engineering, 1(3):289–308. [find](https://scholar.google.com/scholar?q=David+J%2E+C%2E+MacKay+and+Linda+C%2E+Bauman+Peto%2E+1995%2E+A+hierarchical+Dirichlet+language+model%2E+Journal+of+Natural+Language+Engineering%2C+1%283%29%3A289%E2%80%93308%2E)
124. Elman Mansimov, Mitchell Stern, Mia Chen, Orhan Firat, Jakob Uszkoreit, and Puneet Jain. 2020. Towards end-to-end in-image neural machine translation. In Proceedings of the First International Workshop on Natural Language Processing Beyond Text, pages 70–74, Online. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Elman+Mansimov%2C+Mitchell+Stern%2C+Mia+Chen%2C+Orhan+Firat%2C+Jakob+Uszkoreit%2C+and+Puneet+Jain%2E+2020%2E+Towards+end%2Dto%2Dend+in%2Dimage+neural+machine+translation%2E+In+Proceedings+of+the+First+International+Worksho)
125. Carl de Marcken. 1996. Linguistic structure as composition and perturbation. In Proceedings of the 34th Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Carl+de+Marcken%2E+1996%2E+Linguistic+structure+as+composition+and+perturbation%2E+In+Proceedings+of+the+34th+Meeting+of+the+Association+for+Computational+Linguistics%2E)
126. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. [our card](https://md.research.elamur.ai/free/en/papers/marcus-1993-building-large-annotated-corpus-english/)
127. Antonis Maronikolakis, Philipp Dufter, and Hinrich Schütze. 2021. Wine is not v i n. on the compatibility of tokenizations across languages. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2382–2399, Punta Cana, Dominican Republic. Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Antonis+Maronikolakis%2C+Philipp+Dufter%2C+and+Hinrich+Sch%C3%BCtze%2E+2021%2E+Wine+is+not+v+i+n%2E+on+the+compatibility+of+tokenizations+across+languages%2E+In+Findings+of+the+Association+for+Computational+Linguistic)
128. Austin Matthews, Graham Neubig, and Chris Dyer. 2018. Using morphological knowledge in open-vocabulary neural language models. In HLT-NAACL. [find](https://scholar.google.com/scholar?q=Austin+Matthews%2C+Graham+Neubig%2C+and+Chris+Dyer%2E+2018%2E+Using+morphological+knowledge+in+open%2Dvocabulary+neural+language+models%2E+In+HLT%2DNAACL%2E)
129. Yuxian Meng, Wei Wu, Fei Wang, Xiaoya Li, Ping Nie, Fan Yin, Muyu Li, Qinghong Han, Xiaofei Sun, and Jiwei Li. 2019. Glyce: Glyph-vectors for chinese character representations. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Yuxian+Meng%2C+Wei+Wu%2C+Fei+Wang%2C+Xiaoya+Li%2C+Ping+Nie%2C+Fan+Yin%2C+Muyu+Li%2C+Qinghong+Han%2C+Xiaofei+Sun%2C+and+Jiwei+Li%2E+2019%2E+Glyce%3A+Glyph%2Dvectors+for+chinese+character+representations%2E+In+Advances+in+Neural+I)
130. Bart van Merriënboer, Amartya Sanyal, Hugo Larochelle, and Yoshua Bengio. 2017. Multi-scale sequence modeling with a learned dictionary. In MLSLP. [find](https://scholar.google.com/scholar?q=Bart+van+Merri%C3%ABnboer%2C+Amartya+Sanyal%2C+Hugo+Larochelle%2C+and+Yoshua+Bengio%2E+2017%2E+Multi%2Dscale+sequence+modeling+with+a+learned+dictionary%2E+In+MLSLP%2E)
131. Sabrina J. Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark, and Jason Eisner. 2019. What kind of language is hard to language-model? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4975–4989, Florence, Italy. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Sabrina+J%2E+Mielke%2C+Ryan+Cotterell%2C+Kyle+Gorman%2C+Brian+Roark%2C+and+Jason+Eisner%2E+2019%2E+What+kind+of+language+is+hard+to+language%2Dmodel%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+C)
132. Sabrina J. Mielke and Jason Eisner. 2018. Spell once, summon anywhere: A two-level open-vocabulary language model. CoRR, abs/1804.08205. — [arXiv:1804.08205](https://arxiv.org/abs/1804.08205) [find](https://scholar.google.com/scholar?q=Sabrina+J%2E+Mielke+and+Jason+Eisner%2E+2018%2E+Spell+once%2C+summon+anywhere%3A+A+two%2Dlevel+open%2Dvocabulary+language+model%2E+CoRR%2C+abs%2F1804%2E08205%2E+%E2%80%94+%5BarXiv%3A1804%2E08205%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1804%2E08205%29)
133. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems-Volume 2, pages 3111–3119. [our card](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/)
134. Tomáš Mikolov, Ilya Sutskever, Anoop Deoras, Hai-Son Le, Stefan Kombrink, and Jan Čer-nocký. 2012. Subword language modeling with neural networks. preprint (rejected from ICASSP 2012). [find](https://scholar.google.com/scholar?q=Tom%C3%A1%C5%A1+Mikolov%2C+Ilya+Sutskever%2C+Anoop+Deoras%2C+Hai%2DSon+Le%2C+Stefan+Kombrink%2C+and+Jan+C%CC%8Cer%2Dnock%C3%BD%2E+2012%2E+Subword+language+modeling+with+neural+networks%2E+preprint+%28rejected+from+ICASSP+2012%29%2E)
135. Daichi Mochihashi, Takeshi Yamada, and Naonori Ueda. 2009. Bayesian unsupervised word segmentation with nested pitman-yor language modeling. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the A… [find](https://scholar.google.com/scholar?q=Daichi+Mochihashi%2C+Takeshi+Yamada%2C+and+Naonori+Ueda%2E+2009%2E+Bayesian+unsupervised+word+segmentation+with+nested+pitman%2Dyor+language+modeling%2E+In+Proceedings+of+the+Joint+Conference+of+the+47th+Annual+M)
136. Christian Monson, Jaime Carbonell, Alon Lavie, and Lori Levin. 2007. ParaMor: Minimally supervised induction of paradigm structure and morphological analysis. In Proceedings of Ninth Meeting of the ACL Special Interest Group in Computational Morphology and Phonology, pages 117–125, Prague, Czech Rep… [find](https://scholar.google.com/scholar?q=Christian+Monson%2C+Jaime+Carbonell%2C+Alon+Lavie%2C+and+Lori+Levin%2E+2007%2E+ParaMor%3A+Minimally+supervised+induction+of+paradigm+structure+and+morphological+analysis%2E+In+Proceedings+of+Ninth+Meeting+of+the+AC)
137. Christian Monson, Kristy Hollingshead, and Brian Roark. 2009. Probabilistic paramor. In CLEF (Working Notes). Citeseer. [find](https://scholar.google.com/scholar?q=Christian+Monson%2C+Kristy+Hollingshead%2C+and+Brian+Roark%2E+2009%2E+Probabilistic+paramor%2E+In+CLEF+%28Working+Notes%29%2E+Citeseer%2E)
138. Amir More, Özlem Çetinoğlu, Çağrı Çöltekin, Nizar Habash, Benoît Sagot, Djamé Seddah, Dima Taji, and Reut Tsarfaty. 2018. CoNLL-UL: Universal morphological lattices for Universal Dependency parsing. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC… [find](https://scholar.google.com/scholar?q=Amir+More%2C+%C3%96zlem+%C3%87etinog%CC%86lu%2C+%C3%87ag%CC%86r%C4%B1+%C3%87%C3%B6ltekin%2C+Nizar+Habash%2C+Beno%C3%AEt+Sagot%2C+Djam%C3%A9+Seddah%2C+Dima+Taji%2C+and+Reut+Tsarfaty%2E+2018%2E+CoNLL%2DUL%3A+Universal+morphological+lattices+for+Universal+Dependency+parsing%2E)
139. Karthik Narasimhan, Regina Barzilay, and Tommi Jaakkola. 2015. An unsupervised method for uncovering morphological chains. Transactions of the Association for Computational Linguistics, 3:157–167. [find](https://scholar.google.com/scholar?q=Karthik+Narasimhan%2C+Regina+Barzilay%2C+and+Tommi+Jaakkola%2E+2015%2E+An+unsupervised+method+for+uncovering+morphological+chains%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+3%3A157%E2%80%93167%2E)
140. Vít Novotný, Eniafe Festus Ayetiran, Dávid Lupták, Michal Stefánik, and Petr Sojka. 2021. One size does not fit all: Finding the optimal n-gram sizes for fasttext models across languages. CoRR, abs/2102.02585. — [arXiv:2102.02585](https://arxiv.org/abs/2102.02585) [find](https://scholar.google.com/scholar?q=V%C3%ADt+Novotn%C3%BD%2C+Eniafe+Festus+Ayetiran%2C+D%C3%A1vid+Lupt%C3%A1k%2C+Michal+Stef%C3%A1nik%2C+and+Petr+Sojka%2E+2021%2E+One+size+does+not+fit+all%3A+Finding+the+optimal+n%2Dgram+sizes+for+fasttext+models+across+languages%2E+CoRR%2C+abs%2F21)
141. Yirong Pan, Xiao Li, Yating Yang, and Rui Dong. 2020. Morphological word segmentation on agglutinative languages for neural machine translation. CoRR, abs/2001.01589. — [arXiv:2001.01589](https://arxiv.org/abs/2001.01589) [find](https://scholar.google.com/scholar?q=Yirong+Pan%2C+Xiao+Li%2C+Yating+Yang%2C+and+Rui+Dong%2E+2020%2E+Morphological+word+segmentation+on+agglutinative+languages+for+neural+machine+translation%2E+CoRR%2C+abs%2F2001%2E01589%2E+%E2%80%94+%5BarXiv%3A2001%2E01589%5D%28https%3A%2F%2Farxi)
142. Kyubyong Park, Joohong Lee, Seongbo Jang, and Dawoon Jung. 2020. An empirical study of tokenization strategies for various korean nlp tasks. In AACL/IJCNLP. [find](https://scholar.google.com/scholar?q=Kyubyong+Park%2C+Joohong+Lee%2C+Seongbo+Jang%2C+and+Dawoon+Jung%2E+2020%2E+An+empirical+study+of+tokenization+strategies+for+various+korean+nlp+tasks%2E+In+AACL%2FIJCNLP%2E)
143. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… [our card](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/)
144. Yuval Pinter. 2021. Integrating approaches to word representation. [find](https://scholar.google.com/scholar?q=Yuval+Pinter%2E+2021%2E+Integrating+approaches+to+word+representation%2E)
145. Yuval Pinter, Robert Guthrie, and Jacob Eisenstein. 2017. Mimicking word embeddings using subword RNNs. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 102–112, Copenhagen, Denmark. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yuval+Pinter%2C+Robert+Guthrie%2C+and+Jacob+Eisenstein%2E+2017%2E+Mimicking+word+embeddings+using+subword+RNNs%2E+In+Proceedings+of+the+2017+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages)
146. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996–5001, Florence, Italy. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/pires-2019-how-multilingual-is-multilingual-bert/)
147. Barbara Plank, Anders Søgaard, and Yoav Goldberg. 2016. Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 412–418, Berlin,… [our card](https://md.research.elamur.ai/free/en/papers/plank-2016-multilingual-part-speech-tagging-bidirectional/)
148. M. F. Porter. 1980. An algorithm for suffix stripping. Program, 14(3):130–137. [find](https://scholar.google.com/scholar?q=M%2E+F%2E+Porter%2E+1980%2E+An+algorithm+for+suffix+stripping%2E+Program%2C+14%283%29%3A130%E2%80%93137%2E)
149. Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2020. BPE-dropout: Simple and effective subword regularization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1882–1892, Online. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/provilkov-2020-bpe-dropout-simple-effective-subword/)
150. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. preprint. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+preprint%2E)
151. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
152. Jorma Rissanen. 1989. Stochastic Complexity in Statistical Inquiry Theory. World Scientific Publishing Co., Inc., USA. [find](https://scholar.google.com/scholar?q=Jorma+Rissanen%2E+1989%2E+Stochastic+Complexity+in+Statistical+Inquiry+Theory%2E+World+Scientific+Publishing+Co%2E%2C+Inc%2E%2C+USA%2E)
153. José Carlos Rosales Núñez, Guillaume Wisniewski, and Djamé Seddah. 2021. Noisy UGC translation at the character level: Revisiting open-vocabulary capabilities and robustness of char-based models. In Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), pages 199–211, Online.… [find](https://scholar.google.com/scholar?q=Jos%C3%A9+Carlos+Rosales+N%C3%BA%C3%B1ez%2C+Guillaume+Wisniewski%2C+and+Djam%C3%A9+Seddah%2E+2021%2E+Noisy+UGC+translation+at+the+character+level%3A+Revisiting+open%2Dvocabulary+capabilities+and+robustness+of+char%2Dbased+models%2E+In+P)
154. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Join… [our card](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/)
155. Benoît Sagot and Pierre Boullier. 2008. SxPipe 2: architecture pour le traitement pré-syntaxique de corpus bruts. Revue TAL, 49(2):155–188. [find](https://scholar.google.com/scholar?q=Beno%C3%AEt+Sagot+and+Pierre+Boullier%2E+2008%2E+SxPipe+2%3A+architecture+pour+le+traitement+pr%C3%A9%2Dsyntaxique+de+corpus+bruts%2E+Revue+TAL%2C+49%282%29%3A155%E2%80%93188%2E)
156. Elizabeth Salesky, David Etter, and Matt Post. 2021. Robust open-vocabulary translation from visual text representations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7235–7252, Online and Punta Cana, Dominican Republic. Association for Computation… [find](https://scholar.google.com/scholar?q=Elizabeth+Salesky%2C+David+Etter%2C+and+Matt+Post%2E+2021%2E+Robust+open%2Dvocabulary+translation+from+visual+text+representations%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language)
157. Elizabeth Salesky, Andrew Runge, Alex Coda, Jan Niehues, and Graham Neubig. 2018. Optimizing segmentation granularity for neural machine translation. arXiv preprint arXiv:1810.08641. — [arXiv:1810.08641](https://arxiv.org/abs/1810.08641) [find](https://scholar.google.com/scholar?q=Elizabeth+Salesky%2C+Andrew+Runge%2C+Alex+Coda%2C+Jan+Niehues%2C+and+Graham+Neubig%2E+2018%2E+Optimizing+segmentation+granularity+for+neural+machine+translation%2E+arXiv+preprint+arXiv%3A1810%2E08641%2E+%E2%80%94+%5BarXiv%3A1810%2E086)
158. Jonne Saleva and Constantine Lignos. 2021. The effectiveness of morphology-aware segmentation in low-resource neural machine translation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 164–174, Online.… [find](https://scholar.google.com/scholar?q=Jonne+Saleva+and+Constantine+Lignos%2E+2021%2E+The+effectiveness+of+morphology%2Daware+segmentation+in+low%2Dresource+neural+machine+translation%2E+In+Proceedings+of+the+16th+Conference+of+the+European+Chapter+)
159. Cicero dos Santos and Bianca Zadrozny. 2014. Learning character-level representations for part-of-speech tagging. In Proceedings of the 31st International Conference on Machine Learning (ICML-14), pages 1818–1826. [find](https://scholar.google.com/scholar?q=Cicero+dos+Santos+and+Bianca+Zadrozny%2E+2014%2E+Learning+character%2Dlevel+representations+for+part%2Dof%2Dspeech+tagging%2E+In+Proceedings+of+the+31st+International+Conference+on+Machine+Learning+%28ICML%2D14%29%2C+pag)
160. Jürgen Schmidhuber. 1991. Neural sequence chunkers. [find](https://scholar.google.com/scholar?q=J%C3%BCrgen+Schmidhuber%2E+1991%2E+Neural+sequence+chunkers%2E)
161. Jürgen Schmidhuber. 1992. Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2):234– 242. [find](https://scholar.google.com/scholar?q=J%C3%BCrgen+Schmidhuber%2E+1992%2E+Learning+complex%2C+extended+sequences+using+the+principle+of+history+compression%2E+Neural+Computation%2C+4%282%29%3A234%E2%80%93+242%2E)
162. Patrick Schone and Daniel Jurafsky. 2001. Knowledge-free induction of inflectional morphologies. In Second Meeting of the North American Chapter of the Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/schone-2001-knowledge-free-induction-inflectional-morphologies/)
163. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149– 5152. [find](https://scholar.google.com/scholar?q=Mike+Schuster+and+Kaisuke+Nakajima%2E+2012%2E+Japanese+and+korean+voice+search%2E+In+2012+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2C+pages+5149%E2%80%93+5152%2E)
164. Lane Schwartz, Francis Tyers, Lori Levin, Christo Kirov, Patrick Littell, Chi kiu Lo, Emily Prud’hommeaux, Hyunji Hayley Park, Kenneth Steimel, Rebecca Knowles, Jeffrey Micher, Lonny Strunk, Han Liu, Coleman Haley, Katherine J. Zhang, Robbie Jimmerson, Vasilisa Andriyanets, Aldrian Obaja Muis, Naoki… [find](https://scholar.google.com/scholar?q=Lane+Schwartz%2C+Francis+Tyers%2C+Lori+Levin%2C+Christo+Kirov%2C+Patrick+Littell%2C+Chi+kiu+Lo%2C+Emily+Prud%E2%80%99hommeaux%2C+Hyunji+Hayley+Park%2C+Kenneth+Steimel%2C+Rebecca+Knowles%2C+Jeffrey+Micher%2C+Lonny+Strunk%2C+Han+Liu%2C+)
165. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the ACL (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/)
166. Yan Shao, Christian Hardmeier, and Joakim Nivre. 2018. Universal word segmentation: Implementation and interpretation. [find](https://scholar.google.com/scholar?q=Yan+Shao%2C+Christian+Hardmeier%2C+and+Joakim+Nivre%2E+2018%2E+Universal+word+segmentation%3A+Implementation+and+interpretation%2E)
167. Pamela Shapiro and Kevin Duh. 2018. BPE and charcnns for translation of morphology: A cross-lingual comparison and analysis. CoRR, abs/1809.01301. — [arXiv:1809.01301](https://arxiv.org/abs/1809.01301) [find](https://scholar.google.com/scholar?q=Pamela+Shapiro+and+Kevin+Duh%2E+2018%2E+BPE+and+charcnns+for+translation+of+morphology%3A+A+cross%2Dlingual+comparison+and+analysis%2E+CoRR%2C+abs%2F1809%2E01301%2E+%E2%80%94+%5BarXiv%3A1809%2E01301%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1809%2E01301)
168. Chenglei Si, Zhengyan Zhang, Yingfa Chen, Fanchao Qi, Xiaozhi Wang, Zhiyuan Liu, and Maosong Sun. 2021. ShuoWen-JieZi: Linguistically Informed Tokenizers For Chinese Language Model Pretraining. arXiv, abs/2106.00400. — [arXiv:2106.00400](https://arxiv.org/abs/2106.00400) [find](https://scholar.google.com/scholar?q=Chenglei+Si%2C+Zhengyan+Zhang%2C+Yingfa+Chen%2C+Fanchao+Qi%2C+Xiaozhi+Wang%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2021%2E+ShuoWen%2DJieZi%3A+Linguistically+Informed+Tokenizers+For+Chinese+Language+Model+Pretraining%2E+arXiv%2C)
169. Peter Smit, Sami Virpioja, Stig-Arne Grönroos, and Mikko Kurimo. 2014. Morfessor 2.0: Toolkit for statistical morphological segmentation. In Proceedings of the Demonstrations at the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 21–24, Gothenburg, Swe… [our card](https://md.research.elamur.ai/free/en/papers/smit-2014-morfessor-2-0-toolkit-statistical/)
170. Matthew Snover and Michael Brent. 2002. A probabilistic model for learning concatenative morphology. Advances in Neural Information Processing Systems, 15:1537–1544. [find](https://scholar.google.com/scholar?q=Matthew+Snover+and+Michael+Brent%2E+2002%2E+A+probabilistic+model+for+learning+concatenative+morphology%2E+Advances+in+Neural+Information+Processing+Systems%2C+15%3A1537%E2%80%931544%2E)
171. Matthew G. Snover and Michael R. Brent. 2001. A Bayesian model for morpheme and paradigm identification. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics, pages 490–498, Toulouse, France. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Matthew+G%2E+Snover+and+Michael+R%2E+Brent%2E+2001%2E+A+Bayesian+model+for+morpheme+and+paradigm+identification%2E+In+Proceedings+of+the+39th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pag)
172. Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. 2021. Fast Word-Piece tokenization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2089–2103, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xinying+Song%2C+Alex+Salcianu%2C+Yang+Song%2C+Dave+Dopson%2C+and+Denny+Zhou%2E+2021%2E+Fast+Word%2DPiece+tokenization%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+page)
173. James A. Storer and Thomas G. Szymanski. 1982. Data compression via textual substitution. J. ACM, 29(4):928–951. [find](https://scholar.google.com/scholar?q=James+A%2E+Storer+and+Thomas+G%2E+Szymanski%2E+1982%2E+Data+compression+via+textual+substitution%2E+J%2E+ACM%2C+29%284%29%3A928%E2%80%93951%2E)
174. Baohua Sun, Lin Yang, Catherine Chi, Wenhan Zhang, and Michael Lin. 2019. Squared english word: A method of generating glyph to use super characters for sentiment analysis. In Proceedings of the 2nd Workshop on Affective Content Analysis (AffCon 2019) co-located with the Thirty-Third AAAI Conference… [find](https://scholar.google.com/scholar?q=Baohua+Sun%2C+Lin+Yang%2C+Catherine+Chi%2C+Wenhan+Zhang%2C+and+Michael+Lin%2E+2019%2E+Squared+english+word%3A+A+method+of+generating+glyph+to+use+super+characters+for+sentiment+analysis%2E+In+Proceedings+of+the+2nd+W)
175. Baohua Sun, Lin Yang, Patrick Dong, Wenhan Zhang, Jason Dong, and Charles Young. 2018. Super characters: A conversion from sentiment classification to image classification. In Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 309–… [find](https://scholar.google.com/scholar?q=Baohua+Sun%2C+Lin+Yang%2C+Patrick+Dong%2C+Wenhan+Zhang%2C+Jason+Dong%2C+and+Charles+Young%2E+2018%2E+Super+characters%3A+A+conversion+from+sentiment+classification+to+image+classification%2E+In+Proceedings+of+the+9th+W)
176. Zhiqing Sun and Zhi-Hong Deng. 2018. Unsupervised neural word segmentation for Chinese via segmental language modeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4915– 4920, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zhiqing+Sun+and+Zhi%2DHong+Deng%2E+2018%2E+Unsupervised+neural+word+segmentation+for+Chinese+via+segmental+language+modeling%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+P)
177. Ilya Sutskever, James Martens, and Geoffrey Hinton. 2011. Generating text with recurrent neural networks. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), ICML ’11, pages 1017–1024, New York, NY, USA. ACM. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+James+Martens%2C+and+Geoffrey+Hinton%2E+2011%2E+Generating+text+with+recurrent+neural+networks%2E+In+Proceedings+of+the+28th+International+Conference+on+Machine+Learning+%28ICML%2D11%29%2C+ICML+%E2%80%9911%2C+p)
178. Dan Tito Svenstrup, Jonas Hansen, and Ole Winther. 2017. Hash embeddings for efficient word representations. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Dan+Tito+Svenstrup%2C+Jonas+Hansen%2C+and+Ole+Winther%2E+2017%2E+Hash+embeddings+for+efficient+word+representations%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+volume+30%2E+Curran+Associates%2C+Inc%2E)
179. Samson Tan, Shafiq Joty, Lav Varshney, and Min-Yen Kan. 2020. Mind your inflections! Improving NLP for non-standard Englishes with Base-Inflection Encoding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5647–5663, Online. Association for Com… [find](https://scholar.google.com/scholar?q=Samson+Tan%2C+Shafiq+Joty%2C+Lav+Varshney%2C+and+Min%2DYen+Kan%2E+2020%2E+Mind+your+inflections%21+Improving+NLP+for+non%2Dstandard+Englishes+with+Base%2DInflection+Encoding%2E+In+Proceedings+of+the+2020+Conference+on+Em)
180. Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. 2021. Charformer: Fast character transformers via gradient-based subword tokenization. [our card](https://md.research.elamur.ai/free/en/papers/tay-2021-charformer-fast-character-transformers-gradient/)
181. Yee Whye Teh. 2006. A hierarchical bayesian language model based on Pitman-Yor processes. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL, pages 985–992, Sydney, Australia. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/teh-2006-hierarchical-bayesian-language-model-based/)
182. David Vilar and Marcello Federico. 2021. A statistical extension of byte-pair encoding. In Proceedings of the 18th International Conference on Spoken Language Translation (IWSLT 2021), pages 263–275, Bangkok, Thailand (online). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=David+Vilar+and+Marcello+Federico%2E+2021%2E+A+statistical+extension+of+byte%2Dpair+encoding%2E+In+Proceedings+of+the+18th+International+Conference+on+Spoken+Language+Translation+%28IWSLT+2021%29%2C+pages+263%E2%80%93275%2C+)
183. Sami Virpioja, Ville T. Turunen, Sebastian Spiegler, Oskar Kohonen, and Mikko Kurimo. 2011. Empirical comparison of evaluation methods for unsupervised learning of morphology. Traitement Automatique des Langues, 52(2):45–90. [find](https://scholar.google.com/scholar?q=Sami+Virpioja%2C+Ville+T%2E+Turunen%2C+Sebastian+Spiegler%2C+Oskar+Kohonen%2C+and+Mikko+Kurimo%2E+2011%2E+Empirical+comparison+of+evaluation+methods+for+unsupervised+learning+of+morphology%2E+Traitement+Automatique+d)
184. Changhan Wang, Kyunghyun Cho, and Jiatao Gu. 2019. Neural machine translation with byte-level subwords. [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-neural-machine-translation-byte-level/)
185. Chong Wang, Yining Wang, Po-Sen Huang, Abdelrahman Mohamed, Dengyong Zhou, and Li Deng. 2017. Sequence modeling via segmentations. In International Conference on Machine Learning, pages 3674–3683. [find](https://scholar.google.com/scholar?q=Chong+Wang%2C+Yining+Wang%2C+Po%2DSen+Huang%2C+Abdelrahman+Mohamed%2C+Dengyong+Zhou%2C+and+Li+Deng%2E+2017%2E+Sequence+modeling+via+segmentations%2E+In+International+Conference+on+Machine+Learning%2C+pages+3674%E2%80%933683%2E)
186. Haohan Wang, Peiyan Zhang, and Eric P Xing. 2020a. Word shape matters: Robust machine translation with visual embedding. arXiv preprint arXiv:2010.09997. — [arXiv:2010.09997](https://arxiv.org/abs/2010.09997) [find](https://scholar.google.com/scholar?q=Haohan+Wang%2C+Peiyan+Zhang%2C+and+Eric+P+Xing%2E+2020a%2E+Word+shape+matters%3A+Robust+machine+translation+with+visual+embedding%2E+arXiv+preprint+arXiv%3A2010%2E09997%2E+%E2%80%94+%5BarXiv%3A2010%2E09997%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F201)
187. Lihao Wang, Zongyi Li, and Xiaoqing Zheng. 2021a. Unsupervised word segmentation with bi-directional neural language model. CoRR, abs/2103.01421. — [arXiv:2103.01421](https://arxiv.org/abs/2103.01421) [find](https://scholar.google.com/scholar?q=Lihao+Wang%2C+Zongyi+Li%2C+and+Xiaoqing+Zheng%2E+2021a%2E+Unsupervised+word+segmentation+with+bi%2Ddirectional+neural+language+model%2E+CoRR%2C+abs%2F2103%2E01421%2E+%E2%80%94+%5BarXiv%3A2103%2E01421%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2103%2E01421%29)
188. Xinyi Wang, Sebastian Ruder, and Graham Neubig. 2021b. Multi-view subword regularization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 473–482, Online. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Xinyi+Wang%2C+Sebastian+Ruder%2C+and+Graham+Neubig%2E+2021b%2E+Multi%2Dview+subword+regularization%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Lingu)
189. Zihan Wang, Karthikeyan K, Stephen Mayhew, and Dan Roth. 2020b. Extending multilingual BERT to low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2649–2656, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zihan+Wang%2C+Karthikeyan+K%2C+Stephen+Mayhew%2C+and+Dan+Roth%2E+2020b%2E+Extending+multilingual+BERT+to+low%2Dresource+languages%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2020%2C+pages+2)
190. Jonathan J. Webster and Chunyu Kit. 1992. Tokenization as the initial phase in NLP. In COL-ING 1992 Volume 4: The 14th International Conference on Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/webster-1992-tokenization-as-initial-phase-nlp/)
191. J Gerard Wolff. 1975. An algorithm for the segmentation of an artificial language analogue. British Journal of PsychologyJ, 66:79–90. [find](https://scholar.google.com/scholar?q=J+Gerard+Wolff%2E+1975%2E+An+algorithm+for+the+segmentation+of+an+artificial+language+analogue%2E+British+Journal+of+PsychologyJ%2C+66%3A79%E2%80%9390%2E)
192. F. Wood, J. Gasthaus, C. Archambeau, L. James, and Y.W. Teh. 2011. The sequence memoizer. Communications of the ACM, 54(2):91–98. [find](https://scholar.google.com/scholar?q=F%2E+Wood%2C+J%2E+Gasthaus%2C+C%2E+Archambeau%2C+L%2E+James%2C+and+Y%2EW%2E+Teh%2E+2011%2E+The+sequence+memoizer%2E+Communications+of+the+ACM%2C+54%282%29%3A91%E2%80%9398%2E)
193. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833–844,… [our card](https://md.research.elamur.ai/free/en/papers/wu-2019-beto-bentz-becas-surprising-cross/)
194. Jingjing Xu, Hao Zhou, Chun Gan, Zaixiang Zheng, and Lei Li. 2021. Vocabulary learning via optimal transport for neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Proc… [find](https://scholar.google.com/scholar?q=Jingjing+Xu%2C+Hao+Zhou%2C+Chun+Gan%2C+Zaixiang+Zheng%2C+and+Lei+Li%2E+2021%2E+Vocabulary+learning+via+optimal+transport+for+neural+machine+translation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Associatio)
195. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2021. Byt5: Towards a token-free future with pre-trained byte-to-byte models. [find](https://scholar.google.com/scholar?q=Linting+Xue%2C+Aditya+Barua%2C+Noah+Constant%2C+Rami+Al%2DRfou%2C+Sharan+Narang%2C+Mihir+Kale%2C+Adam+Roberts%2C+and+Colin+Raffel%2E+2021%2E+Byt5%3A+Towards+a+token%2Dfree+future+with+pre%2Dtrained+byte%2Dto%2Dbyte+models%2E)
196. David Yarowsky and Richard Wicentowski. 2000. Minimally supervised morphological analysis by multimodal alignment. In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, pages 207–216, Hong Kong. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=David+Yarowsky+and+Richard+Wicentowski%2E+2000%2E+Minimally+supervised+morphological+analysis+by+multimodal+alignment%2E+In+Proceedings+of+the+38th+Annual+Meeting+of+the+Association+for+Computational+Lingui)
197. Xinsong Zhang, Pengshuai Li, and Hang Li. 2021. AMBERT: A pre-trained language model with multi-grained tokenization. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 421–435, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xinsong+Zhang%2C+Pengshuai+Li%2C+and+Hang+Li%2E+2021%2E+AMBERT%3A+A+pre%2Dtrained+language+model+with+multi%2Dgrained+tokenization%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL%2DIJCNLP+2021%2C+pag)
198. Giulio Zhou. 2018. Morphological zero-shot neural machine translation. Master’s thesis, University of Edinburgh. [find](https://scholar.google.com/scholar?q=Giulio+Zhou%2E+2018%2E+Morphological+zero%2Dshot+neural+machine+translation%2E+Master%E2%80%99s+thesis%2C+University+of+Edinburgh%2E)