---
title: "BPEmb: Tokenization-free Pre-trained — Heinzerling 2018"
description: "Token-level embeddings handle rare and unseen words poorly."
authors: ["Benjamin Heinzerling", "Michael Strube"]
year: 2018
venue: "Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)"
source: "https://aclanthology.org/L18-1473/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/heinzerling-2018-bpemb-tokenization-free-pre-trained/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/heinzerling-2018-bpemb-tokenization-free-pre-trained/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2018](https://md.research.elamur.ai/free/papers/2018/) / BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages

# BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages

**Authors:** [Benjamin Heinzerling](https://md.research.elamur.ai/free/authors/benjamin-heinzerling/), [Michael Strube](https://md.research.elamur.ai/free/authors/michael-strube/) · **Year:** 2018 · **Venue:** Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/L18-1473/) · [PDF](https://aclanthology.org/L18-1473.pdf) · [OpenAlex](https://openalex.org/W2962685689)

**Citations:** 181 citations (openalex-snapshot · 2026-10-14) · [75](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Token-level embeddings handle rare and unseen words poorly. Subword methods address this, but BPE unit embeddings were usually trained for a single task and never released. The authors fill this gap with BPEmb. They ran the SentencePiece BPE implementation over every sufficiently large Wikipedia (lowercased, digits mapped to zero) and trained GloVe vectors for the resulting symbols. The release covers 275 languages and a grid of vocabulary sizes and dimensions. Because BPE works directly on raw symbol streams, no tokenizer is needed, which helps with scripts such as Chinese or Japanese.

To test usefulness, the authors used fine-grained entity typing of isolated mentions. They compared BPEmb with FastText, character embeddings and token embeddings, using three encoders and extensive hyper-parameter search. On English, BPEmb was the strongest subword option and matched FastText while being only a tiny fraction of its size. Across other languages the two were mostly on par, with a few gains and one large loss (Khmer). The resource makes compact multilingual subword vectors easy to obtain, including for low-resource and memory-limited settings.

## Key points
- BPEmb offers embeddings for 1000 to 200000 BPE merge operations (1k, 3k, 5k, 10k, 25k, 50k, 100k, 200k) and dimensions 25, 50, 100, 200 and 300. All are trained with GloVe on Wikipedia text extracted with WikiExtractor.
- The test task assigns mentions to the 89 types of Gillick et al. (2014). Mentions come from Wikidata, and types come from mapping to Freebase, giving 3.4 million English instances with random 80,000/20,000 train/test samples. The encoders are averaging, RNN (LSTM) and CNN, tuned with a Tree-structured Parzen Estimator (≥1000 trials for English, ≥50 for others).
- On English, BPE-RNN reached a mean of 0.624 ± 0.029 (max 0.65), against 0.617 ± 0.007 (max 0.63) for FastText-RNN. BPEmb matched FastText with about 11 MB (100k symbols, dim 25) versus roughly 6 GB (3 million vectors, dim 300).
- Multilingual scores (FastText vs BPEmb):
  - English rose from 62.9 to 65.4.
  - Chinese rose from 71.0 to 72.0, and Japanese fell from 62.3 to 61.4, with FastText using pre-tokenized input (Stanford CoreNLP for Chinese, Kuromoji for Japanese).
  - Tibetan gained 3.5 and Lao gained 2.1.
  - Khmer dropped 8.9, which the authors attribute to inconsistent Unicode control-character handling across libraries.
- Other findings:
  - Averaging was the weakest encoder.
  - RNNs slightly beat CNNs but trained much more slowly.
  - FastText showed the lowest variance across settings.
  - The evaluation ignores context and covers only a few of the 275 languages.

## Interesting ideas
- **Segmentation without a tokenizer.** Running BPE on raw text sidesteps tokenizer quality for unsegmented scripts. This could suit mixed-script, noisy or poorly tooled languages, where a weak tokenizer would otherwise become the bottleneck.
- **A grid of released variants.** Publishing embeddings for many vocabulary sizes and dimensions lets users pick their own trade-off between size and quality. Other resource releases could do the same instead of shipping one "best" configuration.
- **Score distributions instead of a single best run.** Showing results across hundreds of hyper-parameter trials makes robustness visible. Here it showed that FastText is forgiving, while BPEmb needs careful tuning, which a single top score would hide.
- **Mention-only entity typing as a probe.** Typing a bare name like "Melfordshire" isolates what subword units carry about meaning. The data can be built automatically from a knowledge base, giving a cheap benchmark that scales to many languages.
- **Small vocabulary and low dimension as a compact baseline.** The paper shows that a small subword vocabulary with low-dimensional vectors can rival multi-gigabyte embeddings. This makes it worth trying such a setup first for on-device or resource-constrained applications.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

## Cited by

Cited by

75 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 70 not yet.

- [FLAIR: An Easy-to-Use Framework for State-of-the-Art NLP](https://md.research.elamur.ai/free/papers/akbik-2019-flair-easy-use-framework-state/)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://md.research.elamur.ai/free/papers/belrose-2023-eliciting-latent-predictions-transformers-tuned/)
- [Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future](https://md.research.elamur.ai/free/papers/cl-2023-klie-annotation-error-detection-analyzing-past/)
- [A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios](https://md.research.elamur.ai/free/papers/hedderich-2021-survey-recent-approaches-natural-language/)
- [Are All Languages Created Equal in Multilingual BERT?](https://md.research.elamur.ai/free/papers/wu-2020-are-all-languages-created-equal/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{heinzerling2018bpemb,
  title = {{BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages}},
  author = {Heinzerling, Benjamin and Strube, Michael},
  booktitle = {Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)},
  year = {2018},
  doi = {10.63317/3eqtu58tsqay},
  url = {https://doi.org/10.63317/3eqtu58tsqay}
}
```

- [← previous Bottom-Up Abstractive Summarization](https://md.research.elamur.ai/free/papers/gehrmann-2018-bottom-up-abstractive-summarization/)
- [next → Breaking NLI Systems with Sentences that Require Simple Lexical Inferences](https://md.research.elamur.ai/free/papers/glockner-2018-breaking-nli-systems-sentences-that/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/papers/2018/)

## References

17 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bergstra, J. S., Bardenet, R., Bengio, Y., and Kégl, B. (2011). Algorithms for hyper-parameter optimization. In Advances in Neural Information Processing Systems, pages 2546–2554. [find](https://scholar.google.com/scholar?q=Bergstra%2C+J%2E+S%2E%2C+Bardenet%2C+R%2E%2C+Bengio%2C+Y%2E%2C+and+Ke%CC%81gl%2C+B%2E+%282011%29%2E+Algorithms+for+hyper%2Dparameter+optimization%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+2546%E2%80%932554%2E)
2. Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. (2017). Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. [find](https://scholar.google.com/scholar?q=Bojanowski%2C+P%2E%2C+Grave%2C+E%2E%2C+Joulin%2C+A%2E%2C+and+Mikolov%2C+T%2E+%282017%29%2E+Enriching+word+vectors+with+subword+information%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+5%3A135%E2%80%93146%2E)
3. Bollacker, K., Evans, C., Paritosh, P., Sturge, T., and Taylor, J. (2008). Freebase: A collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, Vancouver, B.C., Canada, 10–12 June 2008, pages 1247–12… [find](https://scholar.google.com/scholar?q=Bollacker%2C+K%2E%2C+Evans%2C+C%2E%2C+Paritosh%2C+P%2E%2C+Sturge%2C+T%2E%2C+and+Taylor%2C+J%2E+%282008%29%2E+Freebase%3A+A+collaboratively+created+graph+database+for+structuring+human+knowledge%2E+In+Proceedings+of+the+2008+ACM+SIGMOD+Int)
4. Chiu, J. and Nichols, E. (2016). Named entity recognition with bidirectional LSTM-CNNs. Transactions of the Association for Computational Linguistics, 4:357–370. [find](https://scholar.google.com/scholar?q=Chiu%2C+J%2E+and+Nichols%2C+E%2E+%282016%29%2E+Named+entity+recognition+with+bidirectional+LSTM%2DCNNs%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+4%3A357%E2%80%93370%2E) [our card](https://md.research.elamur.ai/free/papers/chiu-2015-named-entity-recognition-bidirectional-lstm/)
5. Clark, K. and Manning, C. D. (2016). Improving coreference resolution by learning entity-level distributed representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, 7–12 August 2016. [find](https://scholar.google.com/scholar?q=Clark%2C+K%2E+and+Manning%2C+C%2E+D%2E+%282016%29%2E+Improving+coreference+resolution+by+learning+entity%2Dlevel+distributed+representations%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computation) [our card](https://md.research.elamur.ai/free/papers/clark-2016-improving-coreference-resolution-learning-entity/)
6. Gillick, D., Lazic, N., Ganchev, K., Kirchner, J., and Huynh, D. (2014). Context-Dependent Fine-Grained Entity Type Tagging. ArXiv e-prints, December. [find](https://scholar.google.com/scholar?q=Gillick%2C+D%2E%2C+Lazic%2C+N%2E%2C+Ganchev%2C+K%2E%2C+Kirchner%2C+J%2E%2C+and+Huynh%2C+D%2E+%282014%29%2E+Context%2DDependent+Fine%2DGrained+Entity+Type+Tagging%2E+ArXiv+e%2Dprints%2C+December%2E)
7. Ling, X. and Weld, D. S. (2012). Fine-grained entity recognition. In AAAI. [find](https://scholar.google.com/scholar?q=Ling%2C+X%2E+and+Weld%2C+D%2E+S%2E+%282012%29%2E+Fine%2Dgrained+entity+recognition%2E+In+AAAI%2E)
8. Luong, M.-T. and Manning, C. D. (2016). Achieving open vocabulary neural machine translation with hybrid word-character models. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1054–1063, Berlin, Germany, August. Association fo… [find](https://scholar.google.com/scholar?q=Luong%2C+M%2E%2DT%2E+and+Manning%2C+C%2E+D%2E+%282016%29%2E+Achieving+open+vocabulary+neural+machine+translation+with+hybrid+word%2Dcharacter+models%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computa) [our card](https://md.research.elamur.ai/free/papers/luong-2016-achieving-open-vocabulary-neural-machine/)
9. Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S. J., and McClosky, D. (2014). The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60. Association for Computat… [find](https://scholar.google.com/scholar?q=Manning%2C+C%2E+D%2E%2C+Surdeanu%2C+M%2E%2C+Bauer%2C+J%2E%2C+Finkel%2C+J%2E%2C+Bethard%2C+S%2E+J%2E%2C+and+McClosky%2C+D%2E+%282014%29%2E+The+Stanford+CoreNLP+natural+language+processing+toolkit%2E+In+Proceedings+of+52nd+Annual+Meeting+of+the+Ass) [our card](https://md.research.elamur.ai/free/papers/manning-2014-stanford-corenlp-natural-language-processing/)
10. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. In Proceedings of the ICLR 2013 Workshop Track. [find](https://scholar.google.com/scholar?q=Mikolov%2C+T%2E%2C+Chen%2C+K%2E%2C+Corrado%2C+G%2E%2C+and+Dean%2C+J%2E+%282013%29%2E+Efficient+estimation+of+word+representations+in+vector+space%2E+In+Proceedings+of+the+ICLR+2013+Workshop+Track%2E)
11. Pennington, J., Socher, R., and Manning, C. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532– 1543, Doha, Qatar, October. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Pennington%2C+J%2E%2C+Socher%2C+R%2E%2C+and+Manning%2C+C%2E+%282014%29%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+Proceedings+of+the+2014+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+pa)
12. Reimers, N. and Gurevych, I. (2017). Reporting score distributions makes a difference: Performance study of lstm-networks for sequence tagging. CoRR, abs/1707.09861. [find](https://scholar.google.com/scholar?q=Reimers%2C+N%2E+and+Gurevych%2C+I%2E+%282017%29%2E+Reporting+score+distributions+makes+a+difference%3A+Performance+study+of+lstm%2Dnetworks+for+sequence+tagging%2E+CoRR%2C+abs%2F1707%2E09861%2E) [our card](https://md.research.elamur.ai/free/papers/reimers-2017-reporting-score-distributions-makes-difference/)
13. Schütze, H. (2017). Nonsymbolic text representation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 785–796, Valencia, Spain, April. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Schu%CC%88tze%2C+H%2E+%282017%29%2E+Nonsymbolic+text+representation%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+the+Association+for+Computational+Linguistics%3A+Volume+1%2C+Long+Papers%2C+pages+785%E2%80%937)
14. Sennrich, R., Haddow, B., and Birch, A. (2016). Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany, August. Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Sennrich%2C+R%2E%2C+Haddow%2C+B%2E%2C+and+Birch%2C+A%2E+%282016%29%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+)
15. Shimaoka, S., Stenetorp, P., Inui, K., and Riedel, S. (2017). Neural architectures for fine-grained entity type classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1271–1280, Valencia, Spain, Apr… [find](https://scholar.google.com/scholar?q=Shimaoka%2C+S%2E%2C+Stenetorp%2C+P%2E%2C+Inui%2C+K%2E%2C+and+Riedel%2C+S%2E+%282017%29%2E+Neural+architectures+for+fine%2Dgrained+entity+type+classification%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+the+Ass)
16. Vrandečić, D. and Krötzsch, M. (2014). Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10):78–85. [find](https://scholar.google.com/scholar?q=Vrandec%CC%8Cic%CC%81%2C+D%2E+and+Kro%CC%88tzsch%2C+M%2E+%282014%29%2E+Wikidata%3A+a+free+collaborative+knowledgebase%2E+Communications+of+the+ACM%2C+57%2810%29%3A78%E2%80%9385%2E)
17. Yaghoobzadeh, Y. and Schütze, H. (2017). Multi-level representations for fine-grained typing of knowledge base entities. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 578–589, Valencia, Spain, April. Asso… [find](https://scholar.google.com/scholar?q=Yaghoobzadeh%2C+Y%2E+and+Schu%CC%88tze%2C+H%2E+%282017%29%2E+Multi%2Dlevel+representations+for+fine%2Dgrained+typing+of+knowledge+base+entities%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+the+Associati)