---
title: "Playing with Words at the National Library — Malmsten 2020"
description: "The authors, working at KBLab of the National Library of Sweden (KB), set out to build a Swedish-specific BERT, because multilingual models tend to…"
authors: ["Martin Malmsten", "Love Borjeson", "Chris Haffenden"]
year: 2020
venue: "arXiv:2007.01658"
source: "https://arxiv.org/abs/2007.01658"
doi: "10.48550/arxiv.2007.01658"
arxiv: "2007.01658"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/malmsten-2020-playing-words-national-library-sweden/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/malmsten-2020-playing-words-national-library-sweden/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2020](https://md.research.elamur.ai/free/papers/2020/page/3/) / Playing with Words at the National Library of Sweden -- Making a Swedish BERT

# Playing with Words at the National Library of Sweden -- Making a Swedish BERT

**Authors:** Martin Malmsten, Love Borjeson, Chris Haffenden · **Year:** 2020 · **Venue:** arXiv:2007.01658 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2007.01658) · [PDF](https://arxiv.org/pdf/2007.01658) · [DOI 10.48550/arxiv.2007.01658](https://doi.org/10.48550/arxiv.2007.01658) · [arXiv 2007.01658](https://arxiv.org/abs/2007.01658) · [OpenAlex](https://openalex.org/W3040245432)

**Citations:** 26 citations (openalex-snapshot · 2026-10-14) · [79](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
The authors, working at KBLab of the National Library of Sweden (KB), set out to build a Swedish-specific BERT, because multilingual models tend to underperform on smaller languages and suitable training data is scarce. They assembled a corpus of modern Swedish (1940s to late 2019) from KB's holdings: OCR'd newspapers, government reports (SOU), legal e-deposits, forum comments and Swedish Wikipedia. Before pre-training they cleaned the text heavily. They repaired systematic OCR errors using a lexicon, rebuilt sentences and paragraphs with spaCy plus custom scripts, and kept emojis as meaningful tokens. They then built a SentencePiece vocabulary and pre-trained the model with Google's original BERT code. On NER and POS tagging with the Stockholm-Umeå Corpus, the resulting KB-BERT beat both Google's M-BERT and an earlier Swedish model from Arbetsförmedlingen. The NER gains were clear, while the POS gains were marginal. The model is publicly released. The work shows that a national library's archives can supply a lower-resourced language with a strong base model, and that diverse registers in the corpus matter.

## Key points
- The cleaned corpus has about 260M sentences, roughly 3,497M words and about 18.3 GB of text. Newspapers (allnews.txt, about 3.0B words) dominate, followed by SOU, e-deposit, social and sw-wiki. Fiction is underrepresented.
- The OCR repair targeted m ↔ rn confusions, such as "om" being turned into "örn" and "-arna" into "-ama". Candidates were checked against Saldo, the morphological resource from Språkbanken. Over 30,000 mappings were found, and about 13 million occurrences were corrected.
- The tokenizer is SentencePiece with a vocabulary of about 50k, chosen because Swedish has many compounds. Emojis and skin-tone modifiers were added to it by hand.
- Pre-training ran in three stages:
  - 1M steps with sequence length 128 and batch size 512;
  - 100k steps with sequence length 512 and batch size 128;
  - roughly 1M more steps at 128/512.
- Compute came partly from in-house hardware and partly from Google TFRC Cloud TPUs, which were used only for non-copyrighted material.
- On NER (SUC 3.0, split 70/20/10), average FB1 was 0.927 for KB-BERT, 0.906 for M-BERT and 0.876 for AF-AI. HFST-SweNER is shown in the results but is not directly comparable. For POS, KB-BERT improved on M-BERT by less than 1%. NER showed little gain between 350k and 1M pre-training steps.

## Interesting ideas
- **Lexicon-checked OCR repair.** For a known confusion pair, generate the swapped spelling of each word. Add the swap to a correction table only when the original is missing from a morphological lexicon and the variant exists. This is cheap and targeted, and it could fit OCR'd Ukrainian or other Cyrillic archives with their own typical character confusions.
- **A minimum paragraph length as a noise filter.** Fragments shorter than about ten sentences were discarded, along with broken sentences at paragraph edges. This removes context-poor OCR debris before pre-training.
- **Emojis as content and punctuation.** Emojis were kept in the vocabulary and allowed to end sentences where they act as end punctuation. This is useful for any model expected to handle social-media text.
- **Diversity over sheer volume.** The authors argue that adding small corpora of new registers (forums, official prose) helps robustness more than adding more of an existing genre. They report that messy social-media data seemed to help beyond that domain. This is a sensible principle when assembling corpora for any lower-resourced language.
- **Splitting compute by copyright status.** Copyrighted material was trained in-house, while freely usable text went to external cloud compute. This is a practical pattern for institutions holding rights-restricted collections.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Named entity recognition](https://md.research.elamur.ai/free/topics/named-entity-recognition/)

## Cited by

Cited by

79 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 76 not yet.

- [ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic](https://md.research.elamur.ai/free/papers/abdul-2021-mageed-arbert-marbert-deep-bidirectional-transformers/)
- [Bridging the Resource Gap: Exploring the Efficacy of English and Multilingual LLMs for Swedish](https://md.research.elamur.ai/free/papers/holmstrom-2023-bridging-resource-gap-exploring-efficacy/)
- [BRAGD: Constrained Multi-Label POS Tagging for Faroese](https://md.research.elamur.ai/free/papers/simonsen-2026-bragd-constrained-multi-label-pos/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{malmsten2020playing,
  title = {{Playing with Words at the National Library of Sweden -- Making a Swedish BERT}},
  author = {Malmsten, Martin and Borjeson, Love and Haffenden, Chris},
  year = {2020},
  doi = {10.48550/arxiv.2007.01658},
  eprint = {2007.01658},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2007.01658}
}
```

- [← previous PhoBERT: Pre-trained language models for Vietnamese](https://md.research.elamur.ai/free/papers/nguyen-2020-phobert-pre-trained-language-models/)
- [next → Politeness Transfer: A Tag and Generate Approach](https://md.research.elamur.ai/free/papers/madaan-2020-politeness-transfer-tag-generate-approach/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/papers/2020/page/3/)

## References

15 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ahearn, L. M. (2017). Living Language: An Introduction to Linguistic Anthropology. John Wiley & Sons, Chichester, second edition edition. [find](https://scholar.google.com/scholar?q=Ahearn%2C+L%2E+M%2E+%282017%29%2E+Living+Language%3A+An+Introduction+to+Linguistic+Anthropology%2E+John+Wiley+%26+Sons%2C+Chichester%2C+second+edition+edition%2E)
2. Anderson, B. (2016). Imagined Communities: Reflections on the Origin and Spread of Nationalism. Verso, London, revised edition edition. [find](https://scholar.google.com/scholar?q=Anderson%2C+B%2E+%282016%29%2E+Imagined+Communities%3A+Reflections+on+the+Origin+and+Spread+of+Nationalism%2E+Verso%2C+London%2C+revised+edition+edition%2E)
3. Borin, L., Forsberg, M., and Lonngren, L. (2013). SALDO: a touch of yin to WordNet’s yang. Language Resources and Evaluation, 47(4):1191–1211, December. Place: Dordrecht Publisher: Springer WOS:000328213000011. [↗](https://doi.org/10.1007/s10579-013-9233-4)
4. Branden, C., Möller, T., Pietsch, M., Soni, T., and Yeung, C. M. (2019). Open Sourcing German BERT: Insights into pre-training BERT from scratch. [find](https://scholar.google.com/scholar?q=Branden%2C+C%2E%2C+Mo%CC%88ller%2C+T%2E%2C+Pietsch%2C+M%2E%2C+Soni%2C+T%2E%2C+and+Yeung%2C+C%2E+M%2E+%282019%29%2E+Open+Sourcing+German+BERT%3A+Insights+into+pre%2Dtraining+BERT+from+scratch%2E)
5. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs], May. arXiv: 1810.04805. [↗](https://arxiv.org/abs/1810.04805)
6. Honnibal, M. and Montani, I. (2017). spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear. [find](https://scholar.google.com/scholar?q=Honnibal%2C+M%2E+and+Montani%2C+I%2E+%282017%29%2E+spaCy+2%3A+Natural+language+understanding+with+Bloom+embeddings%2C+convolutional+neural+networks+and+incremental+parsing%2E+To+appear%2E)
7. Kokkinakis, D., Niemi, J., Hardwick, S., Linden, K., and Borin, L. (2014). HFST-SweNER - A New NER Resource for Swedish. European Language Resources Assoc-Elra, Paris. Pages: 2537-2543 Publication Title: Lrec 2014 - Ninth International Conference on Language Resources and Evaluation WOS:000355611004025. [find](https://scholar.google.com/scholar?q=Kokkinakis%2C+D%2E%2C+Niemi%2C+J%2E%2C+Hardwick%2C+S%2E%2C+Linden%2C+K%2E%2C+and+Borin%2C+L%2E+%282014%29%2E+HFST%2DSweNER+%2D+A+New+NER+Resource+for+Swedish%2E+European+Language+Resources+Assoc%2DElra%2C+Paris%2E+Pages%3A+2537%2D2543+Publication+Tit)
8. Kudo, T. and Richardson, J. (2018). Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. CoRR, abs/1808.06226. [↗](https://arxiv.org/abs/1808.06226) [our card](https://md.research.elamur.ai/free/papers/kudo-2018-sentencepiece-simple-language-independent-subword/)
9. Linder, L., Jungo, M., Hennebert, J., Musat, C., and Fischer, A. (2020). Automatic Creation of Text Corpora for Low-Resource Languages from the Internet: The Case of Swiss German. arXiv:1912.00159 [cs], June. arXiv: 1912.00159. [↗](https://arxiv.org/abs/1912.00159)
10. Martin, L., Muller, B., Suárez, P. J. O., Dupont, Y., Romary, L., de la Clergerie, E. V., Seddah, D., and Sagot, B. (2020). CamemBERT: a Tasty French Language Model. arXiv:1911.03894 [cs], May. arXiv: 1911.03894. [↗](https://arxiv.org/abs/1911.03894) [our card](https://md.research.elamur.ai/free/papers/martin-2019-camembert-tasty-french-language-model/)
11. Pires, T., Schlinger, E., and Garrette, D. (2019). How multilingual is Multilingual BERT? arXiv:1906.01502 [cs], June. arXiv: 1906.01502. [↗](https://arxiv.org/abs/1906.01502) [our card](https://md.research.elamur.ai/free/papers/pires-2019-how-multilingual-is-multilingual-bert/)
12. Rönnqvist, S., Kanerva, J., Salakoski, T., and Ginter, F. (2019). Is Multilingual BERT Fluent in Language Generation? arXiv:1910.03806 [cs], October. arXiv: 1910.03806. [↗](https://arxiv.org/abs/1910.03806)
13. Virtanen, A., Kanerva, J., Ilo, R., Luoma, J., Luotolahti, J., Salakoski, T., Ginter, F., and Pyysalo, S. (2019). Multilingual is not enough: BERT for Finnish. arXiv:1912.07076 [cs], December. arXiv: 1912.07076. [↗](https://arxiv.org/abs/1912.07076) [our card](https://md.research.elamur.ai/free/papers/virtanen-2019-multilingual-enough-bert-finnish/)
14. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, J. (2019). Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. [↗](https://arxiv.org/abs/1910.03771) [our card](https://md.research.elamur.ai/free/papers/wolf-2020-transformers-state-art-natural-language/)
15. Wu, S. and Dredze, M. (2019). Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT. arXiv:1904.09077 [cs], October. arXiv: 1904.09077. [↗](https://arxiv.org/abs/1904.09077) [our card](https://md.research.elamur.ai/free/papers/wu-2019-beto-bentz-becas-surprising-cross/)