---
title: "Tuning Multilingual Transformers for — Arkhipov 2019"
description: "The paper addresses named entity recognition (NER) in four Slavic languages: Bulgarian, Czech, Polish and Russian."
authors: ["Mikhail Arkhipov", "Maria Trofimova", "Yuri Kuratov", "Alexey Sorokin"]
year: 2019
venue: "Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing"
source: "https://aclanthology.org/W19-3712/"
doi: "10.18653/v1/w19-3712"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/arkhipov-2019-tuning-multilingual-transformers-language-specific/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/arkhipov-2019-tuning-multilingual-transformers-language-specific/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2019](https://md.research.elamur.ai/free/papers/2019/page/4/) / Tuning Multilingual Transformers for Language-Specific Named Entity Recognition

# Tuning Multilingual Transformers for Language-Specific Named Entity Recognition

**Authors:** Mikhail Arkhipov, Maria Trofimova, Yuri Kuratov, Alexey Sorokin · **Year:** 2019 · **Venue:** Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/W19-3712/) · [PDF](https://aclanthology.org/W19-3712.pdf) · [DOI 10.18653/v1/w19-3712](https://doi.org/10.18653/v1/w19-3712) · [OpenAlex](https://openalex.org/W2973071945)

**Citations:** 114 citations (openalex-snapshot · 2026-10-14) · [49](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary

The paper addresses named entity recognition (NER) in four Slavic languages: Bulgarian, Czech, Polish and Russian. It was carried out as part of the BSNLP 2019 shared task. The authors started from multilingual BERT and continued its unsupervised training on Wikipedia for Czech, Polish and Bulgarian and on news for Russian. The central idea is to avoid training a model from scratch, which is very expensive. They built a new subword vocabulary with subword-nmt. They computed each new subword's vector as the average of the vectors of the multilingual subwords that make it up. All transformer weights were taken from the multilingual model. On top of the resulting Slavic BERT, a word-level CRF replaces the usual dense layer. This system clearly outperforms both a Bi-LSTM-CRF on FastText and multilingual BERT. The CRF improves the results of both BERT variants. In the competition, the model ranked first on three of the four metrics. The work shows a relatively cheap way to obtain a strong model for a group of related languages. The model and code are open, so the approach can be transferred to other NLP tasks.

## Key points

- **Data:** the BSNLP 2019 corpus consists of web news with five entity types (PER, LOC, ORG, EVT, PRO). Documents about Brexit were used for training, and those about Asia Bibi for validation. Russian makes up only 7.7% of the corpus, and Bulgarian 39%.
- **Further pre-training of Slavic BERT:** it took 9 days on a DGX-1 with eight P100 GPUs. First, the whole model was trained for 1M iterations on sequences of length 128. Then only the positional embeddings were trained for 300k iterations on length 512. Batch size was 256 and the learning rate was 2e-5.
- **NER fine-tuning:** the top layers and the BERT layers had different learning rates (3e-4 and 1e-5). The model was evaluated on the validation set every 10 batches, and an exponential moving average was applied to the parameters.
- **Results:** Slavic BERT-CRF has an error rate more than a third lower than the baseline multilingual BERT. The system ranked first on the REM and SM metrics and on span F1 (CoNLL-2003); it was not first only on RPM. Pre-training on other NER corpora (including BulTreeBank and BSNLP 2017) helped only the models without a CRF.
- **Language differences:** Polish and Czech (Latin script) achieve better results than Russian and Bulgarian (Cyrillic). Recall is especially weak for Bulgarian. The authors give two reasons for this: Bulgarian text had to be split into sentences with the English NLTK tokenizer, and the two Cyrillic languages are less closely related to each other.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Named entity recognition](https://md.research.elamur.ai/free/topics/named-entity-recognition/)

## Cited by

Cited by

49 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 42 not yet.

- [The ADAPT Enhanced Dependency Parser at the IWPT 2020 Shared Task](https://md.research.elamur.ai/free/papers/barry-2020-adapt-enhanced-dependency-parser-iwpt/)
- [On the Role of Morphological Information for Contextual Lemmatization](https://md.research.elamur.ai/free/papers/cl-2024-toporkov-role-morphological-information-contextual-lemmatization/)
- [Applying Occam's Razor to Transformer-Based Dependency Parsing: What Works, What Doesn't, and What is Really Necessary](https://md.research.elamur.ai/free/papers/grunewald-2021-applying-occam-s-razor-transformer/)
- [Bridging the Resource Gap: Exploring the Efficacy of English and Multilingual LLMs for Swedish](https://md.research.elamur.ai/free/papers/holmstrom-2023-bridging-resource-gap-exploring-efficacy/)
- [Multilingual Coreference Resolution with Harmonized Annotations](https://md.research.elamur.ai/free/papers/prazak-2021-multilingual-coreference-resolution-harmonized-annotations/)
- [mGPT: Few-Shot Learners Go Multilingual](https://md.research.elamur.ai/free/papers/shliazhko-2022-mgpt-few-shot-learners-go/)
- [Hybrid Enhanced Universal Dependencies Parsing](https://md.research.elamur.ai/free/papers/ud-2020-heinecke-hybrid-enhanced-universal-dependencies-parsing/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{arkhipov2019tuning,
  title = {{Tuning Multilingual Transformers for Language-Specific Named Entity Recognition}},
  author = {Arkhipov, Mikhail and Trofimova, Maria and Kuratov, Yuri and Sorokin, Alexey},
  booktitle = {Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/w19-3712},
  url = {https://doi.org/10.18653/v1/w19-3712}
}
```

- [← previous TuckER: Tensor Factorization for Knowledge Graph Completion](https://md.research.elamur.ai/free/papers/balazevic-2019-tucker-tensor-factorization-knowledge-graph/)
- [next → Tweet Stance Detection Using an Attention based Neural Ensemble Model](https://md.research.elamur.ai/free/papers/siddiqua-2019-tweet-stance-detection-attention-based/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/papers/2019/page/4/)

## References

18 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606. [find](https://scholar.google.com/scholar?q=Piotr+Bojanowski%2C+Edouard+Grave%2C+Armand+Joulin%2C+and+Tomas+Mikolov%2E+2016%2E+Enriching+word+vectors+with+subword+information%2E+arXiv+preprint+arXiv%3A1607%2E04606%2E)
2. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of machine learning research, 12(Aug):2493–2537. [find](https://scholar.google.com/scholar?q=Ronan+Collobert%2C+Jason+Weston%2C+L%C3%A9on+Bottou%2C+Michael+Karlen%2C+Koray+Kavukcuoglu%2C+and+Pavel+Kuksa%2E+2011%2E+Natural+language+processing+%28almost%29+from+scratch%2E+Journal+of+machine+learning+research%2C+12%28Aug%29%3A2) [our card](https://md.research.elamur.ai/free/papers/collobert-2011-natural-language-processing-almost-scratch/)
3. Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In Advances in neural information processing systems, pages 3079–3087. [find](https://scholar.google.com/scholar?q=Andrew+M+Dai+and+Quoc+V+Le%2E+2015%2E+Semi%2Dsupervised+sequence+learning%2E+In+Advances+in+neural+information+processing+systems%2C+pages+3079%E2%80%933087%2E)
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2E)
5. Georgi Georgiev, Preslav Nakov, Kuzman Ganchev, Petya Osenova, and Kiril Ivanov Simov. 2009. Feature-rich named entity recognition for Bulgarian using conditional random fields. In RANLP. [find](https://scholar.google.com/scholar?q=Georgi+Georgiev%2C+Preslav+Nakov%2C+Kuzman+Ganchev%2C+Petya+Osenova%2C+and+Kiril+Ivanov+Simov%2E+2009%2E+Feature%2Drich+named+entity+recognition+for+Bulgarian+using+conditional+random+fields%2E+In+RANLP%2E)
6. Releasing the Models We release the best BERT based NER model along with the BERT model pre-trained on the four com-Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146. 6 [http://bsnlp-2017.cs.helsinki.fi/](http://bsnlp-2017.cs.helsinki.fi/) shared_task.… [↗](http://bsnlp-2017.cs.helsinki.fi/)
7. Slavic-BERT-NER Model Bi-LSTM-CRF (Lample et al., 2016) Multilingual BERT 5 Multilingual BERT-CRF Slavic BERT Slavic BERT-CRF Span F 1 75.8 79.6 81.4 83.5 87.9 [find](https://scholar.google.com/scholar?q=Slavic%2DBERT%2DNER+Model+Bi%2DLSTM%2DCRF+%28Lample+et+al%2E%2C+2016%29+Multilingual+BERT+5+Multilingual+BERT%2DCRF+Slavic+BERT+Slavic+BERT%2DCRF+Span+F+1+75%2E8+79%2E6+81%2E4+83%2E5+87%2E9)
8. RPM 73.9 77.8 80.9 83.8 85.7 (90.9) REM 72.1 76.1 79.2 82.0 84.3 (86.4) SM 72.3 77.2 79.6 82.2 84.1 (85.7) *Table 1: Metrics for BSNLP on validation set (Asia Bibi documents). Metrics on the test set are in the brackets.* Language cs ru bg pl P 93.6 88.2 90.3 93.4 R 94.7 86.6 84.3 93.1 F 1 93.9 87.3… [find](https://scholar.google.com/scholar?q=RPM+73%2E9+77%2E8+80%2E9+83%2E8+85%2E7+%2890%2E9%29+REM+72%2E1+76%2E1+79%2E2+82%2E0+84%2E3+%2886%2E4%29+SM+72%2E3+77%2E2+79%2E6+82%2E2+84%2E1+%2885%2E7%29+%2ATable+1%3A+Metrics+for+BSNLP+on+validation+set+%28Asia+Bibi+documents%29%2E+Metrics+on+the+test+set+)
9. Kiril Simov, Petya Osenova, Sia Kolkovska, Elisaveta Balabanova, and Dimitar Doikoff. 2004. A language resources infrastructure for Bulgarian. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), Lisbon, Portugal. European Language Resources Associati… [find](https://scholar.google.com/scholar?q=Kiril+Simov%2C+Petya+Osenova%2C+Sia+Kolkovska%2C+Elisaveta+Balabanova%2C+and+Dimitar+Doikoff%2E+2004%2E+A+language+resources+infrastructure+for+Bulgarian%2E+In+Proceedings+of+the+Fourth+International+Conference+on+)
10. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360. [find](https://scholar.google.com/scholar?q=Guillaume+Lample%2C+Miguel+Ballesteros%2C+Sandeep+Subramanian%2C+Kazuya+Kawakami%2C+and+Chris+Dyer%2E+2016%2E+Neural+architectures+for+named+entity+recognition%2E+arXiv+preprint+arXiv%3A1603%2E01360%2E)
11. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Advances+in+neural+information+processi)
12. Edward Loper and Steven Bird. 2002. NLTK: the natural language toolkit. arXiv preprint cs/0205028. [find](https://scholar.google.com/scholar?q=Edward+Loper+and+Steven+Bird%2E+2002%2E+NLTK%3A+the+natural+language+toolkit%2E+arXiv+preprint+cs%2F0205028%2E)
13. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V%2E+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+Jeff+Klingner%2C+Apurva+Shah%2C+Melvin+Johnson%2C+Xiaobing+Liu%2C+Luk)
14. Valerie Mozharova and Natalia Loukachevitch. 2016. Two-stage approach in Russian named entity recognition. In 2016 International FRUCT Conference on Intelligence, Social Media and Web (ISMW FRUCT), pages 1–6. IEEE. [find](https://scholar.google.com/scholar?q=Valerie+Mozharova+and+Natalia+Loukachevitch%2E+2016%2E+Two%2Dstage+approach+in+Russian+named+entity+recognition%2E+In+2016+International+FRUCT+Conference+on+Intelligence%2C+Social+Media+and+Web+%28ISMW+FRUCT%29%2C+pa)
15. Phoebe Mulcaire, Swabha Swayamdipta, and Noah Smith. 2018. Polyglot semantic role labeling. arXiv preprint arXiv:1805.11598. [find](https://scholar.google.com/scholar?q=Phoebe+Mulcaire%2C+Swabha+Swayamdipta%2C+and+Noah+Smith%2E+2018%2E+Polyglot+semantic+role+labeling%2E+arXiv+preprint+arXiv%3A1805%2E11598%2E)
16. Jakub Piskorski, Laska Laskova, Michał Marcińczuk, Lidia Pivovarova, Pavel Přibáň, Josef Steinberger, and Roman Yangarber. 2019. The second cross-lingual challenge on recognition, classification, lemmatization, and linking of named entities across Slavic languages. In Proceedings of the 7th Works… [find](https://scholar.google.com/scholar?q=Jakub+Piskorski%2C+Laska+Laskova%2C+Micha%C5%82+Marcin%CC%81czuk%2C+Lidia+Pivovarova%2C+Pavel+Pr%CC%8Cib%C3%A1n%CC%8C%2C+Josef+Steinberger%2C+and+Roman+Yangarber%2E+2019%2E+The+second+cross%2Dlingual+challenge+on+recognition%2C+classification%2C+l)
17. Jakub Piskorski, Lidia Pivovarova, Jan Šnajder, Josef Steinberger, and Roman Yangarber. 2017. The first cross-lingual challenge on recognition, normalization, and matching of named entities in Slavic languages. In Proceedings of the 6th Workshop on Balto-Slavic Natural Language Processing, pages 76–… [find](https://scholar.google.com/scholar?q=Jakub+Piskorski%2C+Lidia+Pivovarova%2C+Jan+%C5%A0najder%2C+Josef+Steinberger%2C+and+Roman+Yangarber%2E+2017%2E+The+first+cross%2Dlingual+challenge+on+recognition%2C+normalization%2C+and+matching+of+named+entities+in+Slavic+)
18. Erik F Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. arXiv preprint cs/0306050. [find](https://scholar.google.com/scholar?q=Erik+F+Sang+and+Fien+De+Meulder%2E+2003%2E+Introduction+to+the+CoNLL%2D2003+shared+task%3A+Language%2Dindependent+named+entity+recognition%2E+arXiv+preprint+cs%2F0306050%2E)