---
title: "PhoBERT: Pre-trained language models for — Nguyen 2020"
description: "Before this work, Vietnamese had no publicly available large-scale monolingual BERT-style model."
authors: ["Dat Quoc Nguyen", "Anh Tuan Nguyen"]
year: 2020
venue: "Findings of the Association for Computational Linguistics: EMNLP 2020"
source: "https://aclanthology.org/2020.findings-emnlp.92/"
doi: "10.18653/v1/2020.findings-emnlp.92"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nguyen-2020-phobert-pre-trained-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nguyen-2020-phobert-pre-trained-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/) / PhoBERT: Pre-trained language models for Vietnamese

# PhoBERT: Pre-trained language models for Vietnamese

**Authors:** [Dat Quoc Nguyen](https://md.research.elamur.ai/free/en/authors/dat-quoc-nguyen/), [Anh Tuan Nguyen](https://md.research.elamur.ai/free/en/authors/anh-tuan-nguyen/) · **Year:** 2020 · **Venue:** Findings of the Association for Computational Linguistics: EMNLP 2020 · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.findings-emnlp.92/) · [PDF](https://aclanthology.org/2020.findings-emnlp.92.pdf) · [DOI 10.18653/v1/2020.findings-emnlp.92](https://doi.org/10.18653/v1/2020.findings-emnlp.92) · [OpenAlex](https://openalex.org/W3007955273)

**Citations:** 456 citations (openalex-snapshot · 2026-10-14) · [115](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Before this work, Vietnamese had no publicly available large-scale monolingual BERT-style model. Existing monolingual and multilingual models had two weaknesses. They applied BPE directly to syllables, even though Vietnamese uses spaces between syllables and many words span several syllables. They also relied mainly on the small Vietnamese Wikipedia (about 1GB). The authors built PhoBERT in base and large sizes, following the BERT architectures and the RoBERTa pre-training recipe in fairseq. They first word-segmented a 20GB corpus of Wikipedia and deduplicated news text with RDRSegmenter, then applied fastBPE. The models were then fine-tuned and evaluated on four downstream problems: part-of-speech (POS) tagging, named-entity recognition (NER), natural language inference (NLI) and parsing of dependency structure. PhoBERT set new state-of-the-art results on all four and beat XLM-R on every task, even though XLM-R saw roughly seven times more Vietnamese text and has more parameters. This supports the view that dedicated single-language models can outperform multilingual ones. Because the models are released for fairseq and transformers, they serve as a strong baseline for Vietnamese NLP.

## Key points
- **Pre-training data:** about 1GB of Vietnamese Wikipedia plus about 19GB of news text, filtered for duplicates from a 50GB news corpus. After segmentation this gives about 145M sentences and about 3B word tokens, with a 64K subword vocabulary.
- **Training:** both models were trained on 4 V100 GPUs. PhoBERT base took about 3 weeks (≈540K steps) and PhoBERT large about 5 weeks (≈1.08M steps).
- **Benchmarks:** VLSP 2013 POS, VnDT treebank v1.1, VLSP 2016 NER and the Vietnamese portion of XNLI.
- **Best scores:**
  - POS accuracy: 96.8 (PhoBERT large).
  - Dependency parsing: LAS 78.77 / UAS 85.22 (Biaffine parser with PhoBERT base).
  - NER F1: 94.7 (PhoBERT large).
  - NLI accuracy: 80.0 (PhoBERT large).
- **Model size and caveats:** PhoBERT is smaller than XLM-R (135M vs 250M parameters for base, 370M vs 560M for large). However, the large model parsed slightly worse than the base model.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Named entity recognition](https://md.research.elamur.ai/free/en/topics/named-entity-recognition/) · [Dependency parsing](https://md.research.elamur.ai/free/en/topics/dependency-parsing/)

## Cited by

Cited by

115 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 111 not yet.

- [ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic](https://md.research.elamur.ai/free/en/papers/abdul-2021-mageed-arbert-marbert-deep-bidirectional-transformers/)
- [FlauBERT: Unsupervised Language Model Pre-training for French](https://md.research.elamur.ai/free/en/papers/le-2019-flaubert-unsupervised-language-model-pre/)
- [FreeTxt-Vi: A Benchmarked Vietnamese-English Toolkit for Segmentation, Sentiment, and Summarisation](https://md.research.elamur.ai/free/en/papers/nguyen-2026-freetxt-vi-benchmarked-vietnamese-english/)
- [Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing](https://md.research.elamur.ai/free/en/papers/ud-2021-nguyen-trankit-light-weight-transformer-based/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{nguyen2020phobert,
  title = {{PhoBERT: Pre-trained language models for Vietnamese}},
  author = {Nguyen, Dat Quoc and Nguyen, Anh Tuan},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.92},
  url = {https://doi.org/10.18653/v1/2020.findings-emnlp.92}
}
```

- [← previous Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages](https://md.research.elamur.ai/free/en/papers/nekoto-2020-participatory-research-low-resourced-machine/)
- [next → Playing with Words at the National Library of Sweden -- Making a Swedish BERT](https://md.research.elamur.ai/free/en/papers/malmsten-2020-playing-words-national-library-sweden/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/)

## References

34 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mikel Artetxe and Holger Schwenk. 2019. Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond. TACL, 7:597– 610. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe+and+Holger+Schwenk%2E+2019%2E+Massively+Multilingual+Sentence+Embeddings+for+Zero%2DShot+Cross%2DLingual+Transfer+and+Beyond%2E+TACL%2C+7%3A597%E2%80%93+610%2E) [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2018-massively-multilingual-sentence-embeddings-zero/)
2. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of ACL, pages 8440–8451. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+Cross%2Dlingu)
3. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual Language Model Pretraining. In Proceedings of NeurIPS, pages 7059–7069. [find](https://scholar.google.com/scholar?q=Alexis+Conneau+and+Guillaume+Lample%2E+2019%2E+Cross%2Dlingual+Language+Model+Pretraining%2E+In+Proceedings+of+NeurIPS%2C+pages+7059%E2%80%937069%2E)
4. Alexis Conneau, Ruty Rinott, Guillaume Lample, Holger Schwenk, Ves Stoyanov, Adina Williams, and Samuel R. Bowman. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of EMNLP, pages 2475–2485. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Ruty+Rinott%2C+Guillaume+Lample%2C+Holger+Schwenk%2C+Ves+Stoyanov%2C+Adina+Williams%2C+and+Samuel+R%2E+Bowman%2E+2018%2E+XNLI%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+In+Proceedings+of+EMNL)
5. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. 2019. Pre-Training with Whole Word Masking for Chinese BERT. arXiv preprint, arXiv:1906.08101. [find](https://scholar.google.com/scholar?q=Yiming+Cui%2C+Wanxiang+Che%2C+Ting+Liu%2C+Bing+Qin%2C+Ziqing+Yang%2C+Shijin+Wang%2C+and+Guoping+Hu%2E+2019%2E+Pre%2DTraining+with+Whole+Word+Masking+for+Chinese+BERT%2E+arXiv+preprint%2C+arXiv%3A1906%2E08101%2E) [our card](https://md.research.elamur.ai/free/en/papers/cui-2019-pre-training-whole-word-masking/)
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL, pages 4171– 4186. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+NAACL%2C+pages+4171%E2%80%93+4186%2E)
7. Timothy Dozat and Christopher D. Manning. 2017. Deep Biaffine Attention for Neural Dependency Parsing. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Timothy+Dozat+and+Christopher+D%2E+Manning%2E+2017%2E+Deep+Biaffine+Attention+for+Neural+Dependency+Parsing%2E+In+Proceedings+of+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/missing-2016-dozat-deep-biaffine-attention-neural-dependency/)
8. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of NAACL, pages 4129– 4138. [find](https://scholar.google.com/scholar?q=John+Hewitt+and+Christopher+D%2E+Manning%2E+2019%2E+A+structural+probe+for+finding+syntax+in+word+representations%2E+In+Proceedings+of+NAACL%2C+pages+4129%E2%80%93+4138%2E) [our card](https://md.research.elamur.ai/free/en/papers/hewitt-2019-structural-probe-finding-syntax-word/)
9. Ganesh Jawahar, Benoı̂t Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of ACL, pages 3651– 3657. [find](https://scholar.google.com/scholar?q=Ganesh+Jawahar%2C+Beno%C4%B1%CC%82t+Sagot%2C+and+Djam%C3%A9+Seddah%2E+2019%2E+What+does+BERT+learn+about+the+structure+of+language%3F+In+Proceedings+of+ACL%2C+pages+3651%E2%80%93+3657%2E) [our card](https://md.research.elamur.ai/free/en/papers/jawahar-2019-what-does-bert-learn-about/)
10. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv preprint, arXiv:1412.6980. [find](https://scholar.google.com/scholar?q=Diederik+P%2E+Kingma+and+Jimmy+Ba%2E+2014%2E+Adam%3A+A+Method+for+Stochastic+Optimization%2E+arXiv+preprint%2C+arXiv%3A1412%2E6980%2E)
11. Taku Kudo and John Richardson. 2018. Sentence-Piece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In Proceedings of EMNLP: System Demonstrations, pages 66–71. [find](https://scholar.google.com/scholar?q=Taku+Kudo+and+John+Richardson%2E+2018%2E+Sentence%2DPiece%3A+A+simple+and+language+independent+subword+tokenizer+and+detokenizer+for+Neural+Text+Processing%2E+In+Proceedings+of+EMNLP%3A+System+Demonstrations%2C+pag)
12. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint, arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+RoBERTa%3A+A+Robustly+Optimized+BERT+Pretraining+Approach%2E+a)
13. Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+Weight+Decay+Regularization%2E+In+Proceedings+of+ICLR%2E)
14. Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proceedings of ACL, pages 1064–1074. [find](https://scholar.google.com/scholar?q=Xuezhe+Ma+and+Eduard+Hovy%2E+2016%2E+End%2Dto%2Dend+sequence+labeling+via+bi%2Ddirectional+LSTM%2DCNNs%2DCRF%2E+In+Proceedings+of+ACL%2C+pages+1064%E2%80%931074%2E) [our card](https://md.research.elamur.ai/free/en/papers/ma-2016-end-end-sequence-labeling-bi/)
15. Xuezhe Ma, Zecong Hu, Jingzhou Liu, Nanyun Peng, Graham Neubig, and Eduard Hovy. 2018. Stack-Pointer Networks for Dependency Parsing. In Proceedings of ACL, pages 1403–1414. [find](https://scholar.google.com/scholar?q=Xuezhe+Ma%2C+Zecong+Hu%2C+Jingzhou+Liu%2C+Nanyun+Peng%2C+Graham+Neubig%2C+and+Eduard+Hovy%2E+2018%2E+Stack%2DPointer+Networks+for+Dependency+Parsing%2E+In+Proceedings+of+ACL%2C+pages+1403%E2%80%931414%2E)
16. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah, and Benoı̂t Sagot. 2020. CamemBERT: a Tasty French Language Model. In Proceedings of ACL, pages 7203– 7219. [find](https://scholar.google.com/scholar?q=Louis+Martin%2C+Benjamin+Muller%2C+Pedro+Javier+Ortiz+Su%C3%A1rez%2C+Yoann+Dupont%2C+Laurent+Romary%2C+%C3%89ric+Villemonte+de+la+Clergerie%2C+Djam%C3%A9+Seddah%2C+and+Beno%C4%B1%CC%82t+Sagot%2E+2020%2E+CamemBERT%3A+a+Tasty+French+Language+Model) [our card](https://md.research.elamur.ai/free/en/papers/martin-2019-camembert-tasty-french-language-model/)
17. Dat Quoc Nguyen. 2019. A neural joint model for Vietnamese word segmentation, POS tagging and dependency parsing. In Proceedings of ALTA, pages 28– 34. [find](https://scholar.google.com/scholar?q=Dat+Quoc+Nguyen%2E+2019%2E+A+neural+joint+model+for+Vietnamese+word+segmentation%2C+POS+tagging+and+dependency+parsing%2E+In+Proceedings+of+ALTA%2C+pages+28%E2%80%93+34%2E)
18. Dat Quoc Nguyen, Dai Quoc Nguyen, Dang Duc Pham, and Son Bao Pham. 2014a. RDRPOSTagger: A Ripple Down Rules-based Part-Of-Speech Tagger. In Proceedings of the Demonstrations at EACL, pages 17–20. [find](https://scholar.google.com/scholar?q=Dat+Quoc+Nguyen%2C+Dai+Quoc+Nguyen%2C+Dang+Duc+Pham%2C+and+Son+Bao+Pham%2E+2014a%2E+RDRPOSTagger%3A+A+Ripple+Down+Rules%2Dbased+Part%2DOf%2DSpeech+Tagger%2E+In+Proceedings+of+the+Demonstrations+at+EACL%2C+pages+17%E2%80%9320%2E)
19. Dat Quoc Nguyen, Dai Quoc Nguyen, Son Bao Pham, Phuong-Thai Nguyen, and Minh Le Nguyen. 2014b. From Treebank Conversion to Automatic Dependency Parsing for Vietnamese. In Proceedings of NLDB, pages 196–207. [find](https://scholar.google.com/scholar?q=Dat+Quoc+Nguyen%2C+Dai+Quoc+Nguyen%2C+Son+Bao+Pham%2C+Phuong%2DThai+Nguyen%2C+and+Minh+Le+Nguyen%2E+2014b%2E+From+Treebank+Conversion+to+Automatic+Dependency+Parsing+for+Vietnamese%2E+In+Proceedings+of+NLDB%2C+pages+19)
20. Dat Quoc Nguyen, Dai Quoc Nguyen, Thanh Vu, Mark Dras, and Mark Johnson. 2018. A Fast and Accurate Vietnamese Word Segmenter. In Proceedings of LREC, pages 2582–2587. [find](https://scholar.google.com/scholar?q=Dat+Quoc+Nguyen%2C+Dai+Quoc+Nguyen%2C+Thanh+Vu%2C+Mark+Dras%2C+and+Mark+Johnson%2E+2018%2E+A+Fast+and+Accurate+Vietnamese+Word+Segmenter%2E+In+Proceedings+of+LREC%2C+pages+2582%E2%80%932587%2E)
21. Dat Quoc Nguyen and Karin Verspoor. 2018. An improved neural network model for joint POS tagging and dependency parsing. In Proceedings of the CoNLL 2018 Shared Task, pages 81–91. [find](https://scholar.google.com/scholar?q=Dat+Quoc+Nguyen+and+Karin+Verspoor%2E+2018%2E+An+improved+neural+network+model+for+joint+POS+tagging+and+dependency+parsing%2E+In+Proceedings+of+the+CoNLL+2018+Shared+Task%2C+pages+81%E2%80%9391%2E)
22. Dat Quoc Nguyen, Thanh Vu, Dai Quoc Nguyen, Mark Dras, and Mark Johnson. 2017. From word segmentation to POS tagging for Vietnamese. In Proceedings of ALTA, pages 108–113. [find](https://scholar.google.com/scholar?q=Dat+Quoc+Nguyen%2C+Thanh+Vu%2C+Dai+Quoc+Nguyen%2C+Mark+Dras%2C+and+Mark+Johnson%2E+2017%2E+From+word+segmentation+to+POS+tagging+for+Vietnamese%2E+In+Proceedings+of+ALTA%2C+pages+108%E2%80%93113%2E)
23. Huyen Nguyen, Quyen Ngo, Luong Vu, Vu Tran, and Hien Nguyen. 2019a. VLSP Shared Task: Named Entity Recognition. Journal of Computer Science and Cybernetics, 34(4):283–294. [find](https://scholar.google.com/scholar?q=Huyen+Nguyen%2C+Quyen+Ngo%2C+Luong+Vu%2C+Vu+Tran%2C+and+Hien+Nguyen%2E+2019a%2E+VLSP+Shared+Task%3A+Named+Entity+Recognition%2E+Journal+of+Computer+Science+and+Cybernetics%2C+34%284%29%3A283%E2%80%93294%2E)
24. Kim Anh Nguyen, Ngan Dong, and Cam-Tu Nguyen. 2019b. Attentive Neural Network for Named Entity Recognition in Vietnamese. In Proceedings of RIVF. [find](https://scholar.google.com/scholar?q=Kim+Anh+Nguyen%2C+Ngan+Dong%2C+and+Cam%2DTu+Nguyen%2E+2019b%2E+Attentive+Neural+Network+for+Named+Entity+Recognition+in+Vietnamese%2E+In+Proceedings+of+RIVF%2E)
25. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A Fast, Extensible Toolkit for Sequence Modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, pages 48–53. [find](https://scholar.google.com/scholar?q=Myle+Ott%2C+Sergey+Edunov%2C+Alexei+Baevski%2C+Angela+Fan%2C+Sam+Gross%2C+Nathan+Ng%2C+David+Grangier%2C+and+Michael+Auli%2E+2019%2E+fairseq%3A+A+Fast%2C+Extensible+Toolkit+for+Sequence+Modeling%2E+In+Proceedings+of+NAACL%2DHL) [our card](https://md.research.elamur.ai/free/en/papers/ott-2019-fairseq-fast-extensible-toolkit-sequence/)
26. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of ACL, pages 1715–1725. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016%2E+Neural+Machine+Translation+of+Rare+Words+with+Subword+Units%2E+In+Proceedings+of+ACL%2C+pages+1715%E2%80%931725%2E)
27. Dinh Quang Thang, Le Hong Phuong, Nguyen Thi Minh Huyen, Nguyen Cam Tu, Mathias Rossignol, and Vu Xuan Luong. 2008. Word segmentation of Vietnamese texts: a comparison of approaches. In Proceedings of LREC, pages 1933–1936. [find](https://scholar.google.com/scholar?q=Dinh+Quang+Thang%2C+Le+Hong+Phuong%2C+Nguyen+Thi+Minh+Huyen%2C+Nguyen+Cam+Tu%2C+Mathias+Rossignol%2C+and+Vu+Xuan+Luong%2E+2008%2E+Word+segmentation+of+Vietnamese+texts%3A+a+comparison+of+approaches%2E+In+Proceedings+of)
28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems 30, pages 5998–6008. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+All+you+Need%2E+In+Advances+in+Neural+Information+Processi)
29. Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019. BERTje: A Dutch BERT Model. arXiv preprint, arXiv:1912.09582. [find](https://scholar.google.com/scholar?q=Wietse+de+Vries%2C+Andreas+van+Cranenburgh%2C+Arianna+Bisazza%2C+Tommaso+Caselli%2C+Gertjan+van+Noord%2C+and+Malvina+Nissim%2E+2019%2E+BERTje%3A+A+Dutch+BERT+Model%2E+arXiv+preprint%2C+arXiv%3A1912%2E09582%2E) [our card](https://md.research.elamur.ai/free/en/papers/de-2019-vries-bertje-dutch-bert-model/)
30. Thanh Vu, Dat Quoc Nguyen, Dai Quoc Nguyen, Mark Dras, and Mark Johnson. 2018. VnCoreNLP: A Vietnamese Natural Language Processing Toolkit. In Proceedings of NAACL: Demonstrations, pages 56–60. [find](https://scholar.google.com/scholar?q=Thanh+Vu%2C+Dat+Quoc+Nguyen%2C+Dai+Quoc+Nguyen%2C+Mark+Dras%2C+and+Mark+Johnson%2E+2018%2E+VnCoreNLP%3A+A+Vietnamese+Natural+Language+Processing+Toolkit%2E+In+Proceedings+of+NAACL%3A+Demonstrations%2C+pages+56%E2%80%9360%2E)
31. Xuan-Son Vu, Thanh Vu, Son Tran, and Lili Jiang. 2019. ETNLP: A visual-aided systematic approach to select pre-trained embeddings for a downstream task. In Proceedings of RANLP, pages 1285–1294. [find](https://scholar.google.com/scholar?q=Xuan%2DSon+Vu%2C+Thanh+Vu%2C+Son+Tran%2C+and+Lili+Jiang%2E+2019%2E+ETNLP%3A+A+visual%2Daided+systematic+approach+to+select+pre%2Dtrained+embeddings+for+a+downstream+task%2E+In+Proceedings+of+RANLP%2C+pages+1285%E2%80%931294%2E)
32. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of NAACL, pages 1112–1122. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+Broad%2DCoverage+Challenge+Corpus+for+Sentence+Understanding+through+Inference%2E+In+Proceedings+of+NAACL%2C+pages+1112%E2%80%931122%2E)
33. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace’s Transformers: State-of-the-art Natural Language Processing. arXiv preprint, arXiv:1910.03771. [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pier%2Dric+Cistac%2C+Tim+Rault%2C+R%E2%80%99emi+Louf%2C+Morgan+Funtowicz%2C+and+Jamie+Brew%2E+2019%2E+HuggingFace%E2%80%99s+Transformers%3A+St) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
34. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of EMNLP-IJCNLP, pages 833–844. [find](https://scholar.google.com/scholar?q=Shijie+Wu+and+Mark+Dredze%2E+2019%2E+Beto%2C+bentz%2C+becas%3A+The+surprising+cross%2Dlingual+effectiveness+of+BERT%2E+In+Proceedings+of+EMNLP%2DIJCNLP%2C+pages+833%E2%80%93844%2E) [our card](https://md.research.elamur.ai/free/en/papers/wu-2019-beto-bentz-becas-surprising-cross/)