---
title: "JABER and SABER — Ghaddar 2021"
description: "Before this work, several Arabic BERT models were public, but they had been compared under inconsistent settings."
authors: ["Abbas Ghaddar", "Yimeng Wu", "Ahmad Rashid", "Khalil Bibi", "Mehdi Rezagholizadeh", "Chao Xing", "Yasheng Wang", "Duan Xinyu", "Zhefeng Wang", "Baoxing Huai", "Xin Jiang", "Qun Liu", "Philippe Langlais"]
year: 2021
venue: "arXiv:2112.04329"
source: "https://arxiv.org/abs/2112.04329"
doi: "10.48550/arxiv.2112.04329"
arxiv: "2112.04329"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ghaddar-2021-jaber-saber-junior-senior-arabic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ghaddar-2021-jaber-saber-junior-senior-arabic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / JABER and SABER: Junior and Senior Arabic BERt

# JABER and SABER: Junior and Senior Arabic BERt

**Authors:** [Abbas Ghaddar](https://md.research.elamur.ai/free/en/authors/abbas-ghaddar/), [Yimeng Wu](https://md.research.elamur.ai/free/en/authors/yimeng-wu/), [Ahmad Rashid](https://md.research.elamur.ai/free/en/authors/ahmad-rashid/), [Khalil Bibi](https://md.research.elamur.ai/free/en/authors/khalil-bibi/), [Mehdi Rezagholizadeh](https://md.research.elamur.ai/free/en/authors/mehdi-rezagholizadeh/), [Chao Xing](https://md.research.elamur.ai/free/en/authors/chao-xing/), [Yasheng Wang](https://md.research.elamur.ai/free/en/authors/yasheng-wang/), [Duan Xinyu](https://md.research.elamur.ai/free/en/authors/duan-xinyu/), [Zhefeng Wang](https://md.research.elamur.ai/free/en/authors/zhefeng-wang/), [Baoxing Huai](https://md.research.elamur.ai/free/en/authors/baoxing-huai/), [Xin Jiang](https://md.research.elamur.ai/free/en/authors/xin-jiang/), [Qun Liu](https://md.research.elamur.ai/free/en/authors/qun-liu/) et al. (13 authors) · **Year:** 2021 · **Venue:** arXiv:2112.04329 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2112.04329) · [PDF](https://arxiv.org/pdf/2112.04329) · [DOI 10.48550/arxiv.2112.04329](https://doi.org/10.48550/arxiv.2112.04329) · [arXiv 2112.04329](https://arxiv.org/abs/2112.04329) · [OpenAlex](https://openalex.org/W4311210973)

**Citations:** 5 citations (openalex-snapshot · 2026-10-14) · [11](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Before this work, several Arabic BERT models were public, but they had been compared under inconsistent settings. The authors also argue that most of them were trained for too short a time. They built two new Arabic models with the same recipe: JABER, a 12-layer base model, and SABER, a 24-layer large model.

The pre-training data came from Common Crawl, OSCAR, crawled news, the El-Khair corpus and Arabic Wikipedia. Heavy in-house filtering cut this 514GB collection down to 115GB. Tokenization uses byte-level BPE (BBPE) with a 64k vocabulary, plus AraBERT-style text normalization.

Next, the authors put seven existing Arabic BERT variants through a fresh evaluation, using both ALUE and ANER corp NER. JABER beat every other base-size model on the overall average, and SABER reached a new state of the art on ALUE.

The work matters for two reasons. It shows that strong cleaning combined with fewer epochs can beat larger, noisier training runs. It also gives Arabic NLU a fair reference comparison of existing models.

## Key points
- Cleaning kept only 22% of the raw text overall: Common Crawl went from 475GB to 87GB, NEWS from 21GB to 14GB, EL-KHEIR from 16GB to 13GB, and WIKI from 1.6GB to 1GB.
- JABER has 135M parameters and was trained for 15 epochs. SABER has 369M parameters and was trained for 5 epochs. Both use a duplication factor of 3, a 64k BBPE vocabulary, and no tweets in the training data.
- On ALUE's 8 tasks, JABER's average was 2.1 points above ARBERT and 2.3 above MARBERT. SABER added another 3.6 points and reached 77.3%.
- On the hidden ALUE leaderboard test set, JABER beat Arabic-BERT by 6.6 points, a wider gap than the 3.2 points seen on dev.
- On ANER corp (mention-level F1, averaged over 5 runs), JABER scored 84.20. ARBERT followed closely at 84.03, and MARBERT came last at 80.50.

## Interesting ideas
- Throwing away almost four fifths of the raw corpus with strict heuristics did not hurt the model; it made training cheaper and the results better, so data quality can outweigh data volume. — For Mova (learn): Silver trees could be filtered just as harshly, keeping only sentences where witness parsers agree and the text passes noise checks, rather than maximizing the size of the silver set.
- A single table lines up each competing model's vocabulary, tokenizer, data size, duplication factor and epochs, and this makes problems such as under-training easy to spot. — For Mova (learn): Every parser or tagger version could carry a similar configuration card, so that gaps in evaluation scores can be traced to training choices rather than guessed at.
- All baselines were re-run under one shared fine-tuning and evaluation protocol rather than relying on numbers each team reported. — For Mova (tree): Perceptron and gparse runs, and any external parsers, should be scored on the same treebanks with the same UD dialect conversion, so that LAS comparisons are truly like for like.
- Byte-level BPE first turns text into bytes and then learns merges over them, so no input can ever be out of vocabulary, which helps with rare words. — For Mova (tokens): A byte-level fallback for unknown forms in gparse's subword input would mean noisy or foreign strings never fall through to a blank unknown token.
- Results were broken down by text genre: MARBERT was strong on tweets and dialect but weakest on news-based NER, and models gained more on a test set collected at a later time. — For Mova (answer): Reader accuracy could be reported separately for each source and time slice (school tests, fables, word problems), so that hidden domain dependence shows up early.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

## Cited by

Cited by

11 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 9 not yet.

- [On the importance of Data Scale in Pretraining Arabic Language Models](https://md.research.elamur.ai/free/en/papers/ghaddar-2024-importance-data-scale-pretraining-arabic/)
- [Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models](https://md.research.elamur.ai/free/en/papers/sengupta-2023-jais-jais-chat-arabic-centric/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ghaddar2021jaber,
  title = {{JABER and SABER: Junior and Senior Arabic BERt}},
  author = {Ghaddar, Abbas and Wu, Yimeng and Rashid, Ahmad and Bibi, Khalil and Rezagholizadeh, Mehdi and Xing, Chao and Wang, Yasheng and Xinyu, Duan and Wang, Zhefeng and Huai, Baoxing and Jiang, Xin and Liu, Qun and Langlais, Philippe},
  year = {2021},
  doi = {10.48550/arxiv.2112.04329},
  eprint = {2112.04329},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2112.04329}
}
```

- [← previous It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/schick-2021-it-s-not-just-size/)
- [next → Joint Universal Syntactic and Semantic Parsing](https://md.research.elamur.ai/free/en/papers/eskin-2021-joint-universal-syntactic-semantic-parsing/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

42 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2021.acl-long.551) [our card](https://md.research.elamur.ai/free/en/papers/abdul-2021-mageed-arbert-marbert-deep-bidirectional-transformers/)
2. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic lan-[https://www.mindspore.cn/](https://www.mindspore.cn/) [↗](https://www.mindspore.cn/)
3. Aaron Gokaslan and Vanya Cohen. 2019. Openwebtext corpus. URl: [https://skylion007.github](https://skylion007.github). io/OpenWebTextCorpus. [↗](https://skylion007.github)
4. Kyle Gorman and Steven Bedrick. 2019. We need to talk about standard splits. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 2786–2791. [↗](https://doi.org/10.18653/v1/p19-1267)
5. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104. [find](https://scholar.google.com/scholar?q=Go+Inoue%2C+Bashar+Alhafni%2C+Nurpeiis+Baimukan%2C+Houda+Bouamor%2C+and+Nizar+Habash%2E+2021%2E+The+interplay+of+variant%2C+size%2C+and+task+type+in+arabic+pre%2Dtrained+language+models%2E+In+Proceedings+of+the+Sixth+Ara) [our card](https://md.research.elamur.ai/free/en/papers/inoue-2021-interplay-variant-size-task-type/)
6. Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoit Crabbé, Laurent Besacier, and Didier Schwab. 2020. Flaubert: Unsupervised language model pre-training for french. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 2479–2490. [find](https://scholar.google.com/scholar?q=Hang+Le%2C+Lo%C3%AFc+Vial%2C+Jibril+Frej%2C+Vincent+Segonne%2C+Maximin+Coavoux%2C+Benjamin+Lecouteux%2C+Alexandre+Allauzen%2C+Benoit+Crabb%C3%A9%2C+Laurent+Besacier%2C+and+Didier+Schwab%2E+2020%2E+Flaubert%3A+Unsupervised+language+mod) [our card](https://md.research.elamur.ai/free/en/papers/le-2019-flaubert-unsupervised-language-model-pre/)
7. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [↗](https://arxiv.org/abs/1907.11692)
8. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. [↗](https://arxiv.org/abs/1711.05101)
9. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah, and Benoît Sagot. 2019. Camembert: a tasty french language model. arXiv preprint arXiv:1911.03894. [↗](https://arxiv.org/abs/1911.03894) [our card](https://md.research.elamur.ai/free/en/papers/martin-2019-camembert-tasty-french-language-model/)
10. Sebastian Nagel. 2016. Cc-news. URL: [http://web.archive](http://web.archive). org/save/http://commoncrawl. org/2016/10/newsdatasetavailable. [↗](http://web.archive)
11. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL tools: An open source python toolkit for Arabic natural language processing. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Ossama+Obeid%2C+Nasser+Zalmout%2C+Salam+Khalifa%2C+Dima+Taji%2C+Mai+Oudah%2C+Bashar+Alhafni%2C+Go+Inoue%2C+Fadhl+Eryani%2C+Alexander+Erdmann%2C+and+Nizar+Habash%2E+2020%2E+CAMeL+tools%3A+An+open+source+python+toolkit+for+Ara) [our card](https://md.research.elamur.ai/free/en/papers/obeid-2020-camel-tools-open-source-python/)
12. Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. 2020. Green ai. Communications of the ACM, 63(12):54–63. [find](https://scholar.google.com/scholar?q=Roy+Schwartz%2C+Jesse+Dodge%2C+Noah+A+Smith%2C+and+Oren+Etzioni%2E+2020%2E+Green+ai%2E+Communications+of+the+ACM%2C+63%2812%29%3A54%E2%80%9363%2E) [our card](https://md.research.elamur.ai/free/en/papers/schwartz-2019-green-ai/)
13. Haitham Seelawi, Ibraheem Tuffaha, Mahmoud Gzawi, Wael Farhan, Bashar Talafha, Riham Badawi, Zyad Sober, Oday Al-Dweik, Abed Alhakim Freihat, and Hussein Al-Natsheh. 2021. Alue: Arabic language understanding evaluation. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 173–184. [find](https://scholar.google.com/scholar?q=Haitham+Seelawi%2C+Ibraheem+Tuffaha%2C+Mahmoud+Gzawi%2C+Wael+Farhan%2C+Bashar+Talafha%2C+Riham+Badawi%2C+Zyad+Sober%2C+Oday+Al%2DDweik%2C+Abed+Alhakim+Freihat%2C+and+Hussein+Al%2DNatsheh%2E+2021%2E+Alue%3A+Arabic+language+unders)
14. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+Ling)
15. Alexander Sergeev and Mike Del Balso. 2018. Horovod: fast and easy distributed deep learning in tensorflow. arXiv preprint arXiv:1802.05799. [↗](https://arxiv.org/abs/1802.05799)
16. Tatiana Shavrina, Alena Fenogenova, Anton Emelyanov, Denis Shevelev, Ekaterina Artemova, Valentin Malykh, Vladislav Mikhailov, Maria Tikhonova, Andrey Chertok, and Andrey Evlampiev. 2020. Russiansuperglue: A russian language understanding evaluation benchmark. arXiv preprint arXiv:2010.15925. [↗](https://arxiv.org/abs/2010.15925)
17. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7). Leibniz-Institut für Deutsche Sprache. [find](https://scholar.google.com/scholar?q=Pedro+Javier+Ortiz+Su%C3%A1rez%2C+Beno%C3%AEt+Sagot%2C+and+Laurent+Romary%2E+2019%2E+Asynchronous+pipeline+for+processing+huge+corpora+on+medium+to+low+resource+infrastructures%2E+In+7th+Workshop+on+the+Challenges+in+the)
18. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026– 8037. [find](https://scholar.google.com/scholar?q=Adam+Paszke%2C+Sam+Gross%2C+Francisco+Massa%2C+Adam+Lerer%2C+James+Bradbury%2C+Gregory+Chanan%2C+Trevor+Killeen%2C+Zeming+Lin%2C+Natalia+Gimelshein%2C+Luca+Antiga%2C+et+al%2E+2019%2E+Pytorch%3A+An+imperative+style%2C+high%2Dperfor) [our card](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/)
19. Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, et al. 2021. Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. arXiv preprint arXiv:2107.02137. [↗](https://arxiv.org/abs/2107.02137) [our card](https://md.research.elamur.ai/free/en/papers/sun-2021-ernie-3-0-large/)
20. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual bert? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996–5001. [find](https://scholar.google.com/scholar?q=Telmo+Pires%2C+Eva+Schlinger%2C+and+Dan+Garrette%2E+2019%2E+How+multilingual+is+multilingual+bert%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+4996%E2%80%935001%2E) [our card](https://md.research.elamur.ai/free/en/papers/pires-2019-how-multilingual-is-multilingual-bert/)
21. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. Ernie: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223. [↗](https://arxiv.org/abs/1904.09223) [our card](https://md.research.elamur.ai/free/en/papers/sun-2019-ernie-enhanced-representation-through-knowledge/)
22. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. [↗](https://arxiv.org/abs/1910.10683)
23. S Ruder. 2021. Challenges and opportunities in nlp benchmarking. [find](https://scholar.google.com/scholar?q=S+Ruder%2E+2021%2E+Challenges+and+opportunities+in+nlp+benchmarking%2E)
24. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. Kuisail at semeval-2020 task 12: Bert-cnn for offensive speech identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059. [find](https://scholar.google.com/scholar?q=Ali+Safaya%2C+Moutasem+Abdullatif%2C+and+Deniz+Yuret%2E+2020%2E+Kuisail+at+semeval%2D2020+task+12%3A+Bert%2Dcnn+for+offensive+speech+identification+in+social+media%2E+In+Proceedings+of+the+Fourteenth+Workshop+on+Sema) [our card](https://md.research.elamur.ai/free/en/papers/safaya-2020-kuisail-semeval-2020-task-12/)
25. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2020. Ernie 2.0: A continual pre-training framework for language understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8968– 8975. [find](https://scholar.google.com/scholar?q=Yu+Sun%2C+Shuohuan+Wang%2C+Yukun+Li%2C+Shikun+Feng%2C+Hao+Tian%2C+Hua+Wu%2C+and+Haifeng+Wang%2E+2020%2E+Ernie+2%2E0%3A+A+continual+pre%2Dtraining+framework+for+language+understanding%2E+In+Proceedings+of+the+AAAI+Conference+) [our card](https://md.research.elamur.ai/free/en/papers/sun-2019-ernie-2-0-continual/)
26. Erik F Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003-Volume 4, pages 142–147. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Erik+F+Tjong+Kim+Sang+and+Fien+De+Meulder%2E+2003%2E+Introduction+to+the+CoNLL%2D2003+shared+task%3A+Language%2Dindependent+named+entity+recognition%2E+In+Proceedings+of+the+seventh+conference+on+Natural+language)
27. Trieu H Trinh and Quoc V Le. 2018. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847. [↗](https://arxiv.org/abs/1806.02847)
28. Matej Ulčar and Marko Robnik-Šikonja. 2020. Finest bert and crosloengual bert: less is more in multilingual models. arXiv preprint arXiv:2006.07890. [↗](https://arxiv.org/abs/2006.07890)
29. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Advances+in+Neural+Information+Processi)
30. Liang Xu, Xiaojing Lu, Chenyang Yuan, Xuanwei Zhang, Hu Yuan, Huilin Xu, Guoao Wei, Xiang Pan, and Hai Hu. 2021. Fewclue: A chinese few-shot learning evaluation benchmark. arXiv preprint arXiv:2107.07498. [↗](https://arxiv.org/abs/2107.07498)
31. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5754–5764. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Zihang+Dai%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Russ+R+Salakhutdinov%2C+and+Quoc+V+Le%2E+2019%2E+Xlnet%3A+Generalized+autoregressive+pretraining+for+language+understanding%2E+In+Advances+in+neural+inform) [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)
32. Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: Bert for finnish. arXiv preprint arXiv:1912.07076. [↗](https://arxiv.org/abs/1912.07076) [our card](https://md.research.elamur.ai/free/en/papers/virtanen-2019-multilingual-enough-bert-finnish/)
33. Wei Zeng, Xiaozhe Ren, Teng Su, Hui Wang, Yi Liao, Zhiwei Wang, Xin Jiang, ZhenZhang Yang, Kaisheng Wang, Xiaoda Zhang, et al. 2021. Pangu-α: Large-scale autoregressive pretrained chinese language models with auto-parallel computation. arXiv preprint arXiv:2104.12369. [↗](https://arxiv.org/abs/2104.12369) [our card](https://md.research.elamur.ai/free/en/papers/zeng-2021-pangu-alpha-large-scale/)
34. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537. [↗](https://arxiv.org/abs/1905.00537) [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
35. Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and Abdelhak Lakhouaja. 2019. Osian: Open source international arabic news corpus-preparation and integration into the clarin-infrastructure. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 175–182. [find](https://scholar.google.com/scholar?q=Imad+Zeroual%2C+Dirk+Goldhahn%2C+Thomas+Eckart%2C+and+Abdelhak+Lakhouaja%2E+2019%2E+Osian%3A+Open+source+international+arabic+news+corpus%2Dpreparation+and+integration+into+the+clarin%2Dinfrastructure%2E+In+Proceedings)
36. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+Bowman%2E+2018%2E+Glue%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+In+Proceedings+of+the) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
37. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19– 27. [↗](https://doi.org/10.1109/iccv.2015.11) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/)
38. Junqiu Wei, Qun Liu, Yinpeng Guo, and Xin Jiang. 2021. Training multilingual pre-trained language model with byte-level subwords. arXiv preprint arXiv:2101.09469. [↗](https://arxiv.org/abs/2101.09469)
39. Junqiu Wei, Xiaozhe Ren, Xiaoguang Li, Wenyong Huang, Yi Liao, Yasheng Wang, Jiashu Lin, Xin Jiang, Xiao Chen, and Qun Liu. 2019. Nezha: Neural contextualized representation for chinese language understanding. arXiv preprint arXiv:1909.00204. [↗](https://arxiv.org/abs/1909.00204)
40. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2019. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359. [↗](https://arxiv.org/abs/1911.00359) [our card](https://md.research.elamur.ai/free/en/papers/wenzek-2020-ccnet-extracting-high-quality-monolingual/)
41. Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pier-ric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45. [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Julien+Chaumond%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pier%2Dric+Cistac%2C+Morgan+Funtowicz%2C+Joe+Davison%2C+Sam+Shleifer%2C+et+al%2E+2020%2E+Transformers%3A+State%2Dof%2Dthe%2Dart+natu) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
42. Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al. 2020. Clue: A chinese language understanding evaluation benchmark. arXiv preprint arXiv:2004.05986. [↗](https://arxiv.org/abs/2004.05986) [our card](https://md.research.elamur.ai/free/en/papers/xu-2020-clue-chinese-language-understanding/)