---
title: "Revisiting Pre-trained Language Models and — Ghaddar 2022"
description: "The authors argue that Arabic pre-trained language models have not kept pace with English and Chinese for two reasons: pre-training recipes have barely…"
authors: ["Abbas Ghaddar", "Yimeng Wu", "Sunyam Bagga", "Ahmad Rashid", "Khalil Bibi", "Mehdi Rezagholizadeh", "Chao Xing", "Yasheng Wang", "Duan Xinyu", "Zhefeng Wang", "Baoxing Huai", "Xin Jiang", "Qun Liu", "Philippe Langlais"]
year: 2022
venue: "arXiv:2205.10687"
source: "https://arxiv.org/abs/2205.10687"
doi: "10.48550/arxiv.2205.10687"
arxiv: "2205.10687"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ghaddar-2022-revisiting-pre-trained-language-models-their/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ghaddar-2022-revisiting-pre-trained-language-models-their/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / Revisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Understanding

# Revisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Understanding

**Authors:** [Abbas Ghaddar](https://md.research.elamur.ai/free/en/authors/abbas-ghaddar/), [Yimeng Wu](https://md.research.elamur.ai/free/en/authors/yimeng-wu/), Sunyam Bagga, [Ahmad Rashid](https://md.research.elamur.ai/free/en/authors/ahmad-rashid/), [Khalil Bibi](https://md.research.elamur.ai/free/en/authors/khalil-bibi/), [Mehdi Rezagholizadeh](https://md.research.elamur.ai/free/en/authors/mehdi-rezagholizadeh/), [Chao Xing](https://md.research.elamur.ai/free/en/authors/chao-xing/), [Yasheng Wang](https://md.research.elamur.ai/free/en/authors/yasheng-wang/), [Duan Xinyu](https://md.research.elamur.ai/free/en/authors/duan-xinyu/), [Zhefeng Wang](https://md.research.elamur.ai/free/en/authors/zhefeng-wang/), [Baoxing Huai](https://md.research.elamur.ai/free/en/authors/baoxing-huai/), [Xin Jiang](https://md.research.elamur.ai/free/en/authors/xin-jiang/) et al. (14 authors) · **Year:** 2022 · **Venue:** arXiv:2205.10687 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2205.10687) · [PDF](https://arxiv.org/pdf/2205.10687) · [DOI 10.48550/arxiv.2205.10687](https://doi.org/10.48550/arxiv.2205.10687) · [arXiv 2205.10687](https://arxiv.org/abs/2205.10687) · [OpenAlex](https://openalex.org/W4281476622)

**Citations:** 5 citations (openalex-snapshot · 2026-10-14) · [4](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
The authors argue that Arabic pre-trained language models have not kept pace with English and Chinese for two reasons: pre-training recipes have barely been tuned, and evaluation is rarely systematic or reproducible. They test three levers for better pre-training: cleaner data, larger models, and character-level information. They gather Common Crawl, news, EL-KHAIR and Wikipedia text and clean it with heuristics, which leaves 115GB of an original 514GB. On this corpus they train three BERT-style encoders (JABER, Char-JABER, SABER) and two T5-style encoder-decoders (AT5S, AT5B). For evaluation they tune hyperparameters, repeat runs with several seeds, and submit results to the ALUE leaderboard, whose hidden test sets keep the comparison fair. They also use a reproducible subset of ARGEN for generation. Their models beat earlier Arabic models, and SABER sets a new ALUE state of the art. A further finding is that model rankings on dev sets can differ sharply from rankings on hidden test sets, so comparisons made without a leaderboard can mislead.

## Key points
- The cleaning heuristics remove gibberish, noise and duplicates and keep only about 22% of the raw data (514GB → 115GB). Common Crawl shrinks the most, from 475GB to 87GB. The authors also apply AraBERT normalization, which strips emojis, tashkeel, tatweel and HTML.
- JABER is a BERT-base model with a 64k BBPE vocabulary. It beats the strongest earlier Arabic model on ALUE by 1.5%. SABER is the BERT-large version and gives further gains.
- Char-JABER builds a vector for each word from its characters using a small CNN. This vector is added to the sub-token embeddings at the input layer only, with very few extra parameters.
- On generation tasks (QG, TS, QA, EMD), AT5B outperforms AraT5-base, and the smaller AT5S sometimes does too. The T5 models nevertheless trail the BERT models on ALUE classification.
- In the human evaluation, four native-speaker annotators each judged 100 outputs per model per task, and AT5B was preferred in every case. On EMD, 53% of AT5B outputs were rated acceptable even though its BLEU score was only 2.0.

## Interesting ideas
- Filtering pre-training data hard, keeping roughly a fifth of it, beat competitors trained on lightly cleaned corpora of similar or larger size; a smaller, cleaner corpus can be worth more than a bigger, noisy one. — For Mova (learn): Silver trees agreed by the witness parsers could be pruned aggressively with simple heuristics before training, and we could check whether the smaller set trains a better parser.
- Model rankings on dev sets did not match rankings on hidden test sets, so the authors tuned hyperparameters, ran several seeds and reported on held-out data. — For Mova (learn): Keep a frozen held-out test split that is never used for tuning, and report scores as means over seeds, so that parser or reader comparisons are not artifacts of dev-set overfitting.
- Character information can be added cheaply: a small character CNN builds a vector per word, which is combined with subword embeddings only at the input layer, so the rest of the network stays unchanged. — For Mova (tokens): A per-word character encoder feeding the tagger, lemmatizer, or the input of gparse could help with rare words, misspellings and morphology, at almost no extra cost.
- The human evaluation asks two questions: whether each output is acceptable on its own, and which of several shuffled, anonymized outputs is best. It revealed quality that n-gram metrics missed. — For Mova (back): Generated text and compressor round-trips could be judged with this acceptable/best protocol instead of BLEU alone, because a paraphrase that keeps the meaning may share few words with the reference.
- The authors trace weak generation results partly to datasets machine-translated from English, and they argue for data written natively in the target language. — For Mova (answer): Test sets for the school-course and fable QA readers should come from texts originally written in English, not from automatic translation, so their errors reflect real language.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

## Cited by

Cited by

4 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 2 not yet.

- [Arabic Stable LM: Adapting Stable LM 2 1.6B to Arabic](https://md.research.elamur.ai/free/en/papers/alyafeai-2024-arabic-stable-lm-adapting-stable/)
- [On the importance of Data Scale in Pretraining Arabic Language Models](https://md.research.elamur.ai/free/en/papers/ghaddar-2024-importance-data-scale-pretraining-arabic/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ghaddar2022revisiting,
  title = {{Revisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Understanding}},
  author = {Ghaddar, Abbas and Wu, Yimeng and Bagga, Sunyam and Rashid, Ahmad and Bibi, Khalil and Rezagholizadeh, Mehdi and Xing, Chao and Wang, Yasheng and Xinyu, Duan and Wang, Zhefeng and Huai, Baoxing and Jiang, Xin and Liu, Qun and Langlais, Philippe},
  year = {2022},
  doi = {10.48550/arxiv.2205.10687},
  eprint = {2205.10687},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2205.10687}
}
```

- [← previous Revisiting "verbal agreement": The case of Israeli Hebrew](https://md.research.elamur.ai/free/en/papers/lingbulk-2022-shor-revisiting-verbal-agreement-case-israeli/)
- [next → Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation](https://md.research.elamur.ai/free/en/papers/liu-2022-revisiting-gold-standard-grounding-summarization/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

73 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 2021. Moving towards reproducible machine learning. Nature Computational Science. [find](https://scholar.google.com/scholar?q=2021%2E+Moving+towards+reproducible+machine+learning%2E+Nature+Computational+Science%2E)
2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), [https://www.mindspore.cn/pages](https://www.mindspore.cn/pages) 7088–7105, Online. Association for Computational Linguistics. [↗](https://www.mindspore.cn/pages) [our card](https://md.research.elamur.ai/free/en/papers/abdul-2021-mageed-arbert-marbert-deep-bidirectional-transformers/)
3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. [find](https://scholar.google.com/scholar?q=Wissam+Antoun%2C+Fady+Baly%2C+and+Hazem+Hajj%2E+2020%2E+Arabert%3A+Transformer%2Dbased+model+for+arabic+language+understanding%2E+In+LREC+2020+Workshop+Language+Resources+and+Evaluation+Conference+11%E2%80%9316+May+2020%2C+p) [our card](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/)
4. Giuseppe Attardi. 2012. WikiExtractor. [find](https://scholar.google.com/scholar?q=Giuseppe+Attardi%2E+2012%2E+WikiExtractor%2E)
5. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. [find](https://scholar.google.com/scholar?q=Iz+Beltagy%2C+Matthew+E%2E+Peters%2C+and+Arman+Cohan%2E+2020%2E+Longformer%3A+The+long%2Ddocument+transformer%2E)
6. Yassine Benajiba and Paolo Rosso. 2007. Anersys 2.0: Conquering the ner task for the arabic language by combining the maximum entropy with pos-tag information. In IICAI, pages 1814–1823. [find](https://scholar.google.com/scholar?q=Yassine+Benajiba+and+Paolo+Rosso%2E+2007%2E+Anersys+2%2E0%3A+Conquering+the+ner+task+for+the+arabic+language+by+combining+the+maximum+entropy+with+pos%2Dtag+information%2E+In+IICAI%2C+pages+1814%E2%80%931823%2E)
7. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165. [↗](https://arxiv.org/abs/2005.14165)
8. José Canete, Gabriel Chaperon, Rodrigo Fuentes, and Jorge Pérez. 2020. Spanish pre-trained bert model and evaluation data. PML4DC at ICLR, 2020. [find](https://scholar.google.com/scholar?q=Jos%C3%A9+Canete%2C+Gabriel+Chaperon%2C+Rodrigo+Fuentes%2C+and+Jorge+P%C3%A9rez%2E+2020%2E+Spanish+pre%2Dtrained+bert+model+and+evaluation+data%2E+PML4DC+at+ICLR%2C+2020%2E)
9. Rishav Chakravarti, Anthony Ferritto, Bhavani Iyer, Lin Pan, Radu Florian, Salim Roukos, and Avi Sil. 2020. Towards building a robust industry-scale question answering system. In Proceedings of the 28th International Conference on Computational Linguistics: Industry Track, pages 90–101, Online. International Committee on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rishav+Chakravarti%2C+Anthony+Ferritto%2C+Bhavani+Iyer%2C+Lin+Pan%2C+Radu+Florian%2C+Salim+Roukos%2C+and+Avi+Sil%2E+2020%2E+Towards+building+a+robust+industry%2Dscale+question+answering+system%2E+In+Proceedings+of+the+28)
10. Branden Chan, Stefan Schweter, and Timo Möller. 2020. German’s next language model. arXiv preprint arXiv:2010.10906. [↗](https://arxiv.org/abs/2010.10906) [our card](https://md.research.elamur.ai/free/en/papers/chan-2020-german-s-next-language-model/)
11. Yanran Chen, Jonas Belouadi, and Steffen Eger. 2022. Reproducibility issues for bert-based evaluation metrics. arXiv preprint arXiv:2204.00004. [↗](https://arxiv.org/abs/2204.00004)
12. Jason PC Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional lstm-cnns. In Proceedings of the 54st Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jason+PC+Chiu+and+Eric+Nichols%2E+2016%2E+Named+entity+recognition+with+bidirectional+lstm%2Dcnns%2E+In+Proceedings+of+the+54st+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2E) [our card](https://md.research.elamur.ai/free/en/papers/chiu-2015-named-entity-recognition-bidirectional-lstm/)
13. Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller. 2021. Rethinking attention with performers. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Krzysztof+Marcin+Choromanski%2C+Valerii+Likhosherstov%2C+David+Dohan%2C+Xingyou+Song%2C+Andreea+Gane%2C+Tamas+Sarlos%2C+Peter+Hawkins%2C+Jared+Quincy+Davis%2C+Afroz+Mohiuddin%2C+Lukasz+Kaiser%2C+David+Benjamin+Belanger%2C+)
14. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. [↗](https://arxiv.org/abs/2204.02311)
15. Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. 2021. CANINE: Pre-training an efficient tokenization-free encoder for language representation. [find](https://scholar.google.com/scholar?q=Jonathan+H%2E+Clark%2C+Dan+Garrette%2C+Iulia+Turc%2C+and+John+Wieting%2E+2021%2E+CANINE%3A+Pre%2Dtraining+an+efficient+tokenization%2Dfree+encoder+for+language+representation%2E)
16. Jonathan H Clark, Dan Garrette, Iulia Turc, and John Wieting. 2022. Canine: Pre-training an efficient tokenization-free encoder for language representation. Transactions of the Association for Computational Linguistics, 10:73–91. [find](https://scholar.google.com/scholar?q=Jonathan+H+Clark%2C+Dan+Garrette%2C+Iulia+Turc%2C+and+John+Wieting%2E+2022%2E+Canine%3A+Pre%2Dtraining+an+efficient+tokenization%2Dfree+encoder+for+language+representation%2E+Transactions+of+the+Association+for+Computa)
17. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. [↗](https://doi.org/10.18653/v1/n19-1423)
18. Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C Wallace. 2020. Eraser: A benchmark to evaluate rationalized nlp models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4443–4458. [find](https://scholar.google.com/scholar?q=Jay+DeYoung%2C+Sarthak+Jain%2C+Nazneen+Fatema+Rajani%2C+Eric+Lehman%2C+Caiming+Xiong%2C+Richard+Socher%2C+and+Byron+C+Wallace%2E+2020%2E+Eraser%3A+A+benchmark+to+evaluate+rationalized+nlp+models%2E+In+Proceedings+of+the+) [our card](https://md.research.elamur.ai/free/en/papers/deyoung-2020-eraser-benchmark-evaluate-rationalized-nlp/)
19. Luca Di Liello, Matteo Gabburo, and Alessandro Moschitti. 2021. Efficient pre-training objectives for transformers. [find](https://scholar.google.com/scholar?q=Luca+Di+Liello%2C+Matteo+Gabburo%2C+and+Alessandro+Moschitti%2E+2021%2E+Efficient+pre%2Dtraining+objectives+for+transformers%2E)
20. Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. 2021. Glam: Efficient scaling of language models with mixture-of-experts. [find](https://scholar.google.com/scholar?q=Nan+Du%2C+Yanping+Huang%2C+Andrew+M%2E+Dai%2C+Simon+Tong%2C+Dmitry+Lepikhin%2C+Yuanzhong+Xu%2C+Maxim+Krikun%2C+Yanqi+Zhou%2C+Adams+Wei+Yu%2C+Orhan+Firat%2C+Barret+Zoph%2C+Liam+Fedus%2C+Maarten+Bosma%2C+Zongwei+Zhou%2C+Tao+Wang%2C+Yu) [our card](https://md.research.elamur.ai/free/en/papers/du-2021-glam-efficient-scaling-language-models/)
21. Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, and Jun’ichi Tsujii. 2020. CharacterBERT: Reconciling ELMo and BERT for word-level open-vocabulary representations from characters. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6903–6915, Barcelona, Spain (Online). International Committee on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Hicham+El+Boukkouri%2C+Olivier+Ferret%2C+Thomas+Lavergne%2C+Hiroshi+Noji%2C+Pierre+Zweigenbaum%2C+and+Jun%E2%80%99ichi+Tsujii%2E+2020%2E+CharacterBERT%3A+Reconciling+ELMo+and+BERT+for+word%2Dlevel+open%2Dvocabulary+representatio)
22. Ibrahim Abu El-Khair. 2016. 1.5 billion words Arabic Corpus. arXiv preprint arXiv:1611.04033. [↗](https://arxiv.org/abs/1611.04033)
23. Kawin Ethayarajh and Dan Jurafsky. 2020. Utility is in the eye of the user: A critique of nlp leaderboards. arXiv preprint arXiv:2009.13888. [↗](https://arxiv.org/abs/2009.13888)
24. Daniela Gerz, Ivan Vulić, Edoardo Maria Ponti, Roi Reichart, and Anna Korhonen. 2018. On the relation between linguistic typology and (limitations of) multilingual language modeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 316–327. [find](https://scholar.google.com/scholar?q=Daniela+Gerz%2C+Ivan+Vulic%CC%81%2C+Edoardo+Maria+Ponti%2C+Roi+Reichart%2C+and+Anna+Korhonen%2E+2018%2E+On+the+relation+between+linguistic+typology+and+%28limitations+of%29+multilingual+language+modeling%2E+In+Proceedings+o)
25. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556. [↗](https://arxiv.org/abs/2203.15556)
26. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104. [find](https://scholar.google.com/scholar?q=Go+Inoue%2C+Bashar+Alhafni%2C+Nurpeiis+Baimukan%2C+Houda+Bouamor%2C+and+Nizar+Habash%2E+2021%2E+The+interplay+of+variant%2C+size%2C+and+task+type+in+arabic+pre%2Dtrained+language+models%2E+In+Proceedings+of+the+Sixth+Ara) [our card](https://md.research.elamur.ai/free/en/papers/inoue-2021-interplay-variant-size-task-type/)
27. Katikapalli Subramanyam Kalyan, Ajit Rajasekharan, and Sivanesan Sangeetha. 2021. AMMUS: A survey of transformer-based pretrained models in natural language processing. [find](https://scholar.google.com/scholar?q=Katikapalli+Subramanyam+Kalyan%2C+Ajit+Rajasekharan%2C+and+Sivanesan+Sangeetha%2E+2021%2E+AMMUS%3A+A+survey+of+transformer%2Dbased+pretrained+models+in+natural+language+processing%2E)
28. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. 2021. Dynabench: Rethinking benchmarking in nlp. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4110–4124. [find](https://scholar.google.com/scholar?q=Douwe+Kiela%2C+Max+Bartolo%2C+Yixin+Nie%2C+Divyansh+Kaushik%2C+Atticus+Geiger%2C+Zhengxuan+Wu%2C+Bertie+Vidgen%2C+Grusha+Prasad%2C+Amanpreet+Singh%2C+Pratik+Ringshia%2C+et+al%2E+2021%2E+Dynabench%3A+Rethinking+benchmarking+in+)
29. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016. Character-aware neural language models. In Thirtieth AAAI conference on artificial intelligence. [find](https://scholar.google.com/scholar?q=Yoon+Kim%2C+Yacine+Jernite%2C+David+Sontag%2C+and+Alexander+M+Rush%2E+2016%2E+Character%2Daware+neural+language+models%2E+In+Thirtieth+AAAI+conference+on+artificial+intelligence%2E) [our card](https://md.research.elamur.ai/free/en/papers/kim-2015-character-aware-neural-language-models/)
30. Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. 2020. WikiLingua: A new benchmark dataset for cross-lingual abstractive summarization. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4034– 4048, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Faisal+Ladhak%2C+Esin+Durmus%2C+Claire+Cardie%2C+and+Kathleen+McKeown%2E+2020%2E+WikiLingua%3A+A+new+benchmark+dataset+for+cross%2Dlingual+abstractive+summarization%2E+In+Findings+of+the+Association+for+Computational) [our card](https://md.research.elamur.ai/free/en/papers/ladhak-2020-wikilingua-new-benchmark-dataset-cross/)
31. Wuwei Lan, Yang Chen, Wei Xu, and Alan Ritter. 2020. An empirical study of pre-trained transformers for arabic information extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4727–4734. [find](https://scholar.google.com/scholar?q=Wuwei+Lan%2C+Yang+Chen%2C+Wei+Xu%2C+and+Alan+Ritter%2E+2020%2E+An+empirical+study+of+pre%2Dtrained+transformers+for+arabic+information+extraction%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Nat)
32. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. [↗](https://arxiv.org/abs/1909.11942)
33. Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoit Crabbé, Laurent Besacier, and Didier Schwab. 2020. Flaubert: Unsupervised language model pre-training for french. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 2479–2490. [find](https://scholar.google.com/scholar?q=Hang+Le%2C+Lo%C3%AFc+Vial%2C+Jibril+Frej%2C+Vincent+Segonne%2C+Maximin+Coavoux%2C+Benjamin+Lecouteux%2C+Alexandre+Allauzen%2C+Benoit+Crabb%C3%A9%2C+Laurent+Besacier%2C+and+Didier+Schwab%2E+2020%2E+Flaubert%3A+Unsupervised+language+mod) [our card](https://md.research.elamur.ai/free/en/papers/le-2019-flaubert-unsupervised-language-model-pre/)
34. Chanhee Lee, Young-Bum Kim, Dongyub Lee, and Heui-Seok Lim. 2018. Character-level feature extraction with densely connected networks. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3228–3239. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. [link]. [find](https://scholar.google.com/scholar?q=Chanhee+Lee%2C+Young%2DBum+Kim%2C+Dongyub+Lee%2C+and+Heui%2DSeok+Lim%2E+2018%2E+Character%2Dlevel+feature+extraction+with+densely+connected+networks%2E+In+Proceedings+of+the+27th+International+Conference+on+Computation)
35. Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, et al. 2021. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 175–184. [find](https://scholar.google.com/scholar?q=Quentin+Lhoest%2C+Albert+Villanova+del+Moral%2C+Yacine+Jernite%2C+Abhishek+Thakur%2C+Patrick+von+Platen%2C+Suraj+Patil%2C+Julien+Chaumond%2C+Mariama+Drame%2C+Julien+Plu%2C+Lewis+Tunstall%2C+et+al%2E+2021%2E+Datasets%3A+A+commu) [our card](https://md.research.elamur.ai/free/en/papers/lhoest-2021-datasets-community-library-natural-language/)
36. Guangjun Li, Xianzhi Wang, and Minxi Li. 2021. A review of recent trends and industry prospects for artificial intelligence technologies. In 2021 8th International Conference on Behavioral and Social Computing (BESC), pages 1–7. IEEE. [find](https://scholar.google.com/scholar?q=Guangjun+Li%2C+Xianzhi+Wang%2C+and+Minxi+Li%2E+2021%2E+A+review+of+recent+trends+and+industry+prospects+for+artificial+intelligence+technologies%2E+In+2021+8th+International+Conference+on+Behavioral+and+Social+)
37. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+2004%2E+Rouge%3A+A+package+for+automatic+evaluation+of+summaries%2E+In+Text+summarization+branches+out%2C+pages+74%E2%80%9381%2E)
38. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [↗](https://arxiv.org/abs/1907.11692)
39. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. [↗](https://arxiv.org/abs/1711.05101)
40. Wentao Ma, Yiming Cui, Chenglei Si, Ting Liu, Shijin Wang, and Guoping Hu. 2020. CharBERT: Character-aware pre-trained language model. In Proceedings of the 28th International Conference on Computational Linguistics, pages 39–50, Barcelona, Spain (Online). International Committee on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wentao+Ma%2C+Yiming+Cui%2C+Chenglei+Si%2C+Ting+Liu%2C+Shijin+Wang%2C+and+Guoping+Hu%2E+2020%2E+CharBERT%3A+Character%2Daware+pre%2Dtrained+language+model%2E+In+Proceedings+of+the+28th+International+Conference+on+Computatio)
41. Udo Kruschwitz Mahmoud El-Haj and Chris Fox. 2010. Using mechanical turk to create a corpus of Arabic summaries. [find](https://scholar.google.com/scholar?q=Udo+Kruschwitz+Mahmoud+El%2DHaj+and+Chris+Fox%2E+2010%2E+Using+mechanical+turk+to+create+a+corpus+of+Arabic+summaries%2E)
42. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah, and Benoît Sagot. 2019. Camembert: a tasty french language model. arXiv preprint arXiv:1911.03894. [↗](https://arxiv.org/abs/1911.03894) [our card](https://md.research.elamur.ai/free/en/papers/martin-2019-camembert-tasty-french-language-model/)
43. S. Mehri, M. Eric, and D. Hakkani-Tur. 2020. Dialoglue: A natural language understanding benchmark for task-oriented dialogue. ArXiv, abs/2009.13570. [↗](https://arxiv.org/abs/2009.13570)
44. Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heinz, and Dan Roth. 2021. Recent advances in natural language processing via large pre-trained language models: A survey. arXiv preprint arXiv:2111.01243. [↗](https://arxiv.org/abs/2111.01243) [our card](https://md.research.elamur.ai/free/en/papers/min-2021-recent-advances-natural-language-processing/)
45. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022. AraT5: Text-to-text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=El+Moatez+Billah+Nagoudi%2C+AbdelRahim+Elmadany%2C+and+Muhammad+Abdul%2DMageed%2E+2022%2E+AraT5%3A+Text%2Dto%2Dtext+transformers+for+Arabic+language+generation%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Associ) [our card](https://md.research.elamur.ai/free/en/papers/nagoudi-2022-arat5-text-text-transformers-arabic/)
46. El Moatez Billah Nagoudi, AbdelRahim Elmadany, Muhammad Abdul-Mageed, and Tariq Alhindi. 2020. Machine generation and detection of Arabic manipulated and fake news. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 69–84, Barcelona, Spain (Online). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=El+Moatez+Billah+Nagoudi%2C+AbdelRahim+Elmadany%2C+Muhammad+Abdul%2DMageed%2C+and+Tariq+Alhindi%2E+2020%2E+Machine+generation+and+detection+of+Arabic+manipulated+and+fake+news%2E+In+Proceedings+of+the+Fifth+Arabic+)
47. Tarek Naous, Wissam Antoun, Reem Mahmoud, and Hazem Hajj. 2021a. Empathetic bert2bert conversational model: Learning arabic language generation with little data. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 164– 172. [find](https://scholar.google.com/scholar?q=Tarek+Naous%2C+Wissam+Antoun%2C+Reem+Mahmoud%2C+and+Hazem+Hajj%2E+2021a%2E+Empathetic+bert2bert+conversational+model%3A+Learning+arabic+language+generation+with+little+data%2E+In+Proceedings+of+the+Sixth+Arabic+Nat)
48. Tarek Naous, Wissam Antoun, Reem Mahmoud, and Hazem Hajj. 2021b. Empathetic BERT2BERT conversational model: Learning Arabic language generation with little data. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 164–172, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tarek+Naous%2C+Wissam+Antoun%2C+Reem+Mahmoud%2C+and+Hazem+Hajj%2E+2021b%2E+Empathetic+BERT2BERT+conversational+model%3A+Learning+Arabic+language+generation+with+little+data%2E+In+Proceedings+of+the+Sixth+Arabic+Nat)
49. Tarek Naous, Christian Hokayem, and Hazem Hajj. 2020. Empathy-driven Arabic conversational chatbot. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 58–68, Barcelona, Spain (Online). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tarek+Naous%2C+Christian+Hokayem%2C+and+Hazem+Hajj%2E+2020%2E+Empathy%2Ddriven+Arabic+conversational+chatbot%2E+In+Proceedings+of+the+Fifth+Arabic+Natural+Language+Processing+Workshop%2C+pages+58%E2%80%9368%2C+Barcelona%2C+Spa)
50. Subhadarshi Panda, Anjali Agrawal, Jeewon Ha, and Benjamin Bloch. 2021. Shuffled-token detection for refining pre-trained RoBERTa. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 88–93, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Subhadarshi+Panda%2C+Anjali+Agrawal%2C+Jeewon+Ha%2C+and+Benjamin+Bloch%2E+2021%2E+Shuffled%2Dtoken+detection+for+refining+pre%2Dtrained+RoBERTa%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+o)
51. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318. [↗](https://doi.org/10.3115/1073083.1073135) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
52. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026– 8037. [find](https://scholar.google.com/scholar?q=Adam+Paszke%2C+Sam+Gross%2C+Francisco+Massa%2C+Adam+Lerer%2C+James+Bradbury%2C+Gregory+Chanan%2C+Trevor+Killeen%2C+Zeming+Lin%2C+Natalia+Gimelshein%2C+Luca+Antiga%2C+et+al%2E+2019%2E+Pytorch%3A+An+imperative+style%2C+high%2Dperfor) [our card](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/)
53. Joelle Pineau. 2020. Machine learning reproducibility checklist v2.0. [find](https://scholar.google.com/scholar?q=Joelle+Pineau%2E+2020%2E+Machine+learning+reproducibility+checklist+v2%2E0%2E)
54. Yuval Pinter, Amanda Stent, Mark Dredze, and Jacob Eisenstein. 2021. Learning to look inside: Augmenting token-based encoders with character-level information. [find](https://scholar.google.com/scholar?q=Yuval+Pinter%2C+Amanda+Stent%2C+Mark+Dredze%2C+and+Jacob+Eisenstein%2E+2021%2E+Learning+to+look+inside%3A+Augmenting+token%2Dbased+encoders+with+character%2Dlevel+information%2E)
55. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Joint Conference on EMNLP and CoNLL-Shared Task, pages 1– 40. [find](https://scholar.google.com/scholar?q=Sameer+Pradhan%2C+Alessandro+Moschitti%2C+Nianwen+Xue%2C+Olga+Uryupina%2C+and+Yuchen+Zhang%2E+2012%2E+Conll%2D2012+shared+task%3A+Modeling+multilingual+unrestricted+coreference+in+ontonotes%2E+In+Joint+Conference+on+EM) [our card](https://md.research.elamur.ai/free/en/papers/pradhan-2012-conll-2012-shared-task-modeling/)
56. Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, et al. 2021. Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. arXiv preprint arXiv:2107.02137. [↗](https://arxiv.org/abs/2107.02137) [our card](https://md.research.elamur.ai/free/en/papers/sun-2021-ernie-3-0-large/)
57. Alec Radford, Karthik Narasimhan, Tim Salimans, Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi and Ilya Sutskever. 2018. Improving language Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao understanding by generative pre-training. URL Tian, and Hua Wu. 2019. Ernie: Enhanced rep-https://s3-us-west-2.amazonaws. com/openai-resentation through knowledge integration. arXiv assets/researchcovers/languageunsupervised/language preprint arXiv:1904.09223. understanding paper. pdf. [↗](https://arxiv.org/abs/1904.09223) [our card](https://md.research.elamur.ai/free/en/papers/sun-2019-ernie-enhanced-representation-through-knowledge/)
58. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Tian, Hua Wu, and Haifeng Wang. 2020. Ernie 2.0: Lee, Sharan Narang, Michael Matena, Yanqi Zhou, A continual pre-training framework for language un-Wei Li, and Peter J Liu. 2019. Exploring the limits derstanding. In Proceedings of the AAAI Conference of transfer learning with a unified text-to-text trans-on Artificial Intelligence, volume 34, pages 8968– former. arXiv preprint arXiv:1910.10683. 8975. [↗](https://arxiv.org/abs/1910.10683)
59. Hannah Rashkin, Eric Michael Smith, Margaret Li, and Bashar Talafha, Mohammad Ali, Muhy Eddin Za’ter, Y-Lan Boureau. 2019. Towards empathetic open-Haitham Seelawi, Ibraheem Tuffaha, Mostafa Samir, domain conversation models: A new benchmark and Wael Farhan, and Hussein Al-Natsheh. 2020. Multi-dataset. In Proceedings of the 57th Annual Meet-dialect arabic bert for country-level dialect identifi-ing of the Association for Computational Linguistics, cation. In Proceedings of the Fifth Arabic Natural pages 5370–5381. Language Processing Workshop, pages 111–118. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. [find](https://scholar.google.com/scholar?q=Hannah+Rashkin%2C+Eric+Michael+Smith%2C+Margaret+Li%2C+and+Bashar+Talafha%2C+Mohammad+Ali%2C+Muhy+Eddin+Za%E2%80%99ter%2C+Y%2DLan+Boureau%2E+2019%2E+Towards+empathetic+open%2DHaitham+Seelawi%2C+Ibraheem+Tuffaha%2C+Mostafa+Samir%2C+dom)
60. Lewis Tunstall, Leandro von Werra, and Thomas Wolf. 2020. Kuisail at semeval-2020 task 12: Bert-cnn for 2022. Natural Language Processing with Trans-offensive speech identification in social media. In formers. " O’Reilly Media, Inc.". Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059. [find](https://scholar.google.com/scholar?q=Lewis+Tunstall%2C+Leandro+von+Werra%2C+and+Thomas+Wolf%2E+2020%2E+Kuisail+at+semeval%2D2020+task+12%3A+Bert%2Dcnn+for+2022%2E+Natural+Language+Processing+with+Trans%2Doffensive+speech+identification+in+social+media%2E+In)
61. Matej Ulčar and Marko Robnik-Šikonja. 2020. Finest Ananya B. Sai, Akash Kumar Mohankumar, and bert and crosloengual bert: less is more in multilin-Mitesh M. Khapra. 2022. A survey of evaluation gual models. arXiv preprint arXiv:2006.07890. metrics used for nlg systems. ACM Comput. Surv., Elena Voita, David Talbot, Fedor Moiseev, Rico Sen-55(2). nrich, and Ivan Titov. 2019. Analyzing multi-head Haitham Seelawi, Ibraheem Tuffaha, Mahmoud Gzawi, self-attention: Specialized heads do the heavy lift-Wael Farhan, Bashar Talafha, Riham Badawi, Zyad ing, the rest can be pruned. In Proceedings of the Sober, Oday Al-Dweik, Abed Alhakim Freihat, and 57th Annual Meeting of the Association for Com-Hussein Al-Natsheh. 2021. Alue: Arabic language putational Linguistics, pages 5797–5808, Florence, understanding evaluation. In Proceedings of the Italy. Association for Computational Linguistics. Sixth Arabic Natural Language Processing Workshop, pages 173–184. [↗](https://arxiv.org/abs/2006.07890)
62. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Alexander Sergeev and Mike Del Balso. 2018. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Yada+Pruksachatkun%2C+Nikita+Nangia%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Alexander+Sergeev+and+Mike+Del+Balso%2E+2018%2E)
63. Levy, and Samuel R Bowman. 2019. Super-Horovod: fast and easy distributed deep learning in GLUE: A stickier benchmark for general-purpose tensorflow. arXiv preprint arXiv:1802.05799. language understanding systems. arXiv preprint arXiv:1905.00537. Tatiana Shavrina, Alena Fenogenova, Anton Emelyanov, Denis Shevelev, Ekaterina Arte-Alex Wang, Amanpreet Singh, Julian Michael, Felix mova, Valentin Malykh, Vladislav Mikhailov, Maria Hill, Omer Levy, and Samuel Bowman. 2018. Glue: Tikhonova, Andrey Chertok, and Andrey Evlampiev. A multi-task benchmark and analysis platform for 2020. Russiansuperglue: A russian language unnatural language understanding. In Proceedings derstanding evaluation benchmark. arXiv preprint of the 2018 EMNLP Workshop BlackboxNLP: An-arXiv:2010.15925. alyzing and Interpreting Neural Networks for NLP, pages 353–355. [↗](https://arxiv.org/abs/1802.05799)
64. Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. Junqiu Wei, Qun Liu, Yinpeng Guo, and Xin Jiang. [find](https://scholar.google.com/scholar?q=Noam+Shazeer+and+Mitchell+Stern%2E+2018%2E+Adafactor%3A+Adaptive+learning+rates+with+sublinear+memory+cost%2E+Junqiu+Wei%2C+Qun+Liu%2C+Yinpeng+Guo%2C+and+Xin+Jiang%2E)
65. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent 2021. Training multilingual pre-trained language Romary. 2019. Asynchronous pipeline for process-model with byte-level subwords. arXiv preprint ing huge corpora on medium to low resource infras-arXiv:2101.09469. tructures. In 7th Workshop on the Challenges in the Junqiu Wei, Xiaozhe Ren, Xiaoguang Li, Weny-Management of Large Corpora (CMLC-7). Leibniz-ong Huang, Yi Liao, Yasheng Wang, Jiashu Institut für Deutsche Sprache. [↗](https://arxiv.org/abs/2101.09469)
66. Lin, Xin Jiang, Xiao Chen, and Qun Liu. 2019. Nezha: Neural contextualized representation for chinese language understanding. arXiv preprint arXiv:1909.00204. [↗](https://arxiv.org/abs/1909.00204)
67. Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, X. Li, Zhi Yuan Lim, S. Soleman, R. Mahendra, Pascale Fung, Syafri Bahar, and A. Purwarianti. 2020. Indonlu: Benchmark and resources for evaluating indonesian natural language understanding. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing. [find](https://scholar.google.com/scholar?q=Bryan+Wilie%2C+Karissa+Vincentio%2C+Genta+Indra+Winata%2C+Samuel+Cahyawijaya%2C+X%2E+Li%2C+Zhi+Yuan+Lim%2C+S%2E+Soleman%2C+R%2E+Mahendra%2C+Pascale+Fung%2C+Syafri+Bahar%2C+and+A%2E+Purwarianti%2E+2020%2E+Indonlu%3A+Benchmark+and+resou) [our card](https://md.research.elamur.ai/free/en/papers/wilie-2020-indonlu-benchmark-resources-evaluating-indonesian/)
68. Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pier-ric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45. [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Julien+Chaumond%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pier%2Dric+Cistac%2C+Morgan+Funtowicz%2C+Joe+Davison%2C+Sam+Shleifer%2C+et+al%2E+2020%2E+Transformers%3A+State%2Dof%2Dthe%2Dart+natu) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
69. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. [↗](https://arxiv.org/abs/1609.08144)
70. Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al. 2020. Clue: A chinese language understanding evaluation benchmark. arXiv preprint arXiv:2004.05986. [↗](https://arxiv.org/abs/2004.05986) [our card](https://md.research.elamur.ai/free/en/papers/xu-2020-clue-chinese-language-understanding/)
71. Liang Xu, Xiaojing Lu, Chenyang Yuan, Xuanwei Zhang, Hu Yuan, Huilin Xu, Guoao Wei, Xiang Pan, and Hai Hu. 2021. Fewclue: A chinese few-shot learning evaluation benchmark. arXiv preprint arXiv:2107.07498. [↗](https://arxiv.org/abs/2107.07498)
72. Wei Zeng, Xiaozhe Ren, Teng Su, Hui Wang, Yi Liao, Zhiwei Wang, Xin Jiang, ZhenZhang Yang, Kaisheng Wang, Xiaoda Zhang, et al. 2021. Pangu-α: Large-scale autoregressive pretrained chinese language models with auto-parallel computation. arXiv preprint arXiv:2104.12369. [↗](https://arxiv.org/abs/2104.12369) [our card](https://md.research.elamur.ai/free/en/papers/zeng-2021-pangu-alpha-large-scale/)
73. Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and Abdelhak Lakhouaja. 2019. Osian: Open source international arabic news corpus-preparation and integration into the clarin-infrastructure. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 175–182. [find](https://scholar.google.com/scholar?q=Imad+Zeroual%2C+Dirk+Goldhahn%2C+Thomas+Eckart%2C+and+Abdelhak+Lakhouaja%2E+2019%2E+Osian%3A+Open+source+international+arabic+news+corpus%2Dpreparation+and+integration+into+the+clarin%2Dinfrastructure%2E+In+Proceedings)