---
title: "ORCA — paper by Elmadany et al. (2023) — summary and ideas"
description: "The authors build ORCA from 60 publicly available datasets, organised into 29 tasks and seven task… 98 references, cited by 53 papers here."
authors: ["AbdelRahim Elmadany", "ElMoatez Billah Nagoudi", "Muhammad Abdul-Mageed"]
year: 2023
venue: "Findings of the Association for Computational Linguistics: ACL 2023"
source: "https://aclanthology.org/2023.findings-acl.609/"
doi: "10.18653/v1/2023.findings-acl.609"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/elmadany-2023-orca-challenging-benchmark-arabic-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/elmadany-2023-orca-challenging-benchmark-arabic-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / ORCA: A Challenging Benchmark for Arabic Language Understanding

# ORCA: A Challenging Benchmark for Arabic Language Understanding

**Authors:** [AbdelRahim Elmadany](https://md.research.elamur.ai/free/en/authors/abdelrahim-elmadany/), ElMoatez Billah Nagoudi and [Muhammad Abdul-Mageed](https://md.research.elamur.ai/free/en/authors/muhammad-abdul-mageed/) · **Year:** 2023 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2023](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2023](https://md.research.elamur.ai/free/en/venues/findings/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.findings-acl.609/) · [PDF](https://aclanthology.org/2023.findings-acl.609.pdf) · [DOI 10.18653/v1/2023.findings-acl.609](https://doi.org/10.18653/v1/2023.findings-acl.609) · [OpenAlex](https://openalex.org/W4385572487)

**Citations:** 30 citations (OpenAlex, 2026-10-14) · [53](https://md.research.elamur.ai/free/en/papers/elmadany-2023-orca-challenging-benchmark-arabic-language/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Arabic NLU has lacked a broad, fully public benchmark. That gap makes it hard to compare Arabic-specific and multilingual pretrained models. The problem is sharper because "Arabic" covers Modern Standard Arabic and many regional dialects. Earlier efforts each fell short: ALUE is small, and ARLUE depends partly on non-public data and has no leaderboard. The authors build ORCA from 60 publicly available datasets, organised into 29 tasks and seven task clusters:

- sentence classification
- structured prediction
- topic classification
- semantic similarity
- NLI
- QA
- word sense disambiguation

They evaluate 18 multilingual and Arabic language models on it and combine results into a single ORCA score. They also release an interactive leaderboard with dataset metadata and a modular finetuning toolkit. In addition, they compare how long each model takes to converge. The results show the benchmark remains difficult for current models. ORCA's value lies in giving the Arabic NLP community a shared, reproducible yardstick that spans both standard and dialectal varieties.

## Key points
- ORCA has 60 datasets, 29 tasks and 7 clusters (SC, SP, TC, STS, NLI, QA, WSD).
  - The splits total about 487.1K train, 46.0K dev and 55.1K test examples.
  - Sentiment analysis alone draws on 19 datasets and social meaning on 11.
- Earlier Arabic benchmarks were narrower:
  - ALUE has 8 datasets in 3 clusters.
  - ARLUE has 42 datasets in 4 clusters, but many are not public and it has no leaderboard.
- STS appears in two forms: a regression version (STS-reg) and a classification version (STS-cls). Dialect identification comes in three task variants (Dia-b, Dia-r, Dia-c).
- In the compute comparison, models were finetuned for at most 25 epochs, averaged over three runs.
  - Arabic Albert finished fastest (52.26 min) but had the lowest ORCA score.
  - Every other model took roughly 60.32–63.69 min.
  - Among the top five, ARBERT v1 converged fastest, followed by QARiB.
- The authors name these coverage gaps:
  - no data from Mauritania or Djibouti
  - thin coverage of dialects such as Iraqi, Sudanese and Yemeni
  - little Classical Arabic

## Interesting ideas
- Fold many heterogeneous datasets into one headline number while still publishing per-cluster results, so that progress is easy to track without hiding weak areas.
- Report cost to convergence (time and epochs) next to quality, which exposes models that are cheap but weak and models that are strong but cheap.
- Pose the same semantic-similarity data both as graded regression and as binary classification, testing fine and coarse judgement separately.
- State openly which varieties and regions have no data, rather than implying full coverage.
- Attach source metadata and ready-made citations to every component dataset, so credit and provenance travel with the benchmark.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Each stage could report a single aggregate health score next to its per-component breakdown, so that regressions in one module stay visible.
- **I2** (tree): When comparing the perceptron parser with gparse, log training time and epochs to convergence alongside LAS so trade-offs are explicit.
- **I3** (answer): Readers answering school tests could be scored on both exact graded answers and coarse correct/incorrect decisions to locate where reasoning breaks.
- **I4** (tree): Maintain an explicit inventory of genres and UD dialects missing from our treebanks, so that conversion and parsing gaps are documented rather than discovered late.
- **I5** (world): Each fact in the compiled knowledge layer (WordNet, FrameNet, Wikidata) could carry its origin, keeping answers explainable down to the source.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of British Columbia](https://md.research.elamur.ai/free/en/universities/university-of-british-columbia/) — University, Vancouver, Canada · 46 works · 33,673 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 744 works · [🎓 Education](https://md.research.elamur.ai/free/en/shelves/social-sciences/education/) · 112 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 685 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 1,484 works

## Cite

Cite as: AbdelRahim Elmadany, ElMoatez Billah Nagoudi and Muhammad Abdul-Mageed. 2023. ORCA: A Challenging Benchmark for Arabic Language Understanding. Findings of the Association for Computational Linguistics: ACL 2023.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{elmadany2023orca,
  title = {{ORCA: A Challenging Benchmark for Arabic Language Understanding}},
  author = {AbdelRahim Elmadany, ElMoatez Billah Nagoudi and Muhammad Abdul-Mageed},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.609},
  url = {https://doi.org/10.18653/v1/2023.findings-acl.609}
}
```

- [← previous InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback](https://md.research.elamur.ai/free/en/papers/yang-2023-intercode-standardizing-benchmarking-interactive-coding/)
- [next → Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks](https://md.research.elamur.ai/free/en/papers/shayegani-2023-survey-vulnerabilities-large-language-models/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

98 references, 52 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mourad Abbas, Kamel Smaïli, and Daoud Berkani. 2011. Evaluation of topic identification methods on arabic corpora. JDIM, 9(5):185–192. [find](https://scholar.google.com/scholar?q=Mourad+Abbas%2C+Kamel+Sma%C3%AFli%2C+and+Daoud+Berkani%2E+2011%2E+Evaluation+of+topic+identification+methods+on+arabic+corpora%2E+JDIM%2C+9%285%29%3A185%E2%80%93192%2E)
2. Ahmed Abdelali, Hamdy Mubarak, Younes Samih, Sabit Hassan, and Kareem Darwish. 2020. [Arabic Dialect Identification in the Wild](https://md.research.elamur.ai/free/en/papers/abdelali-2020-arabic-dialect-identification-wild/). Proceedings of the Sixth Arabic Natural Language Processing Workshop. [↗](https://doi.org/10.48550/arxiv.2005.06557)
3. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. [ARBERT & MARBERT: Deep bidirectional transformers for Arabic](https://md.research.elamur.ai/free/en/papers/abdul-mageed-2021-arbert-marbert-deep-bidirectional-transformers/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2021.acl-long.551)
4. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020a. [NADI 2020: The first nuanced Arabic dialect identification shared task](https://md.research.elamur.ai/free/en/papers/mageed-2020-nadi-2020-first-nuanced-arabic/). In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 97–110, Barcelona, Spain (Online). Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2010.11334)
5. Muhammad Abdul-Mageed, Chiyu Zhang, Azadeh Hashemi, and El Moatez Billah Nagoudi. 2020b. AraNet: A Deep Learning Toolkit for Arabic Social Media. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 16–23, Marseille, France. European Language Resource Association. https://alliancecan.ca https://arc.ubc.ca/ubc-arc-sockeye [↗](https://doi.org/10.48550/arxiv.1912.13072)
6. Latifah Almuqren and Alexandra Cristea. 2021. Ara-Cust: a Saudi Telecom Tweets corpus for sentiment analysis. PeerJ Computer Science, 7:e510. [find](https://scholar.google.com/scholar?q=Latifah+Almuqren+and+Alexandra+Cristea%2E+2021%2E+Ara%2DCust%3A+a+Saudi+Telecom+Tweets+corpus+for+sentiment+analysis%2E+PeerJ+Computer+Science%2C+7%3Ae510%2E)
7. Ali Alshehri, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2020. Understanding and detecting dangerous speech in social media. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 40–47, Marseille, France. European Language Resource Association. [↗](https://doi.org/10.48550/arxiv.2005.06608)
8. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. [AraBERT: Transformer-based Model for Arabic Language Understanding](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/). In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15. [↗](https://doi.org/10.48550/arxiv.2003.00104)
9. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. [AraELECTRA: Pre-Training Text Discriminators for Arabic Language Understanding](https://md.research.elamur.ai/free/en/papers/antoun-2021-araelectra-pre-training-text-discriminators/). In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 191–195. [↗](https://doi.org/10.48550/arxiv.2012.15516)
10. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. [On the Cross-lingual Transferability of Monolingual Representations](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637.
11. Ramy Baly, Mitra Mohtarami, James Glass, Lluís Màrquez, Alessandro Moschitti, and Preslav Nakov. 2018. [Integrating stance detection and fact checking in a unified corpus](https://md.research.elamur.ai/free/en/papers/baly-2018-integrating-stance-detection-fact-checking/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 21–27.
12. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. [Longformer: The long-document transformer](https://md.research.elamur.ai/free/en/papers/beltagy-2020-longformer-long-document-transformer/). ArXiv, abs/2004.05150. [↗](https://arxiv.org/abs/2004.05150)
13. Yassine Benajiba and Paolo Rosso. 2007. ANERsys 2.0: Conquering the NER Task for the Arabic Language by Combining the Maximum Entropy with POS-tag Information. In IICAI, pages 1814–1823. [find](https://scholar.google.com/scholar?q=Yassine+Benajiba+and+Paolo+Rosso%2E+2007%2E+ANERsys+2%2E0%3A+Conquering+the+NER+Task+for+the+Arabic+Language+by+Combining+the+Maximum+Entropy+with+POS%2Dtag+Information%2E+In+IICAI%2C+pages+1814%E2%80%931823%2E)
14. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. [The MADAR shared task on Arabic fine-grained dialect identification](https://md.research.elamur.ai/free/en/papers/bouamor-2019-madar-shared-task-arabic-fine/). In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 199–207. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec 9568 10
15. Radford, Ilya Sutskever, and Dario Amodei. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
16. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. [SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation](https://md.research.elamur.ai/free/en/papers/cer-2017-semeval-2017-task-1-semantic/). In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/s17-2001)
17. Amina Chouigui, Oussama Ben Khiroun, and Bilel Elayeb. 2017. ANT Corpus : An Arabic News Text Collection for Textual Classification. In 2017 IEEE/ACS 14th International Conference on Computer Systems and Applications (AICCSA), pages 135–142. IEEE. [find](https://scholar.google.com/scholar?q=Amina+Chouigui%2C+Oussama+Ben+Khiroun%2C+and+Bilel+Elayeb%2E+2017%2E+ANT+Corpus+%3A+An+Arabic+News+Text+Collection+for+Textual+Classification%2E+In+2017+IEEE%2FACS+14th+International+Conference+on+Computer+Systems+)
18. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Oliveira Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). [↗](https://doi.org/10.48550/arxiv.2204.02311)
19. Shammur Absar Chowdhury, Ahmed Abdelali, Kareem Darwish, Jung Soon-Gyo, Joni Salminen, and Bernard J. Jansen. 2020. Improving Arabic text categorization using transformer training diversification. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 226–236, Barcelona, Spain (Online). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shammur+Absar+Chowdhury%2C+Ahmed+Abdelali%2C+Kareem+Darwish%2C+Jung+Soon%2DGyo%2C+Joni+Salminen%2C+and+Bernard+J%2E+Jansen%2E+2020%2E+Improving+Arabic+text+categorization+using+transformer+training+diversification%2E+In+)
20. Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc Le, and Jason Wei. 2022. [Scaling instruction-finetuned language models](https://md.research.elamur.ai/free/en/papers/chung-2022-scaling-instruction-finetuned-language-models/). ArXiv, abs/2210.11416. [↗](https://arxiv.org/abs/2210.11416)
21. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555. [↗](https://arxiv.org/abs/2003.10555)
22. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020a. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020a%2E+Unsupervised+cross%2Dling)
23. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, et al. 2020b. Unsupervised Cross-lingual Representation Learning at Scale. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+et+al%2E+2020b%2E+Unsupervised+Cross%2Dlingual+Representation+Learning+at+Scale%2E+Proceedings+of+the+58th+Annual+Meeting)
24. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. Advances in neural information processing systems, 32. [find](https://scholar.google.com/scholar?q=Alexis+Conneau+and+Guillaume+Lample%2E+2019%2E+Cross%2Dlingual+language+model+pretraining%2E+Advances+in+neural+information+processing+systems%2C+32%2E)
25. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1809.05053)
26. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zihang+Dai%2C+Zhilin+Yang%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Quoc+Le%2C+and+Ruslan+Salakhutdinov%2E+2019%2E+Transformer%2DXL%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+In+Proceedings+of+the+57th+Annu)
27. Kareem Darwish, Hamdy Mubarak, Ahmed Abdelali, Mohamed Eldesouki, Younes Samih, Randah Alharbi, Mohammed Attia, Walid Magdy, and Laura Kallmeyer. 2018. Multi-dialect Arabic POS Tagging: A CRF approach. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). [find](https://scholar.google.com/scholar?q=Kareem+Darwish%2C+Hamdy+Mubarak%2C+Ahmed+Abdelali%2C+Mohamed+Eldesouki%2C+Younes+Samih%2C+Randah+Alharbi%2C+Mohammed+Attia%2C+Walid+Magdy%2C+and+Laura+Kallmeyer%2E+2018%2E+Multi%2Ddialect+Arabic+POS+Tagging%3A+A+CRF+approach)
28. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. Kaustubh D. Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abinaya Mahendiran, Simon Mille, Ashish Srivastava, Samson Tan, Tongshuang Sherry Wu, Jascha Narain Sohl-Dickstein, Jinho D. Choi, Eduard H. Hovy, Ondrej Dusek, Sebastian Ruder, Sajant Anand, Nagender Aneja, Rabin Banjade, Lisa 9569 11
29. Barthe, Hanna Behnke, Ian Berlot-Attwell, Connor Boyle, Caroline De Brun, Marco Antonio Sobrevilla Cabezudo, Samuel Cahyawijaya, Emile Chapuis, Wanxiang Che, Mukund Choudhary, Christian Clauss, Pierre Colombo, Filip Cornell, Gautier Dagan, M. Das, Tanay Dixit, Thomas Dopierre, Paul-Alexis Dray, Suchitra Dubey, Tatiana Ekeinhor, Marco Di Giovanni, Rishabh Gupta, Louanes Hamla, Sanghyun Han, Fabrice Harel-Canada, Antoine Honoré, Ishan Jindal, Przemyslaw K. Joniak, Denis Kleyko, Venelin Kovatchev, Kalpesh Krishna, Ashutosh Kumar, Stefan Langer, Seungjae Ryan Lee, Corey J. Levinson, H.-J. Liang, Kaizhao Liang, Zhexiong Liu, Andrey Lukyanenko, Vukosi Marivate, Gerard de Melo, Simon Meoni, Maxime Meyer, Afnan Mir, Nafise Sadat Moosavi, Niklas Muennighoff, Timothy Sum Hon Mun, Kenton W. Murray, Marcin Namysl, Maria Obedkova, Priti Oli, Nivranshu Pasricha, Jan A. Pfister, Richard Plant, Vinay Uday Prabhu, Vasile Florian Pais, Libo Qin, Shahab Raji, Pawan Rajpoot, Vikas Raunak, Roy Rinberg, Nicolas M. Roberts, Juan Diego Rodriguez, Claude Roux, S. VasconcellosP.H., Ananya B. Sai, Robin M. Schmidt, Thomas Scialom, T. Sefara, Saqib Nizam Shamsi, Xudong Shen, Haoyue Shi, Yi ning Shi, Anna V. Shvets, Nick Siegel, Damien Sileo, Jamie Simon, Chandan Singh, Roman Sitelew, Priyanka Soni, Taylor M Sorensen, William Soto, Aman Srivastava, K V Aditya Srivatsa, Tony Sun, T MukundVarma, Afsheen Tabassum, Fiona Anting Tan, Ryan Teehan, Monalisa Tiwari, Marie Tolkiehn, Athena Wang, Zi-Hao Wang, Gloria Wang, Zijie J. Wang, Fuxuan Wei, Bryan Wilie, Genta Indra Winata, Xinyi Wu, Witold Wydmański, Tianbao Xie, Usama Yaseen, M. Yee, Jing Zhang, and Yue Zhang. 2021. [Nl-augmenter: A framework for task-sensitive natural language augmentation](https://md.research.elamur.ai/free/en/papers/dhole-2023-nl-augmenter-framework-task-sensitive/). ArXiv, abs/2112.02721. [↗](https://arxiv.org/abs/2112.02721)
30. Mahmoud El-Haj. 2020. [Habibi: a multi Dialect multi National Arabic Song Lyrics Corpus](https://md.research.elamur.ai/free/en/papers/haj-2020-habibi-multi-dialect-multi-national/). In Proceedings of the 12th Language Resources and Evaluation Conference, pages 1318–1326.
31. Ibrahim Abu El-Khair. 2016. 1.5 Billion Words Arabic Corpus. arXiv preprint arXiv:1611.04033. [↗](https://arxiv.org/abs/1611.04033)
32. Mohammed El-Razzaz, Mohamed Waleed Fakhr, and Fahima A Maghraby. 2021. Arabic gloss WSD using BERT. Applied Sciences, 11(6):2567. [↗](https://doi.org/10.3390/app11062567)
33. Ibrahim Abu Farha and Walid Magdy. 2020. [From Arabic Sentiment Analysis to Sarcasm Detection: The ArSarcasm Dataset](https://md.research.elamur.ai/free/en/papers/farha-2020-arabic-sentiment-analysis-sarcasm-detection/). In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 32–39.
34. Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation. [find](https://scholar.google.com/scholar?q=Jeffrey+Hsu%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Jason+Phang%2C+Laria+Reynolds%2C+Eric+Tang%2C+Anish+Thite%2C+Ben+Wang%2C+Kevin+Wang%2C+and+Andy+Zou%2E+2021%2E+A+framework+for+few%2Dshot+language+model+evaluation%2E)
35. Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020. [SyntaxGym: An online platform for targeted evaluation of language models](https://md.research.elamur.ai/free/en/papers/gauthier-2020-syntaxgym-online-platform-targeted-evaluation/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 70–76, Online. Association for Computational Linguistics.
36. Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. 2021. The GEM benchmark: Natural language generation, its evaluation and metrics. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pages 96–120, Online. Association for Computational Linguistics. [↗](https://doi.org/10.60692/9q1tp-bhr14)
37. Abbas Ghaddar, Yimeng Wu, Ahmad Rashid, Khalil Bibi, Mehdi Rezagholizadeh, Chao Xing, Yasheng Wang, Duan Xinyu, Zhefeng Wang, Baoxing Huai, et al. 2021. [JABER: Junior Arabic BERT](https://md.research.elamur.ai/free/en/papers/ghaddar-2021-jaber-saber-junior-senior-arabic/). arXiv preprint arXiv:2112.04329. [↗](https://arxiv.org/abs/2112.04329)
38. Bilal Ghanem, Jihen Karoui, Farah Benamara, Véronique Moriceau, and Paolo Rosso. 2019. IDAT@FIRE2019: Overview of the Track on Irony Detection in Arabic Tweets. . In Mehta P., Rosso P., Majumder P., Mitra M. (Eds.) Working Notes of the Forum for Information Retrieval Evaluation (FIRE 2019). CEUR Workshop Proceedings. In: CEUR-WS.org, Kolkata, India, December 12-15. [find](https://scholar.google.com/scholar?q=Bilal+Ghanem%2C+Jihen+Karoui%2C+Farah+Benamara%2C+V%C3%A9ronique+Moriceau%2C+and+Paolo+Rosso%2E+2019%2E+IDAT%40FIRE2019%3A+Overview+of+the+Track+on+Irony+Detection+in+Arabic+Tweets%2E+%2E+In+Mehta+P%2E%2C+Rosso+P%2E%2C+Majumder+P%2E%2C+M)
39. Chayma Fourati, Abir Messaoudi, and Hatem Haddad. 2020. [TUNIZI: a Tunisian Arabizi sentiment analysis Dataset](https://md.research.elamur.ai/free/en/papers/fourati-2020-tunizi-tunisian-arabizi-sentiment-analysis/). In AfricaNLP Workshop, Putting Africa on the NLP Map. ICLR 2020, Virtual Event, volume arXiv:3091079. [↗](https://doi.org/10.48550/arxiv.2004.14303)
40. Jiyeon Ham, Yo Joong Choe, Kyubyong Park, Ilji Choi, and Hyungjoon Soh. 2020. KorNLI and KorSTS: New Benchmark Datasets for Korean Natural Language Understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 422–430. [find](https://scholar.google.com/scholar?q=Jiyeon+Ham%2C+Yo+Joong+Choe%2C+Kyubyong+Park%2C+Ilji+Choi%2C+and+Hyungjoon+Soh%2E+2020%2E+KorNLI+and+KorSTS%3A+New+Benchmark+Datasets+for+Korean+Natural+Language+Understanding%2E+In+Findings+of+the+Association+for+Co)
41. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced 9570 12 bert with disentangled attention. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Pengcheng+He%2C+Xiaodong+Liu%2C+Jianfeng+Gao%2C+and+Weizhu+Chen%2E+2021%2E+Deberta%3A+Decoding%2Denhanced+9570+12)
42. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.2009.03300)
43. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International Conference on Machine Learning, pages 4411–4421. PMLR. [find](https://scholar.google.com/scholar?q=Junjie+Hu%2C+Sebastian+Ruder%2C+Aditya+Siddhant%2C+Graham+Neubig%2C+Orhan+Firat%2C+and+Melvin+Johnson%2E+2020%2E+Xtreme%3A+A+massively+multilingual+multi%2Dtask+benchmark+for+evaluating+cross%2Dlingual+generalisation%2E+In)
44. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. [The interplay of variant, size, and task type in Arabic pre-trained language models](https://md.research.elamur.ai/free/en/papers/inoue-2021-interplay-variant-size-task-type/). In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics.
45. Jude Khouja. 2020. Stance prediction and claim verification: An Arabic perspective. In Proceedings of the Third Workshop on Fact Extraction and VERification (FEVER), pages 8–17, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jude+Khouja%2E+2020%2E+Stance+prediction+and+claim+verification%3A+An+Arabic+perspective%2E+In+Proceedings+of+the+Third+Workshop+on+Fact+Extraction+and+VERification+%28FEVER%29%2C+pages+8%E2%80%9317%2C+Online%2E+Association+fo)
46. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dynabench: Rethinking benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4110–4124, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Douwe+Kiela%2C+Max+Bartolo%2C+Yixin+Nie%2C+Divyansh+Kaushik%2C+Atticus+Geiger%2C+Zhengxuan+Wu%2C+Bertie+Vidgen%2C+Grusha+Prasad%2C+Amanpreet+Singh%2C+Pratik+Ringshia%2C+Zhiyi+Ma%2C+Tristan+Thrush%2C+Sebastian+Riedel%2C+Zeerak+)
47. Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. [Reformer: The efficient transformer](https://md.research.elamur.ai/free/en/papers/kitaev-2020-reformer-efficient-transformer/). In International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.2001.04451)
48. Taku Kudo and John Richardson. 2018. [Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71.
49. Kentaro Kurihara, Daisuke Kawahara, and Tomohide Shibata. 2022. [Jglue: Japanese general language understanding evaluation](https://md.research.elamur.ai/free/en/papers/kurihara-2022-jglue-japanese-general-language-understanding/). In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2957–2966. [↗](https://doi.org/10.5715/jnlp.29.711)
50. Wuwei Lan, Yang Chen, Wei Xu, and Alan Ritter. 2020. An Empirical Study of Pre-trained Transformers for Arabic Information Extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4727–4734. [find](https://scholar.google.com/scholar?q=Wuwei+Lan%2C+Yang+Chen%2C+Wei+Xu%2C+and+Alan+Ritter%2E+2020%2E+An+Empirical+Study+of+Pre%2Dtrained+Transformers+for+Arabic+Information+Extraction%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Nat)
51. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. arXiv preprint arXiv:1909.11942. [↗](https://arxiv.org/abs/1909.11942)
52. Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoit Crabbé, Laurent Besacier, and Didier Schwab. 2020. [FlauBERT: Unsupervised Language Model Pre-training for French](https://md.research.elamur.ai/free/en/papers/le-2019-flaubert-unsupervised-language-model-pre/). In Proceedings of The 12th Language Resources and Evaluation Conference, pages 2479–2490. [↗](https://doi.org/10.5281/zenodo.3562902)
53. Trung Le, Tuan Nguyen, Nhat Ho, Hung Bui, and Dinh Phung. 2021. LAMDA: Label Matching Deep Domain Adaptation. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6043–6054. PMLR. [find](https://scholar.google.com/scholar?q=Trung+Le%2C+Tuan+Nguyen%2C+Nhat+Ho%2C+Hung+Bui%2C+and+Dinh+Phung%2E+2021%2E+LAMDA%3A+Label+Matching+Deep+Domain+Adaptation%2E+In+Proceedings+of+the+38th+International+Conference+on+Machine+Learning%2C+volume+139+of+Pro)
54. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. [BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics.
55. Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020b. [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://md.research.elamur.ai/free/en/papers/lewis-2019-mlqa-evaluating-cross-lingual-extractive/). pages 7315–7330.
56. Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, et al. 2021. [Datasets: A community library for natural language processing](https://md.research.elamur.ai/free/en/papers/lhoest-2021-datasets-community-library-natural-language/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 175–184. [↗](https://doi.org/10.48550/arxiv.2109.02846)
57. Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, et al. 2020. XGLUE: A new benchmark datasetfor cross-lingual pre-training, understanding and generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6008–6018. [find](https://scholar.google.com/scholar?q=Yaobo+Liang%2C+Nan+Duan%2C+Yeyun+Gong%2C+Ning+Wu%2C+Fenfei+Guo%2C+Weizhen+Qi%2C+Ming+Gong%2C+Linjun+Shou%2C+Daxin+Jiang%2C+Guihong+Cao%2C+et+al%2E+2020%2E+XGLUE%3A+A+new+benchmark+datasetfor+cross%2Dlingual+pre%2Dtraining%2C+underst)
58. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019a. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692. [↗](https://arxiv.org/abs/1907.11692)
59. Zihan Liu, Yan Xu, Genta Indra Winata, and Pascale Fung. 2019b. Incorporating Word and Subword Units in Unsupervised Machine. Translation Using Language Model Rescoring. 9571 13 [find](https://scholar.google.com/scholar?q=Zihan+Liu%2C+Yan+Xu%2C+Genta+Indra+Winata%2C+and+Pascale+Fung%2E+2019b%2E+Incorporating+Word+and+Subword+Units+in+Unsupervised+Machine%2E+Translation+Using+Language+Model+Rescoring%2E+9571+13)
60. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3470–3487, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Swaroop+Mishra%2C+Daniel+Khashabi%2C+Chitta+Baral%2C+and+Hannaneh+Hajishirzi%2E+2022%2E+Cross%2Dtask+generalization+via+natural+language+crowdsourcing+instructions%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+th)
61. Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. [SemEval-2018 Task 1: Affect in Tweets](https://md.research.elamur.ai/free/en/papers/mohammad-2018-semeval-2018-task-1-affect/). In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 1–17, New Orleans, Louisiana. Association for Computational Linguistics.
62. Hussein Mozannar, Karl El Hajal, Elie Maamary, and Hazem Hajj. 2019. [Neural Arabic Question Answering](https://md.research.elamur.ai/free/en/papers/mozannar-2019-neural-arabic-question-answering/). In Proceedings of the Fourth Arabic Natural Language Processing Workshop, Florence, Italy. Association for Computational Linguistics.
63. Hamdy Mubarak, Kareem Darwish, Walid Magdy, Tamer Elsayed, and Hend Al-Khalifa. 2020. [Overview of OSACT4 Arabic Offensive Language Detection Shared Task](https://md.research.elamur.ai/free/en/papers/mubarak-2020-overview-osact4-arabic-offensive-language/). In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 48–52, Marseille, France. European Language Resource Association.
64. Hamdy Mubarak, Sabit Hassan, and Ahmed Abdelali. 2021. Adult content detection on Arabic Twitter: Analysis and experiments. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 136–144, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Hamdy+Mubarak%2C+Sabit+Hassan%2C+and+Ahmed+Abdelali%2E+2021%2E+Adult+content+detection+on+Arabic+Twitter%3A+Analysis+and+experiments%2E+In+Proceedings+of+the+Sixth+Arabic+Natural+Language+Processing+Workshop%2C+pag)
65. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022. [AraT5: Text-to-Text Transformers for Arabic Language Generation](https://md.research.elamur.ai/free/en/papers/nagoudi-2022-arat5-text-text-transformers-arabic/). Online. Association for Computational Linguistics.
66. El Moatez Billah Nagoudi, AbdelRahim Elmadany, Muhammad Abdul-Mageed, and Tariq Alhindi. 2020. Machine generation and detection of Arabic manipulated and fake news. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 69–84, Barcelona, Spain (Online). Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2011.03092)
67. Wilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi Fasubaa, Taiwo Fagbohungbe, Solomon Oluwole Akinola, Shamsuddeen Muhammad, Salomon Kabongo Kabenamualu, Salomey Osei, Freshia Sackey, Rubungo Andre Niyongabo, Ricky Macharm, Perez Ogayo, Orevaoghene Ahia, Musie Meressa Berhe, Mofetoluwa Adeyemi, Masabata Mokgesi-Selinga, Lawrence Okegbemi, Laura Martinus, Kolawole Tajudeen, Kevin Degila, Kelechi Ogueji, Kathleen Siminyu, Julia Kreutzer, Jason Webster, Jamiil Toure Ali, Jade Abbott, Iroro Orife, Ignatius Ezeani, Idris Abdulkadir Dangana, Herman Kamper, Hady Elsahar, Goodness Duru, Ghollah Kioko, Murhabazi Espoir, Elan van Biljon, Daniel Whitenack, Christopher Onyefuluchi, Chris Chinenye Emezue, Bonaventure F. P. Dossou, Blessing Sibanda, Blessing Bassey, Ayodele Olabiyi, Arshath Ramkilowan, Alp Öktem, Adewale Akinfaderin, and Abdallah Bashir. 2020. [Participatory research for low-resourced machine translation: A case study in African languages](https://md.research.elamur.ai/free/en/papers/nekoto-2020-participatory-research-low-resourced-machine/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2144–2160, Online. Association for Computational Linguistics.
68. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. [Pytorch: An imperative style, high-performance deep learning library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/). Advances in neural information processing systems, 32:8026– 8037. [↗](https://doi.org/10.48550/arxiv.1912.01703)
69. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
70. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21:1–67. [↗](https://doi.org/10.48550/arxiv.1910.10683)
71. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2022. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1). [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J%2E+Liu%2E+2022%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtex)
72. Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. 2021. [XTREME-R: Towards more challenging and nuanced multilingual evaluation](https://md.research.elamur.ai/free/en/papers/ruder-2021-xtreme-r-more-challenging-nuanced/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10215–10245, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
73. Motaz K Saad and Wesam M Ashour. 2010. OSAC: Open Source Arabic Corpora . In 6th ArchEng Int. Symposiums, EEECS, volume 10. [find](https://scholar.google.com/scholar?q=Motaz+K+Saad+and+Wesam+M+Ashour%2E+2010%2E+OSAC%3A+Open+Source+Arabic+Corpora+%2E+In+6th+ArchEng+Int%2E+Symposiums%2C+EEECS%2C+volume+10%2E)
74. Ali Safaya. 2020. Arabic-ALBERT. [find](https://scholar.google.com/scholar?q=Ali+Safaya%2E+2020%2E+Arabic%2DALBERT%2E)
75. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. [KUISAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media](https://md.research.elamur.ai/free/en/papers/safaya-2020-kuisail-semeval-2020-task-12/). In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059, Barcelona (online). International Committee for Computational Linguistics.
76. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108. 9572 14 [↗](https://arxiv.org/abs/1910.01108)
77. Nathan Schneider, Behrang Mohit, Kemal Oflazer, and Noah A Smith. 2012. [Coarse lexical semantic annotation with supersenses: an arabic case study](https://md.research.elamur.ai/free/en/papers/schneider-2012-coarse-lexical-semantic-annotation-supersenses/). In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 253–258.
78. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and Korean Voice Search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149–5152. IEEE. [↗](https://doi.org/10.1109/icassp.2012.6289079)
79. Haitham Seelawi, Ahmad Mustafa, Hesham Al-Bataineh, Wael Farhan, and Hussein T Al-Natsheh. 2019. Nsurl-2019 task 8: Semantic question similarity in arabic. In Proceedings of The First International Workshop on NLP Solutions for Under Resourced Languages (NSURL 2019) co-located with ICNLSP 2019-Short Papers, pages 1–8. [find](https://scholar.google.com/scholar?q=Haitham+Seelawi%2C+Ahmad+Mustafa%2C+Hesham+Al%2DBataineh%2C+Wael+Farhan%2C+and+Hussein+T+Al%2DNatsheh%2E+2019%2E+Nsurl%2D2019+task+8%3A+Semantic+question+similarity+in+arabic%2E+In+Proceedings+of+The+First+International+Wo)
80. Haitham Seelawi, Ibraheem Tuffaha, Mahmoud Gzawi, Wael Farhan, Bashar Talafha, Riham Badawi, Zyad Sober, Oday Al-Dweik, Abed Alhakim Freihat, and Hussein Al-Natsheh. 2021. ALUE: Arabic language understanding evaluation. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 173–184, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Haitham+Seelawi%2C+Ibraheem+Tuffaha%2C+Mahmoud+Gzawi%2C+Wael+Farhan%2C+Bashar+Talafha%2C+Riham+Badawi%2C+Zyad+Sober%2C+Oday+Al%2DDweik%2C+Abed+Alhakim+Freihat%2C+and+Hussein+Al%2DNatsheh%2E+2021%2E+ALUE%3A+Arabic+language+unders)
81. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). arXiv preprint arXiv:2206.04615. [↗](https://arxiv.org/abs/2206.04615)
82. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructure. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7). Leibniz-Institut für Deutsche Sprache. [find](https://scholar.google.com/scholar?q=Pedro+Javier+Ortiz+Su%C3%A1rez%2C+Beno%C3%AEt+Sagot%2C+and+Laurent+Romary%2E+2019%2E+Asynchronous+Pipeline+for+Processing+Huge+Corpora+on+Medium+to+Low+Resource+Infrastructure%2E+In+7th+Workshop+on+the+Challenges+in+the+)
83. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), volume 2012, pages 2214–2218. [find](https://scholar.google.com/scholar?q=J%C3%B6rg+Tiedemann%2E+2012%2E+Parallel+data%2C+tools+and+interfaces+in+opus%2E+In+Proceedings+of+the+Eighth+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%E2%80%9912%29%2C+volume+2012%2C+pages+2214%E2%80%932218%2E)
84. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, pages 6000–6010.
85. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). arXiv preprint arXiv:1905.00537. [↗](https://arxiv.org/abs/1905.00537)
86. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. [Glue: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). arXiv preprint arXiv:1804.07461. [↗](https://arxiv.org/abs/1804.07461)
87. Junqiu Wei, Qun Liu, Yinpeng Guo, and Xin Jiang. 2021. Training multilingual pre-trained language model with byte-level subwords. arXiv preprint arXiv:2101.09469. [↗](https://arxiv.org/abs/2101.09469)
88. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. 2015. [Towards ai-complete question answering: A set of prerequisite toy tasks](https://md.research.elamur.ai/free/en/papers/weston-2015-towards-ai-complete-question-answering/). arXiv preprint arXiv:1502.05698. [↗](https://arxiv.org/abs/1502.05698)
89. Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, et al. 2020. [Indonlu: Benchmark and resources for evaluating indonesian natural language understanding](https://md.research.elamur.ai/free/en/papers/wilie-2020-indonlu-benchmark-resources-evaluating-indonesian/). arXiv preprint arXiv:2009.05387. [↗](https://arxiv.org/abs/2009.05387)
90. Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al. 2020. [Clue: A chinese language understanding evaluation benchmark](https://md.research.elamur.ai/free/en/papers/xu-2020-clue-chinese-language-understanding/). arXiv preprint arXiv:2004.05986. [↗](https://arxiv.org/abs/2004.05986)
91. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. [mt5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2021.naacl-main.41)
92. Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. [Big bird: Transformers for longer sequences](https://md.research.elamur.ai/free/en/papers/zaheer-2020-big-bird-transformers-longer-sequences/). In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA. Curran Associates Inc. [↗](https://doi.org/10.48550/arxiv.2007.14062)
93. Omar F Zaidan and Chris Callison-Burch. 2014. [Arabic Dialect Identification](https://md.research.elamur.ai/free/en/papers/zaidan-2014-arabic-dialect-identification/) . Computational Linguistics, 40(1):171–202. [↗](https://doi.org/10.1162/coli_a_00169)
94. Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and Abdelhak Lakhouaja. 2019. OSIAN: Open Source International Arabic News Corpus - Preparation and Integration into the CLARIN-infrastructure. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 175–182, Florence, Italy. Association for Computational Linguistics. [↗](https://doi.org/10.60692/vm65h-ach57)
95. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020a. [Pegasus: Pre-training with extracted gap-sentences for abstractive summarization](https://md.research.elamur.ai/free/en/papers/zhang-2019-pegasus-pre-training-extracted-gap/). In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org. [↗](https://doi.org/10.48550/arxiv.1912.08777)
96. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020b. [DIALOGPT : Large-scale generative pre-training for conversational response generation](https://md.research.elamur.ai/free/en/papers/zhang-2019-dialogpt-large-scale-generative-pre/). In Proceedings of the 58th Annual Meeting of the Association for Computational 9573 15 Linguistics: System Demonstrations, pages 270–278, Online. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1911.00536)
97. Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced language representation with informative entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1441–1451, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zhengyan+Zhang%2C+Xu+Han%2C+Zhiyuan+Liu%2C+Xin+Jiang%2C+Maosong+Sun%2C+and+Qun+Liu%2E+2019%2E+ERNIE%3A+Enhanced+language+representation+with+informative+entities%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Asso)
98. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. [Aligning books and movies: Towards story-like visual explanations by watching movies and reading books](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/). In Proceedings of the IEEE international conference on computer vision, pages 19–27. 9574 16

*Updated 24 October 2026*