---
title: "AlclaM — paper by Ahmed et al. (2024) — summary and ideas"
description: "Arabic pre-trained encoders such as AraBERT, ArBERT and CAMeL are trained mostly… 98% accuracy. 41 references, cited by 4 papers here."
authors: ["Murtadha Ahmed", "Saghir Alfasly", "Bo Wen", "Jamal Addeen", "Mohammed Ahmed", "Yunfeng Liu"]
year: 2024
venue: "Proceedings of the Second Arabic Natural Language Processing Conference"
source: "https://aclanthology.org/2024.arabicnlp-1.14/"
doi: "10.18653/v1/2024.arabicnlp-1.14"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ahmed-2024-alclam-arabic-dialect-language-model/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ahmed-2024-alclam-arabic-dialect-language-model/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/) / AlclaM: Arabic Dialect Language Model

# AlclaM: Arabic Dialect Language Model

**Authors:** [Murtadha Ahmed](https://md.research.elamur.ai/free/en/authors/murtadha-ahmed/), [Saghir Alfasly](https://md.research.elamur.ai/free/en/authors/saghir-alfasly/), [Bo Wen](https://md.research.elamur.ai/free/en/authors/bo-wen/), Jamal Addeen, [Mohammed Ahmed](https://md.research.elamur.ai/free/en/authors/mohammed-ahmed/) and [Yunfeng Liu](https://md.research.elamur.ai/free/en/authors/yunfeng-liu/) · **Year:** 2024 · **Venue:** [Proceedings of the Second Arabic Natural Language Processing Conference](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2024](https://md.research.elamur.ai/free/en/venues/arabicnlp/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.arabicnlp-1.14/) · [PDF](https://aclanthology.org/2024.arabicnlp-1.14.pdf) · [DOI 10.18653/v1/2024.arabicnlp-1.14](https://doi.org/10.18653/v1/2024.arabicnlp-1.14) · [OpenAlex](https://openalex.org/W4402671397)

**Citations:** 6 citations (OpenAlex, 2026-10-14) · [4](https://md.research.elamur.ai/free/en/papers/ahmed-2024-alclam-arabic-dialect-language-model/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Arabic pre-trained encoders such as AraBERT, ArBERT and CAMeL are trained mostly on Modern Standard Arabic (MSA). As a result, their tokenizers cover regional dialects poorly and favour MSA vocabulary, even though most Arabic benchmark data comes from dialect-heavy social media. The authors scraped posts and comments from Arabic YouTube channels and Facebook pages. Because much of this material was MSA (for example, religious citations), they removed it with a dialect-vs-MSA classifier, leaving about 3.4M dialectal sentences. They combined these with Arabic text from opensubtitles, built a new vocabulary, and pre-trained a BERT-base model, AlcLaM, from scratch on only 13 GB of text. That is a small fraction of the data behind CAMeL, MARBERT and ArBERT. AlcLaM was tested on dialect identification, sentiment analysis, and hate/offensive speech detection. It gave the best scores on most datasets, with the largest gains in dialect identification. The work shows that well-targeted in-domain pre-training data can outweigh sheer data volume.

## Key points
- The corpus has 3,372,744 dialectal sentences and 54,557,408 tokens. MSA was filtered out by CAMeL fine-tuned on MADAR as a binary MSA/Dialect classifier, which reached 98% accuracy.
- AlcLaM's 13 GB of training text is 7.8%, 10.2% and 21.3% of the data used for CAMeL, MARBERT and ArBERT, respectively.
- The setup follows BERT-base: 12 layers, 768 hidden units, 12 heads, roughly 125M parameters.
  - Training used whole word masking (15% of tokens, split 80/10/10) and no NSP loss.
  - Batch size was 64, maximum sequence length 128, 5 epochs, learning rate 5e-5.
- Dialect identification F1 improved over the best baseline on several MADAR variants:
  - MADAR-2: 98.2 vs 87.5 for CAMeL.
  - MADAR-9: 81.9 vs 78.2 for MARBERT.
  - MADAR-26: 66.3 vs 62.9 for CAMeL.
- AlcLaM did not win everywhere. MARBERT was better on NADI (28.6 vs 25.6 F1) and CAMeL on LABR. All results are averages over five seeds, with t-test significance at p\<0.05.

## Interesting ideas
- **Classifier-based register filtering:** a cheap register classifier strips off-target text (here MSA) from scraped data before pre-training, so the corpus stays focused on the variety you care about.
- **In-domain vocabulary:** building the vocabulary from in-domain text, rather than reusing one dominated by the standard variety, lets the tokenizer recognise non-standard word forms instead of splitting them into fragments.
- **Evaluation at several granularities:** coarser label sets were derived from one fine-grained dataset (MADAR-26 collapsed into 9 regions and into a binary split), showing where a model fails.
- **Seeds plus significance testing:** averaging over five random seeds and testing significance makes small gains trustworthy rather than noise.
- **Focused data over volume:** a model trained on a small but well-targeted corpus beat models trained on much more general text.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Before silver parses are agreed by witness parsers, a genre/register filter could keep or separate informal web English, so training batches match the target domain.
- **I2** (tokens): The English tokenizer and lemma lexicon could be extended from informal text (slang, creative spellings), reducing unknown forms that later corrupt tags and lemmas.
- **I3** (learn): Tagger and parser output could be scored against the same gold at several levels (e.g., fine relation subtypes vs. collapsed UD relations), separating near-misses from real structural errors.
- **I4** (learn): Comparisons of perceptron vs. gparse, or before/after repair rules, should report multi-seed means with a significance test alongside negative controls.
- **I5** (tree): A modest treebank of dialectal or informal English, converted to UD, may help gparse on such input more than adding more standard newswire.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Mayo Clinic](https://md.research.elamur.ai/free/en/labs/mayo-clinic/) — Research institute, Rochester, United States · 5 works · 38,950 citations
- [Northwestern Polytechnical University](https://md.research.elamur.ai/free/en/universities/northwestern-polytechnical-university/) — University, Xi'an, China · 13 works · 67 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Murtadha Ahmed, Saghir Alfasly, Bo Wen, Jamal Addeen, Mohammed Ahmed and Yunfeng Liu. 2024. AlclaM: Arabic Dialect Language Model. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ahmed2024alclam,
  title = {{AlclaM: Arabic Dialect Language Model}},
  author = {Murtadha Ahmed, Saghir Alfasly, Bo Wen, Jamal Addeen, Mohammed Ahmed and Yunfeng Liu},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.14},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.14}
}
```

- [← previous A Corpus for Sentence-Level Subjectivity Detection on English News Articles](https://md.research.elamur.ai/free/en/papers/antici-2024-corpus-sentence-level-subjectivity-detection/)
- [next → Arabic Diacritics in the Wild: Exploiting Opportunities for Improved Diacritization](https://md.research.elamur.ai/free/en/papers/elgamal-2024-arabic-diacritics-wild-exploiting-opportunities/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/)

## References

41 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Muhammad Abdul-Mageed, AbdelRahim A. Elmadany, and El Moatez Billah Nagoudi. 2021. AR-BERT & MARBERT: deep bidirectional transformers for arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJC-NLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 7088–7105. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Muhammad+Abdul%2DMageed%2C+AbdelRahim+A%2E+Elmadany%2C+and+El+Moatez+Billah+Nagoudi%2E+2021%2E+AR%2DBERT+%26+MARBERT%3A+deep+bidirectional+transformers+for+arabic%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Assoc)
2. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020. [NADI 2020: The first nuanced Arabic dialect identification shared task](https://md.research.elamur.ai/free/en/papers/mageed-2020-nadi-2020-first-nuanced-arabic/). In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 97–110, Barcelona, Spain (Online). Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2010.11334)
3. Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. 2020. [Towards a human-like open-domain chatbot](https://md.research.elamur.ai/free/en/papers/adiwardana-2020-towards-human-like-open-domain/). CoRR, abs/2001.09977. [↗](https://arxiv.org/abs/2001.09977)
4. Murtadha Ahmed, Shengfeng Pan, Wen Bo, Jianlin Su, Xinxin Cao, Wenze Zhang, and Yunfeng Liu. 2023. Rank-aware negative training for semi-supervised text classification. Trans. Assoc. Comput. Linguistics, 11:771–786. [↗](https://doi.org/10.1162/tacl_a_00574)
5. Basma Alharbi, Hind Alamro, Manal Alshehri, Zuhair Khayyat, Manal Kalkatawi, Inji Ibrahim Jaber, and Xiangliang Zhang. 2020. [ASAD: A twitter-based benchmark arabic sentiment analysis dataset](https://md.research.elamur.ai/free/en/papers/alharbi-2020-asad-twitter-based-benchmark-arabic/). CoRR, abs/2011.00578. [↗](https://arxiv.org/abs/2011.00578)
6. Mohamed A. Aly and Amir F. Atiya. 2013. [LABR: A large scale arabic book reviews dataset](https://md.research.elamur.ai/free/en/papers/aly-2013-labr-large-scale-arabic-book/). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, 4-9 August 2013, Sofia, Bulgaria, Volume 2: Short Papers, pages 494–498. The Association for Computer Linguistics. [↗](https://doi.org/10.13140/2.1.3960.5761)
7. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. [AraBERT: Transformer-based model for Arabic language understanding](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/). In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. [↗](https://doi.org/10.48550/arxiv.2003.00104)
8. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. [The MADAR shared task on arabic fine-grained dialect identification](https://md.research.elamur.ai/free/en/papers/bouamor-2019-madar-shared-task-arabic-fine/). In Proceedings of the Fourth Arabic Natural Language Processing Workshop, WANLP@ACL 2019, Florence, Italy, August 1, 2019, pages 199–207. Association for Computational Linguistics.
9. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam Mc-Candlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. [↗](https://doi.org/10.48550/arxiv.2005.14165)
10. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2023. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). J. Mach. Learn. Res., 24:240:1–240:113. [↗](https://doi.org/10.48550/arxiv.2204.02311)
11. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.acl-main.747)
12. Slawomir Dadas, Michal Perelkiewicz, and Rafal Poswiata. 2020. Pre-training polish transformer-based language models at scale. In Artificial Intelligence and Soft Computing - 19th International Conference, ICAISC 2020, Zakopane, Poland, October 12-14, 2020, Proceedings, Part II, volume 12416 of Lecture Notes in Computer Science, pages 301–314. Springer. [find](https://scholar.google.com/scholar?q=Slawomir+Dadas%2C+Michal+Perelkiewicz%2C+and+Rafal+Poswiata%2E+2020%2E+Pre%2Dtraining+polish+transformer%2Dbased+language+models+at+scale%2E+In+Artificial+Intelligence+and+Soft+Computing+%2D+19th+International+Confer)
13. Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019. [Bertje: A dutch BERT model](https://md.research.elamur.ai/free/en/papers/de-vries-2019-bertje-dutch-bert-model/). CoRR, abs/1912.09582. [↗](https://arxiv.org/abs/1912.09582)
14. Workshop on Semantic Evaluation (SemEval-2016), pages 42–51, San Diego, California. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Workshop+on+Semantic+Evaluation+%28SemEval%2D2016%29%2C+pages+42%E2%80%9351%2C+San+Diego%2C+California%2E+Association+for+Computational+Linguistics%2E)
15. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. [ALBERT: A lite BERT for self-supervised learning of language representations](https://md.research.elamur.ai/free/en/papers/lan-2019-albert-lite-bert-self-supervised/). In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
16. Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoît Crabbé, Laurent Besacier, and Didier Schwab. 2020. [Flaubert: Unsupervised language model pre-training for french](https://md.research.elamur.ai/free/en/papers/le-2019-flaubert-unsupervised-language-model-pre/). In Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020, pages 2479–2490. European Language Resources Association. [↗](https://doi.org/10.5281/zenodo.3562902)
17. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018a. [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). CoRR, abs/1810.04805. [↗](https://arxiv.org/abs/1810.04805)
18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). CoRR, abs/1907.11692. [↗](https://arxiv.org/abs/1907.11692)
19. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018b. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). CoRR, abs/1810.04805. [↗](https://arxiv.org/abs/1810.04805)
20. Martin Malmsten, Love Börjeson, and Chris Haffenden. 2020. [Playing with words at the national library of sweden - making a swedish BERT](https://md.research.elamur.ai/free/en/papers/malmsten-2020-playing-words-national-library-sweden/). CoRR, abs/2007.01658. [↗](https://arxiv.org/abs/2007.01658)
21. A Elmadany, Hamdy Mubarak, and Walid Magdy. 2018. Arsas: An arabic speech-act and sentiment corpus of tweets. OSACT, 3:20. [find](https://scholar.google.com/scholar?q=A+Elmadany%2C+Hamdy+Mubarak%2C+and+Walid+Magdy%2E+2018%2E+Arsas%3A+An+arabic+speech%2Dact+and+sentiment+corpus+of+tweets%2E+OSACT%2C+3%3A20%2E)
22. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. [Language-agnostic BERT sentence embedding](https://md.research.elamur.ai/free/en/papers/feng-2020-language-agnostic-bert-sentence-embedding/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 878–891. Association for Computational Linguistics.
23. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. [The interplay of variant, size, and task type in Arabic pre-trained language models](https://md.research.elamur.ai/free/en/papers/inoue-2021-interplay-variant-size-task-type/). In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics.
24. Einav Itamar and Alon Itai. 2008. Using Movie Subtitles for Creating a Large-Scale Bilingual Corpora. In Proceedings of the International Conference on Language Resources and Evaluation, LREC 2008, 26 May - 1 June 2008, Marrakech, Morocco. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Einav+Itamar+and+Alon+Itai%2E+2008%2E+Using+Movie+Subtitles+for+Creating+a+Large%2DScale+Bilingual+Corpora%2E+In+Proceedings+of+the+International+Conference+on+Language+Resources+and+Evaluation%2C+LREC+2008%2C+26)
25. Svetlana Kiritchenko, Saif Mohammad, and Mohammad Salameh. 2016. [SemEval-2016 task 7: Determining sentiment intensity of English and Arabic phrases](https://md.research.elamur.ai/free/en/papers/kiritchenko-2016-semeval-2016-task-7-determining/). In Proceedings of the 10th International Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020. Camembert: a tasty french language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7203–7219. Association for Computational Linguistics.
26. Hamdy Mubarak, Kareem Darwish, Walid Magdy, Tamer Elsayed, and Hend Al-Khalifa. 2020. [Overview of OSACT4 Arabic offensive language detection shared task](https://md.research.elamur.ai/free/en/papers/mubarak-2020-overview-osact4-arabic-offensive-language/). In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 48–52, Marseille, France. European Language Resource Association.
27. Hamdy Mubarak, Sabit Hassan, and Ahmed Abdelali. 2021. Adult content detection on Arabic Twitter: Analysis and experiments. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 136–144, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Hamdy+Mubarak%2C+Sabit+Hassan%2C+and+Ahmed+Abdelali%2E+2021%2E+Adult+content+detection+on+Arabic+Twitter%3A+Analysis+and+experiments%2E+In+Proceedings+of+the+Sixth+Arabic+Natural+Language+Processing+Workshop%2C+pag)
28. Ahmed Murtadha, Shengfeng Pan, Bo Wen, Jianlin Su, Wenze Zhang, and Yunfeng Liu. 2022. BERT-ASC: auxiliary-sentence construction for implicit aspect learning in sentiment analysis. CoRR, abs/2203.11702. [↗](https://arxiv.org/abs/2203.11702)
29. Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). CoRR, abs/2307.09288. [↗](https://arxiv.org/abs/2307.09288)
30. Ahmed Murtadha, Bo Wen, Luo Ao, Shengfeng Pan, Jianlin Su, Xinxin Cao, and Yunfeng Liu. 2024. Towards robust learning with noisy and pseudo labels for text classification. Inf. Sci., 661:120160. [↗](https://doi.org/10.1016/j.ins.2024.120160)
31. Mahmoud Nabil, Mohamed A. Aly, and Amir F. Atiya. 2015. [ASTD: arabic sentiment tweets dataset](https://md.research.elamur.ai/free/en/papers/nabil-2015-astd-arabic-sentiment-tweets-dataset/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 2515–2519. The Association for Computational Linguistics. [↗](https://doi.org/10.60692/3ee2g-zjt73)
32. Dat Quoc Nguyen and Anh Tuan Nguyen. 2020. [Phobert: Pre-trained language models for vietnamese](https://md.research.elamur.ai/free/en/papers/nguyen-2020-phobert-pre-trained-language-models/). In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 1037–1042. Association for Computational Linguistics.
33. OpenAI. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). abs/2303.08774. [↗](https://arxiv.org/abs/2303.08774)
34. CoRR, Sampo Pyysalo, Jenna Kanerva, Antti Virtanen, and Filip Ginter. 2021. Wikibert models: Deep transfer learning for many languages. In Proceedings of the 23rd Nordic Conference on Computational Linguistics, NoDaLiDa 2021, Reykjavik, Iceland (Online), May 31 - June 2, 2021, pages 1–10. Linköping University Electronic Press, Sweden. [↗](https://doi.org/10.48550/arxiv.2006.01538)
35. Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. [Multilingual is not enough: BERT for finnish](https://md.research.elamur.ai/free/en/papers/virtanen-2019-multilingual-enough-bert-finnish/). CoRR, abs/1912.07076. [↗](https://arxiv.org/abs/1912.07076)
36. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). J. Mach. Learn. Res., 21:140:1–140:67. [↗](https://doi.org/10.48550/arxiv.1910.10683)
37. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. [mt5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 483–498. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2010.11934)
38. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. [KUISAIL at semeval-2020 task 12: BERT-CNN for offensive speech identification in social media](https://md.research.elamur.ai/free/en/papers/safaya-2020-kuisail-semeval-2020-task-12/). In Proceedings of the Fourteenth Workshop on Semantic Evaluation, SemEval@COLING 2020, Barcelona (online), December 12-13, 2020, pages 2054–2059. International Committee for Computational Linguistics.
39. Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. [Aligning books and movies: Towards story-like visual explanations by watching movies and reading books](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/). In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 19–27. IEEE Computer Society.
40. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Neurocomputing, 568:127063. [↗](https://doi.org/10.1016/j.neucom.2023.127063)
41. Bashar Talafha, Mohammad Ali, Muhy Eddin Za’ter, Haitham Seelawi, Ibraheem Tuffaha, Mostafa Samir, Wael Farhan, and Hussein T. Al-Natsheh. 2020. [Multi-dialect arabic BERT for country-level dialect identification](https://md.research.elamur.ai/free/en/papers/talafha-2020-multi-dialect-arabic-bert-country/). In Proceedings of the Fifth Arabic Natural Language Processing Workshop, WANLP@COLING 2020, Barcelona, Spain (Online), December 12, 2020, pages 111–118. Association for Computational Linguistics. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, [↗](https://doi.org/10.48550/arxiv.2007.05612)

*Updated 1 November 2026*