---
title: "MA at AraFinNLP2024 — paper by Ramadan et al. (2024)"
description: "The paper tackles banking intent detection across Arabic dialects for subtask-1 of the AraFinNLP 2024… 14 references, cited by 2 papers here."
authors: ["Asmaa Ramadan", "Manar Amr", "Marwan Torki", "Nagwa El-Makky"]
year: 2024
venue: "Proceedings of the Second Arabic Natural Language Processing Conference"
source: "https://aclanthology.org/2024.arabicnlp-1.41/"
doi: "10.18653/v1/2024.arabicnlp-1.41"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ramadan-2024-ma-arafinnlp2024-bert-based-ensemble/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ramadan-2024-ma-arafinnlp2024-bert-based-ensemble/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/12/) / MA at AraFinNLP2024: BERT-based Ensemble for Cross-dialectal Arabic Intent Detection

# MA at AraFinNLP2024: BERT-based Ensemble for Cross-dialectal Arabic Intent Detection

**Authors:** Asmaa Ramadan, Manar Amr, [Marwan Torki](https://md.research.elamur.ai/free/en/authors/marwan-torki/) and [Nagwa El-Makky](https://md.research.elamur.ai/free/en/authors/nagwa-el-makky/) · **Year:** 2024 · **Venue:** [Proceedings of the Second Arabic Natural Language Processing Conference](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2024](https://md.research.elamur.ai/free/en/venues/arabicnlp/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.arabicnlp-1.41/) · [PDF](https://aclanthology.org/2024.arabicnlp-1.41.pdf) · [DOI 10.18653/v1/2024.arabicnlp-1.41](https://doi.org/10.18653/v1/2024.arabicnlp-1.41) · [OpenAlex](https://openalex.org/W4402683875)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/ramadan-2024-ma-arafinnlp2024-bert-based-ensemble/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper tackles banking intent detection across Arabic dialects for subtask-1 of the AraFinNLP 2024 shared task. ArBanking77, the only Arabic banking intent dataset, covers just MSA and Palestinian, so systems trained on it struggle with other dialects. The authors expanded the training data by machine-translating the MSA queries into further dialects with the distilled 600M NLLB model. They then removed stray diacritics and symbols that the translation introduced. On this enlarged data they fine-tuned six BERT classifiers built on AraBERTv0.2-Twitter and MARBERTv2. Some classifiers combined cross-entropy with an NT-Xent contrastive term. The six models were made deliberately different in backbone, loss, data cleaning and dialect coverage. Their predicted probabilities are averaged with equal weight. The ensemble reached an F1 of about 0.877 and ranked first in the subtask, beating every single member. The work shows that cheap synthetic dialect data and a carefully diversified ensemble can make up for missing dialect resources in a narrow domain.

## Key points
- ArBanking77 has 31,404 MSA and Palestinian queries over 77 intents. After augmentation, the training set holds 85,957 samples across eight varieties: MSA, PAL, South and North Levantine, Egyptian, Tunisian, Moroccan and Saudi, with roughly 10.7k samples each. The shared-task test set contains 11,721 queries.
- The ensemble has six members: four AraBERTv0.2-Twitter models (about 136M parameters) and two MARBERTv2 models (about 160M parameters). Each member has a single linear classification head, and outputs are combined by averaging the probabilities.
- The loss is cross-entropy plus α·NT-Xent, with α set to 10. A custom batch sampler limits how many distinct classes appear in each batch, so every sample has both positives and negatives.
- Ensemble test F1 is 0.8773 in the results table; the abstract reports 0.8771. The best single model is m3 (AraBERT, preprocessed data, contrastive loss) at 0.8578. The worst is m4, trained only on MSA and PAL, at 0.7913.
- Preprocessing helped, raising F1 from 0.8434 (m2) to 0.8547 (m1). With identical settings, AraBERTv0.2-Twitter outperformed MARBERTv2, 0.8434 (m2) against 0.8198 (m5).

## Interesting ideas
- Use an MT model to turn labelled standard-language examples into dialect copies that keep their labels, giving cheap coverage of unseen varieties despite translation noise.
- Build ensemble diversity on purpose by varying the backbone, the loss, cleaned versus raw data, and dialect subsets, because members that err differently make averaging more effective.
- Train some models on raw text and others on cleaned text, so the group as a whole tolerates imperfect input.
- Control batch composition so that a contrastive objective always has in-class positives and out-of-class negatives, instead of relying on random batches.
- Keep a control member trained only on the original narrow data (m4) to measure directly what augmentation adds.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Paraphrasing silver sentences into informal, regional or non-native English could widen training coverage. Labels that survive the rewrite would need to be checked by witness-parser agreement before use.
- **I2** (learn): Witness parsers for silver data should be chosen to fail in different ways. Their agreement then carries more evidence than votes from near-identical models.
- **I3** (tokens): Training tagger and lemmatizer variants on both normalized and noisy text could make the pipeline robust to typos and odd symbols. A disagreement between the variants would also flag a suspicious token in an explainable way.
- **I4** (frames): If frames or VerbNet classes are ever learned with a contrastive objective, a class-limited sampler would ensure rare roles meet real contrasts in every batch.
- **I5** (learn): Every augmentation or repair-rule experiment should be compared against a baseline trained without it, which serves as a built-in negative control.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Alexandria University](https://md.research.elamur.ai/free/en/universities/alexandria-university/) — University, Alexandria, Egypt · 39 works · 287 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Asmaa Ramadan, Manar Amr, Marwan Torki and Nagwa El-Makky. 2024. MA at AraFinNLP2024: BERT-based Ensemble for Cross-dialectal Arabic Intent Detection. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ramadan2024ma,
  title = {{MA at AraFinNLP2024: BERT-based Ensemble for Cross-dialectal Arabic Intent Detection}},
  author = {Asmaa Ramadan, Manar Amr, Marwan Torki and Nagwa El-Makky},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.41},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.41}
}
```

- [← previous Learnings from curating a trustworthy, well-annotated, and useful dataset of disordered English speech](https://md.research.elamur.ai/free/en/papers/jiang-2024-learnings-curating-trustworthy-well-annotated/)
- [next → Mela at ArAIEval Shared Task: Propagandistic Techniques Detection in Arabic with a Multilingual Approach](https://md.research.elamur.ai/free/en/papers/riyadh-2024-mela-araieval-shared-task-propagandistic/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/12/)

## References

14 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. [ARBERT & MARBERT: Deep bidirectional transformers for Arabic](https://md.research.elamur.ai/free/en/papers/abdul-mageed-2021-arbert-marbert-deep-bidirectional-transformers/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2021.acl-long.551)
2. Bushra Algotiml, AbdelRahim Elmadany, and Walid Magdy. 2019. [Arabic tweet-act: Speech act recognition for Arabic asynchronous conversations](https://md.research.elamur.ai/free/en/papers/algotiml-2019-arabic-tweet-act-speech-act/). In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 183–191, Florence, Italy. Association for Computational Linguistics.
3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. [Arabert: Transformer-based model for arabic language understanding](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/). Preprint, arXiv:2003.00104. [↗](https://arxiv.org/abs/2003.00104)
4. Sanad Malaysha, Mo El-Haj, Saad Ezzini, Mohammad Khalilia, Mustafa Jarrar, Sultan Nasser, and Ismail Berrada. 2024. [AraFinNlp 2024: The first arabic financial nlp shared task](https://md.research.elamur.ai/free/en/papers/malaysha-2024-arafinnlp-2024-first-arabic-financial/). In Proceedings of the 2nd Arabic Natural Language Processing Conference (Arabic-NLP), Part of the ACL 2024. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2407.09818)
5. Ridha Mezzi, Aymen Yahyaoui, Mohamed Wassim Krir, Wadii Boulila, and Anis Koubaa. 2022. Mental health intent recognition for arabic-speaking patients using the mini international neuropsychiatric interview (mini) and bert model. Sensors, 22(3). [↗](https://doi.org/10.3390/s22030846)
6. Anbang Xu, Zhe Liu, Yufan Guo, Vibha Sinha, and Rama Akkiraju. 2017. A new chatbot for customer service on social media. [find](https://scholar.google.com/scholar?q=Anbang+Xu%2C+Zhe+Liu%2C+Yufan+Guo%2C+Vibha+Sinha%2C+and+Rama+Akkiraju%2E+2017%2E+A+new+chatbot+for+customer+service+on+social+media%2E)
7. Cecilia Ying and Stephen Thomas. 2022. Label errors in BANKING77. In Proceedings of the Third Workshop on Insights from Negative Results in NLP, pages 139– 143, Dublin, Ireland. Association for Computational Linguistics. [↗](https://doi.org/10.48448/e34y-q648)
8. Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. [Efficient intent detection with dual sentence encoders](https://md.research.elamur.ai/free/en/papers/casanueva-2020-efficient-intent-detection-dual-sentence/). Preprint, arXiv:2003.04807. [↗](https://arxiv.org/abs/2003.04807)
9. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. [A simple framework for contrastive learning of visual representations](https://md.research.elamur.ai/free/en/papers/chen-2020-simple-framework-contrastive-learning-visual/). Preprint, arXiv:2002.05709. [↗](https://arxiv.org/abs/2002.05709)
10. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. [↗](https://arxiv.org/abs/2207.04672)
11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). Preprint, arXiv:1810.04805. [↗](https://arxiv.org/abs/1810.04805)
12. Thomas G Dietterich. 2000. Ensemble methods in machine learning. In International workshop on multiple classifier systems, pages 1–15. Springer. [find](https://scholar.google.com/scholar?q=Thomas+G+Dietterich%2E+2000%2E+Ensemble+methods+in+machine+learning%2E+In+International+workshop+on+multiple+classifier+systems%2C+pages+1%E2%80%9315%2E+Springer%2E)
13. Mustafa Jarrar, Ahmet Birim, Mohammed Khalilia, Mustafa Erden, and Sana Ghanem. 2023. [Arbanking77: Intent detection neural model and a new dataset in modern and dialectical arabic](https://md.research.elamur.ai/free/en/papers/jarrar-2023-arbanking77-intent-detection-neural-model/). In Proceedings of ArabicNLP 2023, Singapore (Hybrid), December 7, 2023, pages 276–287. Association for Computational Linguistics.
14. Xianzhi Li, Will Aitken, Xiaodan Zhu, and Stephen W. Thomas. 2022. Learning better intent representations for financial open intent classification. In Proceedings of the Fourth Workshop on Financial Technology and Natural Language Processing (FinNLP), pages 68–77, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xianzhi+Li%2C+Will+Aitken%2C+Xiaodan+Zhu%2C+and+Stephen+W%2E+Thomas%2E+2022%2E+Learning+better+intent+representations+for+financial+open+intent+classification%2E+In+Proceedings+of+the+Fourth+Workshop+on+Financial+T)

*Updated 1 November 2026*