---
title: "Sakinah-AI at MentalQA — paper by Elden and Abukar (2025)"
description: "Team Sakinah-AI entered Track 1, Sub-Task 1 of the MentalQA 2025 shared task. 12 references, cited by 1 paper here."
authors: ["Fatimah Mohamed Emad Elden", "Mumina Ab. Abukar"]
year: 2025
venue: "Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks"
source: "https://aclanthology.org/2025.arabicnlp-sharedtasks.21/"
doi: "10.18653/v1/2025.arabicnlp-sharedtasks.21"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/elden-2025-sakinah-ai-mentalqa-comparative-study/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/elden-2025-sakinah-ai-mentalqa-comparative-study/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/15/) / Sakinah-AI at MentalQA: A Comparative Study of Few-Shot, Optimized, and Ensemble Methods for Arabic Mental Health Question Classification

# Sakinah-AI at MentalQA: A Comparative Study of Few-Shot, Optimized, and Ensemble Methods for Arabic Mental Health Question Classification

**Authors:** [Fatimah Mohamed Emad Elden](https://md.research.elamur.ai/free/en/authors/fatimah-mohamed-emad-elden/) and [Mumina Ab. Abukar](https://md.research.elamur.ai/free/en/authors/mumina-ab-abukar/) · **Year:** 2025 · **Venue:** [Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2025](https://md.research.elamur.ai/free/en/venues/arabicnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.arabicnlp-sharedtasks.21/) · [PDF](https://aclanthology.org/2025.arabicnlp-sharedtasks.21.pdf) · [DOI 10.18653/v1/2025.arabicnlp-sharedtasks.21](https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.21) · [OpenAlex](https://openalex.org/W4416037841)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/elden-2025-sakinah-ai-mentalqa-comparative-study/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Team Sakinah-AI entered Track 1, Sub-Task 1 of the MentalQA 2025 shared task. The task is to tag Arabic mental health questions with one or more of seven topic categories. The questions are informal and full of dialect. Only 500 annotated items were available: 300 for training, 50 for development and 150 for testing.

The authors compared three strategies directly:
- **Fine-tuning:** Arabic BERT encoders (CAMeL-BERT-DA-Sentiment and AraBERTv2), with hyperparameters tuned by Optuna.
- **Ensembling:** five fold-trained models whose predictions were averaged.
- **Few-shot prompting:** five LLMs served through NVIDIA NIM, each given 3–5 labelled examples. The LLMs were chosen to separate the effects of domain specialisation, architecture and size.

The medical LLM Palmyra-Med-70B scored best, with a weighted F1 of 0.605. Tuned CAMeL-BERT came almost level at 0.597. Ensembling lowered scores for both encoders, and the large general-purpose LLMs performed poorly.

The study is useful practical evidence for small, specialised datasets. In that setting, choosing a well-matched base model and tuning it pays off more than adding scale or combining models. Code and fine-tuned models are publicly released.

## Key points
- **Labels and metrics:** questions can be tagged as Treatment or Diagnosis, as Epidemiology or Anatomy/Physiology, as Provider Choices or Healthy Lifestyle, or as Other, giving seven labels in total. The main metric is weighted F1, with the Jaccard score as a secondary multi-label measure.
- **Fine-tuned models:** CAMeL-BERT scored 0.597 as a tuned single model and 0.537 as a 5-fold ensemble. AraBERTv2 scored 0.543 as a single model and fell to 0.328 as an ensemble.
- **Few-shot LLMs:** Palmyra-Med-70B 0.605, Mixtral-8x22B 0.563, Qwen3-235B 0.325, Gpt-Oss-20B 0.147, Colosseum-355B 0.014. Size did not predict quality.
- **Tuning setup:** Optuna searched learning rates from 1e-5 to 5e-5, batch sizes 8 and 16, and 10–20 epochs, with early stopping on the dev set. The ensembles pooled train and dev data (350 items) for stratified 5-fold training.
- **Single-label collapse:** every LLM tended to return just one label. This happened even though the prompt explicitly asked for all fitting categories and showed multi-label sample answers such as "Final Answer: A,D".

## Interesting ideas
- The candidate LLMs were chosen so that each comparison isolates one factor (domain knowledge, mixture-of-experts versus dense, small versus huge), which makes a result attributable to a cause rather than to a bundle of differences.
- The authors explain the ensemble failure this way: models trained on folds of the same tiny dataset are not diverse, so they learn the same systematic biases, and averaging reinforces those biases instead of cancelling random noise.
- Errors were broken down per category as missed instances (support times one minus recall), which exposes where a model collapses on specific classes even when the overall score looks moderate.
- LLMs ignored an explicit instruction to output several labels and fell back to one, so format compliance has to be measured separately rather than assumed from the prompt.
- A domain-matched model (a medical LLM, or a BERT tuned on dialectal user text) beat much larger generic ones, suggesting that a base model should be chosen by how close its pretraining text is to the target register.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): when comparing witness parsers or LLM teacher tiers, pick pairs that differ in one property only, so we know whether data, architecture or size caused an improvement.
- **I2** (learn): silver data agreed by several witness parsers is only trustworthy if the witnesses err differently, so we should measure their error overlap on gold data and give less weight to agreement between witnesses trained on the same treebank.
- **I3** (tree): reporting missed arcs per UD deprel for the perceptron parser and gparse would show which relations break, instead of hiding them inside one LAS number.
- **I4** (learn): when an LLM teacher annotates multi-valued outputs (several roles or frames for one clause), count how often it returns fewer items than gold, and enforce structure with validation rather than relying on prompt wording.
- **I5** (tree): for gparse, choose or adapt an encoder whose pretraining text resembles our target genres (school texts, fables, tales) instead of defaulting to the largest available model.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Cairo University](https://md.research.elamur.ai/free/en/universities/cairo-university/) — University, Giza, Egypt · 42 works · 1,350 citations
- [University of South Wales](https://md.research.elamur.ai/free/en/universities/university-of-south-wales/) — University, Pontypridd, United Kingdom · 2 works · 2 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,552 works · [💭 Social Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/social-psychology/) · 306 works

🧭 Topics (1): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works

## Cite

Cite as: Fatimah Mohamed Emad Elden and Mumina Ab. Abukar. 2025. Sakinah-AI at MentalQA: A Comparative Study of Few-Shot, Optimized, and Ensemble Methods for Arabic Mental Health Question Classification. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{elden2025sakinah,
  title = {{Sakinah-AI at MentalQA: A Comparative Study of Few-Shot, Optimized, and Ensemble Methods for Arabic Mental Health Question Classification}},
  author = {Fatimah Mohamed Emad Elden and Mumina Ab. Abukar},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.21},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.21}
}
```

- [← previous Safe at the Margins: A General Approach to Safety Alignment in Low-Resource English Languages — A Singlish Case Study](https://md.research.elamur.ai/free/en/papers/lim-2025-safe-margins-general-approach-safety/)
- [next → SATLab at BAREC Shared Task 2025: Optimizing a Language-Independent System for Fine-Grained Readability Assessment](https://md.research.elamur.ai/free/en/papers/bestgen-2025-satlab-barec-shared-task-2025/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/15/)

## References

12 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ashwag Alasmari. 2025. A scoping review of arabic natural language processing for mental health. Healthcare, 13(9):963. [↗](https://doi.org/10.3390/healthcare13090963)
2. Norah S. Alghamdi, Hanan A. H. Mahmoud, Ajith Abraham, Samar A. Alanazi, and Laura García-Hernández. 2020. Predicting depression symptoms in an arabic psychological forum. IEEE Access, 8:57317–57334. [↗](https://doi.org/10.1109/access.2020.2981834)
3. Hassan Alhuzali, Aseel Alasmari, and Hajar Alsaleh. 2024. Mentalqa: An annotated arabic corpus for questions and answers of mental healthcare. IEEE Access. [↗](https://doi.org/10.1109/access.2024.3430068)
4. Hassan Alhuzali and Ashwag Alasmari. 2025. Pre-trained language models for mental health: An empirical study on arabic q&a classification. Healthcare, 13(9):985. [↗](https://doi.org/10.3390/healthcare13090985)
5. Hassan Alhuzali, Farah Shamout, Muhammad Abdul-Mageed, Chaimae Abouzahir, Mouath Abu-Daoud, Ashwag Alasmari, Walid Al-Eisawi, Renad Al-Monef, Ali Alqahtani, Lama Ayash, and 1 others. 2025. AraHealthQA 2025 shared task description paper. arXiv preprint arXiv:2508.20047. [↗](https://arxiv.org/abs/2508.20047)
6. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. [AraBERT: Transformer-based Model for Arabic Language Understanding](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/). In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools, pages 9–15, Marseille, France. European Language Resources Association. [↗](https://doi.org/10.48550/arxiv.2003.00104)
7. Nouhaila Bensalah, Habib Ayad, Abdellah Adib, and Abdelhamid Ibn El Farouk. 2024. Mindwave app: Leveraging ai for mental health support in english and arabic. In 2024 IEEE 12th International Symposium on Signal, Image, Video and Communications (ISIVC), Marrakech, Morocco. IEEE. [find](https://scholar.google.com/scholar?q=Nouhaila+Bensalah%2C+Habib+Ayad%2C+Abdellah+Adib%2C+and+Abdelhamid+Ibn+El+Farouk%2E+2024%2E+Mindwave+app%3A+Leveraging+ai+for+mental+health+support+in+english+and+arabic%2E+In+2024+IEEE+12th+International+Symposium)
8. Go Inoue, Bashar Al-Khafaji, and Nizar Habash. 2021. [The interplay of variant, size, and task type in arabic pre-trained language models](https://md.research.elamur.ai/free/en/papers/inoue-2021-interplay-variant-size-task-type/). In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 15–28.
9. Kiran Kamble and Waseem Alshikh. 2023. Palmyra-med: Instruction-based fine-tuning of llms enhancing medical domain performance. Preprint available on ResearchGate. Under review. [↗](https://doi.org/10.13140/rg.2.2.30939.75046)
10. Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to Information Retrieval. Cambridge University Press, New York, NY, USA. [↗](https://doi.org/10.1017/cbo9780511809071)
11. Marina Sokolova and Guy Lapalme. 2009. A systematic analysis of performance measures for classification tasks. Information Processing & Management, 45(4):427–437. [↗](https://doi.org/10.1016/j.ipm.2009.03.002)
12. Noureldin Zahran, Aya E. Fouda, Radwa J. Hanafy, and Mohammed E. Fouda. 2025. [A comprehensive evaluation of large language models on mental illnesses in arabic context](https://md.research.elamur.ai/free/en/papers/zahran-2025-comprehensive-evaluation-large-language-models/). arXiv preprint. Preprint, arXiv:2501.06859. [↗](https://arxiv.org/abs/2501.06859)

*Updated 1 November 2026*