---
title: "Domain-Adaptive Pre-Training for Arabic — Alyami 2025"
description: "The paper looks at Arabic aspect-based sentiment analysis (ABSA), where labelled data is scarce."
authors: ["Salha Alyami", "Amani Jamal", "Areej Alhothali"]
year: 2025
venue: "arXiv:2509.16788"
source: "https://arxiv.org/abs/2509.16788"
doi: "10.48550/arxiv.2509.16788"
arxiv: "2509.16788"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/alyami-2025-domain-adaptive-pre-training-arabic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/alyami-2025-domain-adaptive-pre-training-arabic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/) / Domain-Adaptive Pre-Training for Arabic Aspect-Based Sentiment Analysis: A Comparative Study of Domain Adaptation and Fine-Tuning Strategies

# Domain-Adaptive Pre-Training for Arabic Aspect-Based Sentiment Analysis: A Comparative Study of Domain Adaptation and Fine-Tuning Strategies

**Authors:** Salha Alyami, Amani Jamal, [Areej Alhothali](https://md.research.elamur.ai/free/en/authors/areej-alhothali/) · **Year:** 2025 · **Venue:** arXiv:2509.16788 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2509.16788) · [PDF](https://arxiv.org/pdf/2509.16788) · [DOI 10.48550/arxiv.2509.16788](https://doi.org/10.48550/arxiv.2509.16788) · [arXiv 2509.16788](https://arxiv.org/abs/2509.16788) · [OpenAlex](https://openalex.org/W4414744727)

**Citations:** 0 citations (openalex-snapshot · 2026-10-14)

## Summary
The paper looks at Arabic aspect-based sentiment analysis (ABSA), where labelled data is scarce. Arabic BERT-style models are mostly pre-trained on factual text such as Wikipedia, which may not suit opinion-heavy review language. The authors present what they describe as the first systematic study of domain-adaptive pre-training for Arabic ABSA. They continue pre-training several Arabic contextualized models on different in-domain and related corpora. They then test the models on two tasks using the SemEval-2016 Arabic hotel reviews benchmark: identifying the sentiment expressed toward each aspect (ASC) and pulling out the spans that name what an opinion is about (OTE).

They compare three fine-tuning regimes: feature extraction, full fine-tuning and adapters. They also try multi-task learning with adapter fusion and losses designed for class imbalance. Further analysis covers vocabulary overlap, dialect mix and adaptation-corpus size.

The gains from in-domain adaptation are small. Adapters come close to full fine-tuning while training far fewer parameters. An explainability-driven error analysis finds problems both in model behaviour and in the gold annotations. The authors conclude that syntax- and semantics-aware models, such as GCNs, are needed for long-distance aspect–opinion links.

## Key points
- The tasks are ASC (sentiment toward a given aspect or category) and OTE extraction, framed as BIO tagging with B-OTE / I-OTE / O labels. Both use the SemEval-2016 Arabic hotel reviews dataset.
- Domain-adaptive pre-training is tested across multiple adaptation corpora and Arabic BERT models. Results are analysed against:
  - vocabulary overlap between the model vocabulary and the task data,
  - dialect distribution,
  - adaptation-corpus size.
- Adapter-based fine-tuning trades a small amount of accuracy for much lower training cost. Adapter fusion is explored for multi-task setups.
- The error analysis inspects the 50 highest-loss samples using Integrated Gradients (Transformers Interpret / Captum) token attributions.
  - **ASC failures:** wrong gold sentiment, misread contrastive markers, a positivity bias toward early terms, conflicting opinions, and subword tokenization problems.
  - **OTE failures:** mislabeled targets, confusion over syntactic roles, multi-word expressions, and shallow heuristics.
- The related work notes that earlier Arabic ABSA relied mostly on SVM (for ASC), CRF (for OTE) and RNN variants. Transformer-based prior work includes Bensoltane and Zaki (2022), who used AraBERT with Bi-GRU and CRF.

## Interesting ideas
- Ranking examples by training loss and running token attributions on the worst ones separates annotation mistakes from genuine model weaknesses, so one inspection serves both data cleaning and diagnosis. — For Mova (learn): Apply the same top-loss audit to silver data agreed by witness parsers, flagging cases where the "agreed" label itself looks wrong before it feeds training or repair-rule induction.
- Treating the overlap between a model's subword vocabulary and the target dataset as an explanatory variable helps explain why adaptation helps or fails, beyond just reporting scores. — For Mova (tokens): Measure how often aspect-like content words are split into many wordpieces in each genre, and use that as a predictor of where gparse inputs and taggers will be fragile.
- Small adapters per domain or task reach near full fine-tuning quality at a fraction of the trainable parameters, and fusion lets several such modules be combined. — For Mova (tree): Train lightweight genre adapters for gparse (fables, school tests, maths word problems) instead of retraining the full transformer parser for each text type.
- The error taxonomy names concrete linguistic traps: contrastive connectives, early-position positivity bias, and mixed opinions in one sentence. These make a ready checklist for targeted test sets. — For Mova (frames): Build negative-control probes in prag where "but"/"although" clauses flip evaluative stance, and check that frame-level attitude attaches to the correct target rather than to the first mention.
- The authors argue that opinion targets are best found through syntactic relations rather than surface heuristics, which points to structure-aware modelling of who evaluates what. — For Mova (world): Derive evaluated entities and their attributed states from UD arcs (e.g., the nsubj of an evaluative adjective), recording opinions as explainable entity states instead of bag-of-words cues.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{alyami2025domain,
  title = {{Domain-Adaptive Pre-Training for Arabic Aspect-Based Sentiment Analysis: A Comparative Study of Domain Adaptation and Fine-Tuning Strategies}},
  author = {Alyami, Salha and Jamal, Amani and Alhothali, Areej},
  year = {2025},
  doi = {10.48550/arxiv.2509.16788},
  eprint = {2509.16788},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2509.16788}
}
```

- [← previous Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://md.research.elamur.ai/free/en/papers/yue-2025-reinforcement-learning-really-incentivize-reasoning/)
- [next → EHSAN: Leveraging ChatGPT in a Hybrid Framework for Arabic Aspect-Based Sentiment Analysis in Healthcare](https://md.research.elamur.ai/free/en/papers/alamoudi-2025-ehsan-leveraging-chatgpt-hybrid-framework/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/)

## References

30 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Accuracy (%) INSIGHT-1 Ruder et al. (2016b) 82.72 [find](https://scholar.google.com/scholar?q=Accuracy+%28%25%29+INSIGHT%2D1+Ruder+et+al%2E+%282016b%29+82%2E72)
2. H-LSTM Ruder et al. (2016a) 82.90 [find](https://scholar.google.com/scholar?q=H%2DLSTM+Ruder+et+al%2E+%282016a%29+82%2E90)
3. AB-LSTM-PC Al-Smadi et al. (2019) 82.60 [find](https://scholar.google.com/scholar?q=AB%2DLSTM%2DPC+Al%2DSmadi+et+al%2E+%282019%29+82%2E60)
4. IAN-BGRU Al-Dabet et al. (2021) 87.31 [find](https://scholar.google.com/scholar?q=IAN%2DBGRU+Al%2DDabet+et+al%2E+%282021%29+87%2E31)
5. MBRA Abdelgwad et al. (2021) 83.98 [find](https://scholar.google.com/scholar?q=MBRA+Abdelgwad+et+al%2E+%282021%29+83%2E98)
6. AraBERT-linear-pair Abdelgwad et al. (2022) 89.51 [find](https://scholar.google.com/scholar?q=AraBERT%2Dlinear%2Dpair+Abdelgwad+et+al%2E+%282022%29+89%2E51)
7. Dialect Normalization+Arabic-BERT Chennafi et al. (2022) 84.65 [find](https://scholar.google.com/scholar?q=Dialect+Normalization%2BArabic%2DBERT+Chennafi+et+al%2E+%282022%29+84%2E65)
8. ABNM (Abdelgwad et al., 2021): the researchers used an encoder–encoder with an attention mechanism between them followed by CRF. [find](https://scholar.google.com/scholar?q=ABNM+%28Abdelgwad+et+al%2E%2C+2021%29%3A+the+researchers+used+an+encoder%E2%80%93encoder+with+an+attention+mechanism+between+them+followed+by+CRF%2E)
9. BF-BiLSTM-CRF (Fadel et al., 2022): the authors concatenated Arabic Flair string embedding and AraBERT embedding, which fed into Bi-LSTM layers followed by CRF. [find](https://scholar.google.com/scholar?q=BF%2DBiLSTM%2DCRF+%28Fadel+et+al%2E%2C+2022%29%3A+the+authors+concatenated+Arabic+Flair+string+embedding+and+AraBERT+embedding%2C+which+fed+into+Bi%2DLSTM+layers+followed+by+CRF%2E)
10. Table 17 shows the comparison of our models in terms of micro-F1 to different OTE extraction deep learning models on the SemEval-2016 Arabic hotel reviews dataset. [find](https://scholar.google.com/scholar?q=Table+17+shows+the+comparison+of+our+models+in+terms+of+micro%2DF1+to+different+OTE+extraction+deep+learning+models+on+the+SemEval%2D2016+Arabic+hotel+reviews+dataset%2E)
11. Table 17: Performance comparison of Opinion Target Expression Extraction models on the SemEval-2016 Arabic Hotel Review Dataset. The highest F1 score is highlighted in bold. [find](https://scholar.google.com/scholar?q=Table+17%3A+Performance+comparison+of+Opinion+Target+Expression+Extraction+models+on+the+SemEval%2D2016+Arabic+Hotel+Review+Dataset%2E+The+highest+F1+score+is+highlighted+in+bold%2E)
12. RNN-Softmax Al-Smadi et al. (2018) 48.00 [find](https://scholar.google.com/scholar?q=RNN%2DSoftmax+Al%2DSmadi+et+al%2E+%282018%29+48%2E00)
13. Bi-LSTM-CRF Al-Smadi et al. (2019) 69.98 [find](https://scholar.google.com/scholar?q=Bi%2DLSTM%2DCRF+Al%2DSmadi+et+al%2E+%282019%29+69%2E98)
14. CNN-Bi-GRU-CRF Al-Dabet et al. (2020) 72.83 [find](https://scholar.google.com/scholar?q=CNN%2DBi%2DGRU%2DCRF+Al%2DDabet+et+al%2E+%282020%29+72%2E83)
15. ABNM Abdelgwad et al. (2021) 70.67 [find](https://scholar.google.com/scholar?q=ABNM+Abdelgwad+et+al%2E+%282021%29+70%2E67)
16. BF-BiLSTM-CRF Fadel et al. (2022) 79.70 [find](https://scholar.google.com/scholar?q=BF%2DBiLSTM%2DCRF+Fadel+et+al%2E+%282022%29+79%2E70)
17. Ours (In-DAPT CAMeLBERT-MSA Full fine-tuning) 77.02 Ours (In-DAPT CAMeLBERT-MSA Lang+Task Stacked Adapter) 76.31 Ours (In-DAPT CAMeLBERT-MSA Full fine-tuning + Focal Loss) 76.42 6. Discussion 6.1. Overview of Findings and Fine-tuning Strategy Our study examined how domain-adaptive and task-adaptive pre-training affect Arabic ABSA tasks, specifically ASC and OTE extraction. Using Arabic hotel reviews, we tested several pre-trained Arabic BERT models and fine-tuning strategies. The results showed that CAMeLBERT-MSA, QARiB, and MARBERTv2 outperformed other Arabic BERT models. Several factors may explain their strong performance. For example, CAMeLBERT-MSA is trained only on MSA data, which enhances its ability to represent this dialect. This is especially relevant since MSA makes up 65% of the SemEval-2016 [find](https://scholar.google.com/scholar?q=Ours+%28In%2DDAPT+CAMeLBERT%2DMSA+Full+fine%2Dtuning%29+77%2E02+Ours+%28In%2DDAPT+CAMeLBERT%2DMSA+Lang%2BTask+Stacked+Adapter%29+76%2E31+Ours+%28In%2DDAPT+CAMeLBERT%2DMSA+Full+fine%2Dtuning+%2B+Focal+Loss%29+76%2E42+6%2E+Discussion+6%2E1%2E+Ove)
18. Arabic hotel reviews dataset (see Figure 2). On the other hand, QARiB and MARBERTv2 benefit from more vast and mixed datasets, including various dialects, providing a more comprehensive linguistic exposure. Their performance is further improved by undergoing a larger number of training iterations and using larger vocabulary dictionaries (see Table A.20). Notably, the fully fine-tuned Arabic BERT models outperformed those that used BERT as a feature extractor with all layers frozen. Thus, we conducted our subsequent experiments using full fine-tuning and adapters. These results suggest that pre-trained model parameters updates are necessary for these models to accurately comprehend ABSA tasks, which aligns with the findings of Abdelgwad et al. (2022) and Fadel et al. [find](https://scholar.google.com/scholar?q=Arabic+hotel+reviews+dataset+%28see+Figure+2%29%2E+On+the+other+hand%2C+QARiB+and+MARBERTv2+benefit+from+more+vast+and+mixed+datasets%2C+including+various+dialects%2C+providing+a+more+comprehensive+linguistic+exp)
19. (2022). 6.2. Effects of Domain- and Task-Adaptive Pre-training The results show that for ASC, all three Arabic BERT models with In-DAPT outperform TAPT, Out-DAPT, and No-DAPT in both accuracy and macro-F1. In OTE extraction, In-DAPT+TAPT achieves the best performance for QARiB and MARBERTv2. Notably, across all adaptation settings, In-DAPT CAMeLBERT-MSA outperformed both QARiB and MARBERTv2 in both tasks, demonstrating its effectiveness in domain adaptation. [find](https://scholar.google.com/scholar?q=%282022%29%2E+6%2E2%2E+Effects+of+Domain%2D+and+Task%2DAdaptive+Pre%2Dtraining+The+results+show+that+for+ASC%2C+all+three+Arabic+BERT+models+with+In%2DDAPT+outperform+TAPT%2C+Out%2DDAPT%2C+and+No%2DDAPT+in+both+accuracy+and+macr)
20. The strong performance of CAMeLBERT-MSA might be linked to two factors. First, CAMeLBERT-MSA is pre-trained on MSA, which constitutes the largest portion of SemEval-2016 and HARD datasets (see Figure 2). Second, CAMeLBERT-MSA has the fewest learnable parameters among the tested Arabic BERT pre-trained models, which makes it less prone to overfitting (see Table A.20). [find](https://scholar.google.com/scholar?q=The+strong+performance+of+CAMeLBERT%2DMSA+might+be+linked+to+two+factors%2E+First%2C+CAMeLBERT%2DMSA+is+pre%2Dtrained+on+MSA%2C+which+constitutes+the+largest+portion+of+SemEval%2D2016+and+HARD+datasets+%28see+Figure+)
21. CAMeLBERT-MSA with In-DAPT achieved the highest improvement over the baseline No-DAPT, with an accuracy increase of 0.69% and a Macro-F1 increase of 6.44% in ASC, and a Micro-F1 increase of 0.67% in OTE. This improvement may be attributed to CAMeLBERT-MSA’s low vocabulary overlap (7%–9%) with the HARD and SemEval-2016 Arabic hotel review datasets (Figure 3). This observation aligns with the heuristic proposed by Gururangan et al. (2020), suggesting that lower vocabulary overlap allows models to benefit more from domain adaptation by learning essential, previously unseen domain-specific vocabulary. 6.3. Out-of-Domain TAPT/OutDAPT and Domain Relevance The impact of adaptive pre-training with Out-DAPT and Out-DAPT+TAPT corpora significantly reduced the performance of CAMeLBERT-MSA, QARiB, and MARBERTv2 across both tasks compared to baseline No-DAPT. This aligns with the findings of Gururangan et al. (2020), highlighting the effectiveness of using in-domain data for adaptive pretraining. Unlike the findings of Gururangan et al., where TAPT improved performance in the review-level sentiment analysis, our results indicated a decline in the ASC task. This decrease may be attributed to data imbalance or the limited size of the TAPT corpus, as evidenced by a more significant drop in macro-F1 compared to accuracy (see Figure 4). In contrast, for OTE extraction, TAPT improved performance for all models except MARBERTv2, which experienced only a slight reduction. This suggests that OTE extraction is more sensitive to domain relevance than sentiment classification, aligning with the observations by Xu et al. [find](https://scholar.google.com/scholar?q=CAMeLBERT%2DMSA+with+In%2DDAPT+achieved+the+highest+improvement+over+the+baseline+No%2DDAPT%2C+with+an+accuracy+increase+of+0%2E69%25+and+a+Macro%2DF1+increase+of+6%2E44%25+in+ASC%2C+and+a+Micro%2DF1+increase+of+0%2E67%25+in+O)
22. Xu et al. (2019, 2020). 6.4. Corpus Size vs. Domain Relevance (Ablation) [find](https://scholar.google.com/scholar?q=Xu+et+al%2E+%282019%2C+2020%29%2E+6%2E4%2E+Corpus+Size+vs%2E+Domain+Relevance+%28Ablation%29)
23. To determine whether the superior performance of In-DAPT was due to domain relevance rather than corpus size, we performed adaptive pre-training on subsamples of the In-DAPT corpus, matching the sizes of the TAPT and Out-DAPT datasets (4,248 and 60,084 samples, respectively). At the same sample size, ASC (see Figure 14), both TAPT and Out-DAPT outperformed In-DAPT . This suggests that even out-of-domain adaptive pretraining can be beneficial due to cross-domain sentiment similarities. However, for OTE extraction, only TAPT outperformed In-DAPT, while Out-DAPT underperformed In-DAPT, indicating that domain relevance is crucial, as OTE terms vary by domain Xu et al. (2019, 2020). Notably, the largest In-DAPT corpus, with 399,839 samples, achieved the best overall performance, suggesting that while domain relevance is important—especially for domain-sensitive tasks like OTE extrac-In-Domain Adaptation Performance Metrics Across Various Sample Sizes Adaptation Method 88.10% 88.86% 89.67% 87.86% 87.63% In-DAPT 4248 Samples In-DAPT 60084 Samples In-DAPT 399839 All-samples TAPT Out-DAPT Max-acc 74.78% 74.90% 75.16% 67.66% 73.39% Performance Metric Max-M-F1 87.54% 87.84% 88.26% 87.29% 87.15% AVG-acc 70.01% 72.89% 73.77% 64.28% 67.19% AVG-M-F1 [find](https://scholar.google.com/scholar?q=To+determine+whether+the+superior+performance+of+In%2DDAPT+was+due+to+domain+relevance+rather+than+corpus+size%2C+we+performed+adaptive+pre%2Dtraining+on+subsamples+of+the+In%2DDAPT+corpus%2C+matching+the+sizes)
24. Metric Value (%) Figure 14: Comparison of average and maximum accuracy, and macro F1 scores for aspect-sentiment classification across in-domain models (In-DAPT 4248, 60084, 399839 samples), TAPT, and Out-DAPT. In-Domain Adaptation Performance Metrics Across Various Sample Sizes Adaptation Method In-DAPT 4248 Samples In-DAPT 60084 Samples In-DAPT 399839 All-samples TAPT Out-DAPT 76.13% 76.58% Max-F1 77.02% 76.48% Performance Metric 76.56% 75.99% 76.47% AVG-F1 76.47% 76.29% 76.22% Metric Value (%) Figure 15: Comparison of average and maximum micro F1 scores for opinion target extraction across in-domain models (In-DAPT 4248, 60084, 399839 samples), TAPT, and Out-DAPT. tion—, increasing corpus size can also improve performance in both tasks. 6.5. Statistical Significance and Training Choices Our results show that In-DAPT techniques led to performance improvements in both ASC and OTE extraction, but these gains were not statistically significant (p-values: 0.18–0.54 for accuracy, 0.07–0.81 for macro-F1 in ASC; 0.06–0.55 for micro-F1 in OTE). This is likely due to the small, imbalanced end-task dataset, which limits model generalization. To address this, we employed adapter-based fine-tuning, class weighting, and focal loss. For OTE, we also experimented with CRF, BiLSTM, and BiGRU layers in the CAMeLBERT-MSA model. Although these configurations did not improve overall F1, the BiGRU+Softmax setup increased recall to 79.85%, improving span detection at the expense of precision. 6.6. Adapters, Fusion, and Loss Functions For OTE extraction, In-DAPT significantly improved performance, while most TAPT adapters had a negative impact, contradicting the CAMeLBERT-MSA TAPT’s slight performance gain. Figure 6 shows that combining In-DAPT or TAPT language adapters with task adapters results in lower accuracy on the ASC task compared to using task adapters alone (No-DAPT). However, most models achieved higher macro-F1 scores, except for In-DAPT with the Pfeiffer adapter, suggesting better performance across all classes. This indicates that domain-adapted models may improve the balance of classes. In the OTE task (Figure 7), In-DAPT adapters improved performance, likely because they helped the model better understand domain-specific expressions. However, most TAPT adapters led to lower performance than No-DAPT. These findings partly contrast with full fine-tuned TAPT CAMeLBERT-MSA’s , which showed a slight improvement (Figure 5). Our findings show that adapter fusion outperforms single-task adapters in the ASC task; however, it decreased performance in the OTE extraction task (see Figure 12). A possible reason for this could be that ASC benefits from shared representations that utilize broader contextual knowledge, improving the accuracy of the classification. In contrast, OTE tasks depend on task-specific features that can be negatively affected by generalized representations and feature dilution introduced through multi-task learning, as noted in previous research (Yang et al., 2021; Varia et al., 2023). [find](https://scholar.google.com/scholar?q=Metric+Value+%28%25%29+Figure+14%3A+Comparison+of+average+and+maximum+accuracy%2C+and+macro+F1+scores+for+aspect%2Dsentiment+classification+across+in%2Ddomain+models+%28In%2DDAPT+4248%2C+60084%2C+399839+samples%29%2C+TAPT%2C+and)
25. Integrating focal loss and class weighting improved sensitivity to minority classes, particularly improving recall for both ASC and OTE extraction tasks, achieving the highest accuracy of 89.86% on the ASC task (see Figure 13). Although precision slightly decreased, this deliberate trade-off prioritized recall is crucial for imbalanced datasets. These results demonstrate the efficacy of advanced loss functions and weighting techniques in handling skewed class distributions. 6.7. Comparison to Prior Work Our two top-performing ASC models, based on In-DAPT CAMeLBERT-MSA and optimized with focal and cross-entropy losses, outperform previous deep learning approaches on the SemEval-2016 Arabic Hotel Reviews dataset. AraBERT-linear-pair by Abdelgwad et al. (2022) follows closely with an accuracy of 89.51%, which is 0.35% and 0.16% lower than our models. Unlike their approach, which uses explicit OTE-review pairs as BERT inputs and excludes implicit aspects, our models leverage aspect-category–review pairs. This enables us to address both explicit and implicit aspects, which are commonly found in user reviews. Unlike many previous models, our approach does not add additional layers or components beyond the base transformer. We fine-tune the pretrained model directly, without applying complex architectures such as memory networks (Abdelgwad et al., 2021) or attention-based interaction modules (Al-Dabet et al., 2021). Even with this simpler setup, our models outperform MBRA (Abdelgwad et al., 2021), IAN-BiGRU (Al-Dabet et al., 2021), and earlier CNN-, RNN-, and BiLSTM-based models (Ruder et al., 2016b,a; Al-Smadi et al., 2018, 2019; Chennafi et al., 2022). [find](https://scholar.google.com/scholar?q=Integrating+focal+loss+and+class+weighting+improved+sensitivity+to+minority+classes%2C+particularly+improving+recall+for+both+ASC+and+OTE+extraction+tasks%2C+achieving+the+highest+accuracy+of+89%2E86%25+on+th)
26. For the OTE extraction task, our best model based on full fine-tuning of In-DAPT CAMeLBERT-MSA ranks second to the BF-BiLSTM-CRF model (Fadel et al., 2022), while our adapter-based variants still achieve higher performance than models such as CNN-BiGRU-CRF (Al-Dabet et al., 2020), ABNM (Abdelgwad et al., 2021), BiLSTM-CRF (Al-Smadi et al., 2019), and RNN-Softmax (Al-Smadi et al., 2018). These results highlight the importance of contextualized embeddings adapted to the domain and balanced loss functions for enhanced performance in Arabic ABSA. 6.8. Error Analysis To investigate the limited performance of our model, we analyzed predictions from the best-performing In-DAPT CAMeL-BERT, focusing on cases where predictions did not match the correct labels in the ASC and OTE tasks. We then analyzed the 50 samples with the highest loss, as computed by the model’s loss function, to identify where the model struggled and why. [find](https://scholar.google.com/scholar?q=For+the+OTE+extraction+task%2C+our+best+model+based+on+full+fine%2Dtuning+of+In%2DDAPT+CAMeLBERT%2DMSA+ranks+second+to+the+BF%2DBiLSTM%2DCRF+model+%28Fadel+et+al%2E%2C+2022%29%2C+while+our+adapter%2Dbased+variants+still+achi)
27. We used the Transformers Interpret library 5 —an explainability tool for Hugging Face Transformers built on PyTorch’s Captum—and applied Integrated Gradients (IG) to identify which words most influenced misclassifications. We chose IG because it is based on well-established theoretical principles: Sensitivity, which ensures that important input features receive non-zero attribution, and Implementation Invariance, which guarantees consistent attributions across models with identical behavior, regardless of internal differences (Sundararajan et al., 2017). 6.8.1. Aspect Sentiment Classification (ASC) (see Table 18) [find](https://scholar.google.com/scholar?q=We+used+the+Transformers+Interpret+library+5+%E2%80%94an+explainability+tool+for+Hugging+Face+Transformers+built+on+PyTorch%E2%80%99s+Captum%E2%80%94and+applied+Integrated+Gradients+%28IG%29+to+identify+which+words+most+influenc)
28. The ASC model showed recurring errors. First, annotation mistakes: explicit sentiment often conflicted with the assigned   labels. For example, negative terms like “ éÖß Y ¯ ” (old) and “  ËAK ð ” (dilapidated) in Ex1 were labeled as neutral; positive éJ . expressions such as “ éª K@P ” (wonderful) in Ex7 and “ á K. X ñÓ ” (polite) in Ex10 were also tagged as neutral; and strongly  negative words like “ é J ƒ ” (bad) in Ex12 and “ ZA¢ k @ ” (mistakes) in Ex13 were incorrectly marked as positive. Second, the model missed contrastive shifts signaled by “ áºË ” (but) and “ áºËð ” (however), leading to errors in mixed sentiment: in Ex11, it ignored the positive “ Y J k. Pñ ¢ ¯ ” (good breakfast) after ... é  J  ƒ ” (bad... expensive), and negative cues like “ A ¢ ëA K. in Ex7 and Ex6 it overlooked key contrasts in service and location. Third, a positivity bias gave excess weight to early positive terms (e.g., “ © K@P ” in Ex2, “ ÉÔg. @ ” in Ex6) while downplaying later negatives such as “ Zù ¢. ” (slow) in Ex9. Fourth, the model misattributed sentiment across aspects: general hotel praise in Ex2 overrode negative service feedback, and scenic location terms in Ex6 obscured access issues. Fifth, when conflicting opinions appeared within the same aspect, it defaulted to the 5 [https://github.com/cdpierse/transformers-interpret](https://github.com/cdpierse/transformers-interpret) dominant tone: Ex8 prioritized “ i.« QÓ ” (annoying) over “ é ®J ¢  ” (clean), Ex11 effectively averaged opposing cues, and Ex7 favored positive spatial descriptions over slow service. Finally, subword tokenization introduced noise: partial tokens such as  “## P@ ” in Ex5 and “## é ®‚Ë ” in Ex4 led to context-insensitive interpretations. 6.8.2. Opinion Target Expression (OTE) Extraction (see Table 19) Errors in the OTE extraction model mainly stem from shallow token-level heuristics rather than syntactic and semantic analysis. The model often confused roles, misidentifying sub- jects or locations as targets: in Ex1 it tagged “ ‡K Q ¯ ” (team)  instead of the true target “ ÈAJ. ®JƒB@ ” (reception), and in Ex2 and   Ex3 it treated “ ÐQmÌ '@ ” (the sanctuary) and “ Ð@Q Ë@ é¢m× ” (tram station) as targets. It also over-relied on the definite article heuris- tic (“ È@ ”, “the”), producing false positives such as “ é KñºÊJ. Ë@ ” (the balcony) in Ex6. Multi-word expressions—especially Idafa constructions—were not handled well, as with “ øñ J‚Ó É¿ B@ ” (food quality) in Ex4 and “ á kA‚Ë@ ZAÖÏ @ PAK. ” (hot water bar) in Ex5. Morphologically rich words were over-segmented,   e.g., “ èP@ Y ¯ ” (filthiness) in Ex7 was split into meaningless subtokens. Finally, annotation inconsistencies added noise: “  g ” (bathrooms) was labeled a non-target in Ex8, while HAÓAÔ the conjunction “ ð ” (and) was incorrectly included as part of a target in Ex9. 7. Limitations Our models achieved promising results in Arabic ABSAfo-cused on hotel reviews, but there are different limitations to consider. First, our findings may not generalize beyond the hotel domain or to other languages. We specifically explored ASC and OTE extraction tasks separately, limiting broader applicability. Additionally, we faced challenges with computational resources, which restricted our ability to experiment with multiple pre-trained BERT models and complete hyper-parameter  tuning using Optuna. This led us to rely on pre-trained Arabic models and original BERT hyperparameters, potentially affecting the results. Finally, the scarcity of publicly available code for Arabic ABSA posed significant obstacles, forcing us to develop our own code from scratch, which consumed considerable time and resources. 8. Conclusion and Future Work This study evaluated the impact of domain-adaptive pre-training on Arabic ABSA, focusing on ASC and OTE tasks. We explored various fine-tuning strategies on pre-trained Arabic BERT models using multiple adaptation corpora. In-DAPT CAMeLBERT-MSA achieved the highest performance, reaching 89.86% accuracy in ASC and 77.02% micro-F1 in OTE. While full fine-tuning produced the best results, adapter-based methods provided a strong trade-off between efficiency and performance. Based on our error analysis, future work in ASC should consider integrating syntax- and semantics-enhanced GCN models—such as Syntactic Dependency GCNs (SD-GCN) or Syntactic and Semantic Enhanced GCN (SSEGCN)—to better model long-distance dependencies, aspect–opinion alignment, and contrastive cues highlighted by errors in our model (Zhao et al., 2020b; Zhang et al., 2022). For Opinion Target Expression Extraction, utilizing dependency-aware graph models like attention-based relational GCNs (ARGCN) that propagate dependency-informed attention from target to candidate spans, can handle model misclassification of irrelevant nouns and improve multi-word/Idafa span detection (Jiang et al., 2021; Pouran Ben Veyseh et al., 2020). [↗](https://github.com/cdpierse/transformers-interpret)
29. Future work should extend to other domains and dialects, and explore complex ABSA tasks such as triplet or quadruple extraction Zhang et al. (2021). Recent advances in instruction tuning Scaria et al. (2023) and in-context learning with large language models (LLMs)—such as AceGPT-7B, GPT-4o, and Gemini 1.5—offer promising directions for zero/few-shot adaptation to Arabic ABSA. For example, Khaled et al. (2024) has shown that AceGPT-7B achieves state-of-the-art performance on sentence- and review-level sentiment classification tasks using a Retrieval-Augmented Generation (RAG) approach across benchmark datasets. . Moreover, future work may compare adapter variants such as Prefix-Tuning Li and Liang (2021) and Table 18: Selected ASC examples from the 50 highest-loss cases, listing gold label (True), model prediction+probability (Predicted), attribution polarity (Attr.), IG score (Score), and token-level Word Importance. Highlights are Integrated Gradients attributions (Transformers Interpret/Captum): green = positive, red = negative; darker shade = stronger contribution. [find](https://scholar.google.com/scholar?q=Future+work+should+extend+to+other+domains+and+dialects%2C+and+explore+complex+ABSA+tasks+such+as+triplet+or+quadruple+extraction+Zhang+et+al%2E+%282021%29%2E+Recent+advances+in+instruction+tuning+Scaria+et+al%2E)
30. LoRA Hu et al. (2021). Finally, we encourage the development of open-source tools and datasets to improve reproducibility and accelerate progress in Arabic ABSA. [find](https://scholar.google.com/scholar?q=LoRA+Hu+et+al%2E+%282021%29%2E+Finally%2C+we+encourage+the+development+of+open%2Dsource+tools+and+datasets+to+improve+reproducibility+and+accelerate+progress+in+Arabic+ABSA%2E)