---
title: "Leveraging BERT Language Model for Arabic — AL-Qurishi 2023"
description: "Transformer classifiers such as BERT accept only a limited input window, and their cost grows quickly with sequence length."
authors: ["Muhammad AL-Qurishi"]
year: 2023
venue: "arXiv:2305.03519"
source: "https://arxiv.org/abs/2305.03519"
doi: "10.48550/arxiv.2305.03519"
arxiv: "2305.03519"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/al-2023-qurishi-leveraging-bert-language-model-arabic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/al-2023-qurishi-leveraging-bert-language-model-arabic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/) / Leveraging BERT Language Model for Arabic Long Document Classification

# Leveraging BERT Language Model for Arabic Long Document Classification

**Authors:** Muhammad AL-Qurishi · **Year:** 2023 · **Venue:** arXiv:2305.03519 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2305.03519) · [PDF](https://arxiv.org/pdf/2305.03519) · [DOI 10.48550/arxiv.2305.03519](https://doi.org/10.48550/arxiv.2305.03519) · [arXiv 2305.03519](https://arxiv.org/abs/2305.03519) · [OpenAlex](https://openalex.org/W4375870129)

**Citations:** 0 citations (openalex-snapshot · 2026-10-14)

## Summary
Transformer classifiers such as BERT accept only a limited input window, and their cost grows quickly with sequence length. This makes long documents hard to handle, and such documents are common in Arabic web, legal, medical and news content. The paper proposes two lightweight BERT-based classifiers for long Arabic texts, both built on AraBERT-v2.

1. **Sentence aggregation.** The document is split into linguistically sound sentences. Each sentence is classified on its own, and the sentence predictions are then pooled per document ID before a final softmax.
2. **Key-sentence selection.** A BERT-based similarity procedure (MMR) picks the most relevant sentences, and only those are fed to the classifier.

Both are compared with Longformer and RoBERT (Recurrence over BERT), each adapted to Arabic and fine-tuned. The test data are a 22-class Mawdoo3 corpus and a merged 8-class Arabic news corpus. The two proposed models match or beat the long-context baselines while using short inputs: aggregation reaches 98.4% macro F1 on news, and MMR reaches 83% on Mawdoo3. The main takeaway is that cheap segment-level processing can rival long-attention architectures.

## Key points
- **Datasets.** Mawdoo3 has 22 classes, with about 1,000 long articles sampled per class. The Arabic News set merges several sources (including Abbas and Smaili 2005) into 8 shared categories, with about 4,000 articles per class.
- **Training setup.**
  - Data split: 80/10/10 train/validation/test.
  - Metric: macro F1.
  - Base model: AraBERT-v2 base (768-dimensional embeddings), trained on Google Colab Pro.
  - Maximum sequence length: 128 for the proposed models and 1024 for Longformer.
- **Mawdoo3 results.** The MMR key-sentence model scores 83% macro F1, very close to the aggregation model and Longformer. RoBERT collapses to 21%.
- **Arabic News results.**

  | Model | Macro F1 |
  |---|---|
  | Sentence aggregation | 98.4% |
  | MMR key-sentence model | 96.2% |
  | Longformer | 96% |
  | RoBERT | 74.4% |

- **Leakage removal.** The opening part of each Mawdoo3 article was removed in preprocessing because it carried easy class giveaways.

## Interesting ideas
- **Classify small units, then vote per document.** Labeling each sentence separately and pooling the labels by document ID avoids long-context models entirely, and it worked best on news. — For Mova (answer): Readers answering questions about a whole fable could score sentence-level evidence first and then combine it, keeping each sentence's contribution visible as an explanation.
- **Pick the salient sentences before heavy processing.** A relevance-plus-diversity selection (MMR) passes only the most informative sentences to the expensive model, on the bet that meaning concentrates in a few places. — For Mova (world): When building the world of a long text, MMR-style ranking could prioritise which sentences get full frame and state extraction first, with the ranking itself logged as an auditable step.
- **Segment along the language's real sentence structure.** Splitting with attention to Arabic sentence boundaries keeps each chunk meaningful instead of cutting at arbitrary token counts. — For Mova (tokens): The en crate's sentence splitter should be judged on whether its segments stay parseable units, since every later stage inherits its boundary errors.
- **Remove label shortcuts before evaluating.** Stripping article headers that revealed the class prevented inflated scores from a trivial cue. — For Mova (learn): Evaluation with negative controls could routinely mask obvious surface cues (titles, keywords) and check that accuracy survives, exposing shortcut learning.
- **Report on more than one dataset.** The ranking of models flipped between the two corpora (RoBERT ranged from 21% to 74%), which shows that conclusions from a single benchmark can mislead. — For Mova (learn): Every parser or reader change should be scored on several heterogeneous test sets, such as tales, school tests and maths problems, before being accepted.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{alqurishi2023leveraging,
  title = {{Leveraging BERT Language Model for Arabic Long Document Classification}},
  author = {AL-Qurishi, Muhammad},
  year = {2023},
  doi = {10.48550/arxiv.2305.03519},
  eprint = {2305.03519},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.03519}
}
```

- [← previous LEVER: Learning to Verify Language-to-Code Generation with Execution](https://md.research.elamur.ai/free/en/papers/ni-2023-lever-learning-verify-language-code/)
- [next → Leveraging Data Collection and Unsupervised Learning for Code-switched Tunisian Arabic Automatic Speech Recognition](https://md.research.elamur.ai/free/en/papers/abdallah-2023-leveraging-data-collection-unsupervised-learning/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/)

## References

24 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abbas, M.; and Berkani, D. 2006. Topic identification by statistical methods for arabic language. WSEAS Transactions on Computers, 5(9): 1908–1913. [find](https://scholar.google.com/scholar?q=Abbas%2C+M%2E%3B+and+Berkani%2C+D%2E+2006%2E+Topic+identification+by+statistical+methods+for+arabic+language%2E+WSEAS+Transactions+on+Computers%2C+5%289%29%3A+1908%E2%80%931913%2E)
2. Abbas, M.; and Smaili, K. 2005. Comparison of topic identification methods for arabic language. In Proceedings of International Conference on Recent Advances in Natural Language Processing, RANLP, 14–17. [find](https://scholar.google.com/scholar?q=Abbas%2C+M%2E%3B+and+Smaili%2C+K%2E+2005%2E+Comparison+of+topic+identification+methods+for+arabic+language%2E+In+Proceedings+of+International+Conference+on+Recent+Advances+in+Natural+Language+Processing%2C+RANLP%2C+14%E2%80%93)
3. Adhikari, A.; Ram, A.; Tang, R.; and Lin, J. 2019. Docbert: Bert for document classification. arXiv preprint arXiv:1904.08398. [↗](https://arxiv.org/abs/1904.08398)
4. Antoun, W.; Baly, F.; and Hajj, H. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. [↗](https://arxiv.org/abs/2003.00104) [our card](https://md.research.elamur.ai/free/en/papers/antoun-2020-arabert-transformer-based-model-arabic/)
5. Beltagy, I.; Peters, M. E.; and Cohan, A. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150. [↗](https://arxiv.org/abs/2004.05150)
6. Carbonell, J.; and Goldstein, J. 1998. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, 335–336. [find](https://scholar.google.com/scholar?q=Carbonell%2C+J%2E%3B+and+Goldstein%2C+J%2E+1998%2E+The+use+of+MMR%2C+diversity%2Dbased+reranking+for+reordering+documents+and+producing+summaries%2E+In+Proceedings+of+the+21st+annual+international+ACM+SIGIR+conference+)
7. Chouigui, A.; Ben Khiroun, O.; and Elayeb, B. 2021. An arabic multi-source news corpus: experimenting on single-document extractive summarization. Arabian Journal for Science and Engineering, 46(4): 3925–3938. [find](https://scholar.google.com/scholar?q=Chouigui%2C+A%2E%3B+Ben+Khiroun%2C+O%2E%3B+and+Elayeb%2C+B%2E+2021%2E+An+arabic+multi%2Dsource+news+corpus%3A+experimenting+on+single%2Ddocument+extractive+summarization%2E+Arabian+Journal+for+Science+and+Engineering%2C+46%284%29%3A+3)
8. Dai, X.; Chalkidis, I.; Darkner, S.; and Elliott, D. 2022. Revisiting Transformer-based Models for Long Document Classification. arXiv preprint arXiv:2204.06683. [↗](https://arxiv.org/abs/2204.06683)
9. Dai, Z.; Yang, Z.; Yang, Y.; Carbonell, J.; Le, Q. V.; and Salakhutdinov, R. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860. [↗](https://arxiv.org/abs/1901.02860)
10. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [↗](https://arxiv.org/abs/1810.04805)
11. Ding, M.; Zhou, C.; Yang, H.; and Tang, J. 2020. Cogltx: Applying bert to long texts. Advances in Neural Information Processing Systems, 33: 12792–12804. *Table 2: Overall results of all models in the long text classification task on Mawdoo3 Dataset* Model Macro F1 Macro Precision Macro Recall Accuracy Our Model Aggregating Our Model-MMR Longformer RoBERT 0.82187 0.82732 0.82347 0.21157 0.82338 0.83162 0.82497 0.19309 0.83049 0.83555 0.83263 0.29507 0.83083 0.83522 0.83291 0.36461 *Table 3: Overall results of all models in the long text classification task on Arabic News Dataset* Model Macro F1 Macro Precision Macro Recall Accuracy Our Model Aggregating Our Model-MMR Longformer RoBERT 0.98411 0.96217 0.95908 0.73062 0.98591 0.96240 0.95956 0.75382 0.98264 0.96206 0.95880 0.75124 0.98434 0.96263 0.95961 0.75142 [find](https://scholar.google.com/scholar?q=Ding%2C+M%2E%3B+Zhou%2C+C%2E%3B+Yang%2C+H%2E%3B+and+Tang%2C+J%2E+2020%2E+Cogltx%3A+Applying+bert+to+long+texts%2E+Advances+in+Neural+Information+Processing+Systems%2C+33%3A+12792%E2%80%9312804%2E+%2ATable+2%3A+Overall+results+of+all+models+in+the)
12. He, J.; Wang, L.; Liu, L.; Feng, J.; and Wu, H. 2019. Long document classification from local word glimpses via recurrent attention learning. IEEE Access, 7: 40707–40718. [find](https://scholar.google.com/scholar?q=He%2C+J%2E%3B+Wang%2C+L%2E%3B+Liu%2C+L%2E%3B+Feng%2C+J%2E%3B+and+Wu%2C+H%2E+2019%2E+Long+document+classification+from+local+word+glimpses+via+recurrent+attention+learning%2E+IEEE+Access%2C+7%3A+40707%E2%80%9340718%2E)
13. Huang, W.; Tao, Z.; Huang, X.; Xiong, L.; and Yu, J. 2021. Hierarchical Self-Attention Hybrid Sparse Networks for Document Classification. Mathematical Problems in Engineering, 2021. [find](https://scholar.google.com/scholar?q=Huang%2C+W%2E%3B+Tao%2C+Z%2E%3B+Huang%2C+X%2E%3B+Xiong%2C+L%2E%3B+and+Yu%2C+J%2E+2021%2E+Hierarchical+Self%2DAttention+Hybrid+Sparse+Networks+for+Document+Classification%2E+Mathematical+Problems+in+Engineering%2C+2021%2E)
14. Pappagari, R.; Zelasko, P.; Villalba, J.; Carmiel, Y.; and Dehak, N. 2019. Hierarchical transformers for long document classification. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 838–844. IEEE. [find](https://scholar.google.com/scholar?q=Pappagari%2C+R%2E%3B+Zelasko%2C+P%2E%3B+Villalba%2C+J%2E%3B+Carmiel%2C+Y%2E%3B+and+Dehak%2C+N%2E+2019%2E+Hierarchical+transformers+for+long+document+classification%2E+In+2019+IEEE+Automatic+Speech+Recognition+and+Understanding+Works)
15. Park, H. H.; Vyas, Y.; and Shah, K. 2022. Efficient Classification of Long Documents Using Transformers. arXiv preprint arXiv:2203.11258. [↗](https://arxiv.org/abs/2203.11258)
16. Rae, J. W.; Potapenko, A.; Jayakumar, S. M.; and Lillicrap, T. P. 2019. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv:1911.05507. [↗](https://arxiv.org/abs/1911.05507) [our card](https://md.research.elamur.ai/free/en/papers/rae-2019-compressive-transformers-long-range-sequence/)
17. Si, Y.; and Roberts, K. 2021. Hierarchical transformer networks for longitudinal clinical document classification. arXiv preprint arXiv:2104.08444. [↗](https://arxiv.org/abs/2104.08444)
18. Sukhbaatar, S.; Grave, E.; Bojanowski, P.; and Joulin, A. 2019. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799. [↗](https://arxiv.org/abs/1905.07799) [our card](https://md.research.elamur.ai/free/en/papers/sukhbaatar-2019-adaptive-attention-span-transformers/)
19. Sun, C.; Qiu, X.; Xu, Y.; and Huang, X. 2019. How to fine-tune bert for text classification? In China national conference on Chinese computational linguistics, 194–206. Springer. [find](https://scholar.google.com/scholar?q=Sun%2C+C%2E%3B+Qiu%2C+X%2E%3B+Xu%2C+Y%2E%3B+and+Huang%2C+X%2E+2019%2E+How+to+fine%2Dtune+bert+for+text+classification%3F+In+China+national+conference+on+Chinese+computational+linguistics%2C+194%E2%80%93206%2E+Springer%2E) [our card](https://md.research.elamur.ai/free/en/papers/sun-2019-fine-tune-bert-text-classification/)
20. Tay, Y.; Dehghani, M.; Abnar, S.; Shen, Y.; Bahri, D.; Pham, P.; Rao, J.; Yang, L.; Ruder, S.; and Metzler, D. 2020a. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006. [↗](https://arxiv.org/abs/2011.04006) [our card](https://md.research.elamur.ai/free/en/papers/tay-2020-long-range-arena-benchmark-efficient/)
21. Tay, Y.; Dehghani, M.; Bahri, D.; and Metzler, D. 2020b. Efficient transformers: A survey. ACM Computing Surveys (CSUR). [find](https://scholar.google.com/scholar?q=Tay%2C+Y%2E%3B+Dehghani%2C+M%2E%3B+Bahri%2C+D%2E%3B+and+Metzler%2C+D%2E+2020b%2E+Efficient+transformers%3A+A+survey%2E+ACM+Computing+Surveys+%28CSUR%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/tay-2020-efficient-transformers-survey/)
22. Wagh, V.; Khandve, S.; Joshi, I.; Wani, A.; Kale, G.; and Joshi, R. 2021. Comparative study of long document classification. In TENCON 2021-2021 IEEE Region 10 Conference (TENCON), 732–737. IEEE. [find](https://scholar.google.com/scholar?q=Wagh%2C+V%2E%3B+Khandve%2C+S%2E%3B+Joshi%2C+I%2E%3B+Wani%2C+A%2E%3B+Kale%2C+G%2E%3B+and+Joshi%2C+R%2E+2021%2E+Comparative+study+of+long+document+classification%2E+In+TENCON+2021%2D2021+IEEE+Region+10+Conference+%28TENCON%29%2C+732%E2%80%93737%2E+IEEE%2E)
23. Wang, Z.; Ng, P.; Ma, X.; Nallapati, R.; and Xiang, B. 2019. Multi-passage bert: A globally normalized bert model for open-domain question answering. arXiv preprint arXiv:1908.08167. [↗](https://arxiv.org/abs/1908.08167) [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-multi-passage-bert-globally-normalized/)
24. Wang, Z.; Wang, C.; Zhang, H.; Duan, Z.; Zhou, M.; and Chen, B. 2020. Learning dynamic hierarchical topic graph with graph convolutional network for document classification. In International Conference on Artificial Intelligence and Statistics, 3959–3969. PMLR. [find](https://scholar.google.com/scholar?q=Wang%2C+Z%2E%3B+Wang%2C+C%2E%3B+Zhang%2C+H%2E%3B+Duan%2C+Z%2E%3B+Zhou%2C+M%2E%3B+and+Chen%2C+B%2E+2020%2E+Learning+dynamic+hierarchical+topic+graph+with+graph+convolutional+network+for+document+classification%2E+In+International+Conferenc)