--- title: "BERT Fine-tuning For Arabic Text — Elmadani 2020" description: "Arabic summarization research has mostly relied on sentence-scoring extractive methods, and there are few benchmark corpora." authors: ["Khalid N. Elmadani", "Mukhtar Elgezouli", "Anas Showk"] year: 2020 venue: "arXiv:2004.14135" source: "https://arxiv.org/abs/2004.14135" doi: "10.48550/arxiv.2004.14135" arxiv: "2004.14135" licence: "arXiv — non-exclusive license to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/elmadani-2020-bert-fine-tuning-arabic-text/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/elmadani-2020-bert-fine-tuning-arabic-text/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/) / BERT Fine-tuning For Arabic Text Summarization # BERT Fine-tuning For Arabic Text Summarization **Authors:** [Khalid N. Elmadani](https://md.research.elamur.ai/free/en/authors/khalid-n-elmadani/), Mukhtar Elgezouli, Anas Showk · **Year:** 2020 · **Venue:** arXiv:2004.14135 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2004.14135) · [PDF](https://arxiv.org/pdf/2004.14135) · [DOI 10.48550/arxiv.2004.14135](https://doi.org/10.48550/arxiv.2004.14135) · [arXiv 2004.14135](https://arxiv.org/abs/2004.14135) · [OpenAlex](https://openalex.org/W3022085849) **Citations:** 25 citations (openalex-snapshot · 2026-10-14) · 4 in our library, of the 140884 papers whose bibliographies we read ## Summary Arabic summarization research has mostly relied on sentence-scoring extractive methods, and there are few benchmark corpora. The authors adapt the BERTSUM framework of Liu & Lapata to Arabic. Because no Arabic-only BERT was available to them, they use multilingual BERT (M-BERT) as the encoder. They train the abstractive variant BERTSUMABS once with English BERT and once with M-BERT on the CNN data to check that the switch costs little. They then train both extractive and abstractive models on KALIMAT, an Arabic corpus with extractive summaries. They compare these against same-shaped Transformer models trained from scratch. On CNN, M-BERT nearly matches monolingual BERT. On KALIMAT, pretraining gives large gains in both tasks, and extractive models score far higher than abstractive ones against the extractive references. The authors present this as the first documented abstractive summarizer for Arabic. They also show that a pretrained multilingual encoder can make up for scarce in-language training data. ## Key points - On the CNN test set, BERTSUMABS gave ROUGE-1/2/L of 30.45/11.62/28.02 with BERT and 30.35/11.33/25.3 with M-BERT. Both runs were trained for 45,000 steps. M-BERT covers 104 languages. - Results on the KALIMAT test set (R1/R2/RL): - BERTSUMEXT: 42.02/24.59/41.99 - TRANSFORMEREXT: 28.75/14.80/28.72 - BERTSUMABS: 12.21/4.36/12.19 - TRANSFORMERABS: 6.93/1.78/6.88 - The architecture follows BERTSUM: - a [CLS] token is placed before each sentence, and interval segment embeddings separate sentences; - the extractive head is a sigmoid classifier on each [CLS]; - the abstractive decoder is a randomly initialized 6-layer Transformer, with separate optimizers for the encoder and decoder. - Arabic resources mentioned include EASC (153 articles, 765 human extractive summaries) and KALIMAT (20,291 articles, with categories such as culture, economy, religion and sport). For contrast, English has CNN/Daily-Mail (287K articles) and XSum (226,711 articles). - Preprocessing steps: - The KALIMAT files were stored as Latin-1, so they were converted to UTF-8. - The CoreNLP tokenizer was replaced with the stanford-arabic-corenlp-20181005 models. - Text was tokenized with the M-BERT vocabulary into 8 PyTorch files of 2,000 entries each. - Code: github.com/mukhtar-algezoli/Arabic PreSumm. ## Interesting ideas - Before moving a pipeline to a new language, first swap in the multilingual encoder on a familiar English benchmark, so the swap's cost is measured separately from the new data. — For Mova (learn): When gparse or the tagger changes backbone, rerun on a fixed English treebank first, so drops from the swap are not confused with drops from new silver data. - A from-scratch twin with the same architecture serves as a control that isolates exactly what pretraining contributes. — For Mova (learn): Every pretrained or LLM-taught component could ship with an untrained twin as a negative control, showing how much of the score really comes from the teacher signal. - Using different optimizers or learning-rate schedules for a pretrained encoder and a freshly initialized head keeps training stable when the two parts start at very different maturity. — For Mova (tree): gparse's randomly initialized arc and label scorers could get their own schedule, separate from the pretrained transformer layers, to avoid wrecking the encoder early in training. - Placing a marker token before each sentence gives one learned vector per sentence, which a simple per-unit classifier can then score for inclusion. — For Mova (answer): QA on fables could score each sentence's frames for relevance to the question, and cite the selected sentences as explainable evidence. - The extractive models' big lead on extractive references warns that the metric favors systems shaped like the reference. — For Mova (back): When evaluating the text → graph → text compressor, Mova should not score its regenerated text only against copy-style references, or ROUGE-like overlap will penalize faithful paraphrase. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{elmadani2020bert, title = {{BERT Fine-tuning For Arabic Text Summarization}}, author = {Elmadani, Khalid N. and Elgezouli, Mukhtar and Showk, Anas}, year = {2020}, doi = {10.48550/arxiv.2004.14135}, eprint = {2004.14135}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2004.14135} } ``` - [← previous Becoming Linguistically Mature: Modeling English and German Children's Writing Development Across School Grades](https://md.research.elamur.ai/free/en/papers/kerz-2020-becoming-linguistically-mature-modeling-english/) - [next → BERT-ATTACK: Adversarial Attack Against BERT Using BERT](https://md.research.elamur.ai/free/en/papers/li-2020-bert-attack-adversarial-attack-against/) - [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/) ## References 10 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Lamees Mahmoud Al Qassem, Di Wang, Zaid Al Mahmoud, Hassan Barada, Ahmad Al-Rubaie, and Nawaf I Almoosa. Automatic arabic summarization: A survey of methodologies and systems. Procedia Computer Science, 117:10–18, 2017. [find](https://scholar.google.com/scholar?q=Lamees+Mahmoud+Al+Qassem%2C+Di+Wang%2C+Zaid+Al+Mahmoud%2C+Hassan+Barada%2C+Ahmad+Al%2DRubaie%2C+and+Nawaf+I+Almoosa%2E+Automatic+arabic+summarization%3A+A+survey+of+methodologies+and+systems%2E+Procedia+Computer+Scienc) 2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. [↗](https://arxiv.org/abs/1810.04805) 3. Nizar Y Habash. Introduction to arabic natural language processing. Synthesis Lectures on Human Language Technologies, 3(1):1–187, 2010. [find](https://scholar.google.com/scholar?q=Nizar+Y+Habash%2E+Introduction+to+arabic+natural+language+processing%2E+Synthesis+Lectures+on+Human+Language+Technologies%2C+3%281%29%3A1%E2%80%93187%2C+2010%2E) 4. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Advances in neural information processing systems, pp. 1693–1701, 2015. [find](https://scholar.google.com/scholar?q=Karl+Moritz+Hermann%2C+Tomas+Kocisky%2C+Edward+Grefenstette%2C+Lasse+Espeholt%2C+Will+Kay%2C+Mustafa+Suleyman%2C+and+Phil+Blunsom%2E+Teaching+machines+to+read+and+comprehend%2E+In+Advances+in+neural+information+proce) [our card](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/) 5. Younes Jaafar and Karim Bouzoubaa. Towards a new hybrid approach for abstractive summarization. Procedia computer science, 142:286–293, 2018. [find](https://scholar.google.com/scholar?q=Younes+Jaafar+and+Karim+Bouzoubaa%2E+Towards+a+new+hybrid+approach+for+abstractive+summarization%2E+Procedia+computer+science%2C+142%3A286%E2%80%93293%2C+2018%2E) 6. Chin-Yew Lin and FJ Och. Looking for a few good metrics: Rouge and its evaluation. In Ntcir Workshop, 2004. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin+and+FJ+Och%2E+Looking+for+a+few+good+metrics%3A+Rouge+and+its+evaluation%2E+In+Ntcir+Workshop%2C+2004%2E) 7. Yang Liu and Mirella Lapata. arXiv:1908.08345, 2019. [↗](https://arxiv.org/abs/1908.08345) 8. Text summarization with pretrained encoders. Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018. [↗](https://arxiv.org/abs/1808.08745) 9. Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual bert? arXiv preprint arXiv:1906.01502, 2019. [↗](https://arxiv.org/abs/1906.01502) [our card](https://md.research.elamur.ai/free/en/papers/pires-2019-how-multilingual-is-multilingual-bert/) 10. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998–6008, 2017. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+Attention+is+all+you+need%2E+In+Advances+in+neural+information+processing+sys)