---
title: "Multimodal Arabic Captioning with — Elchafei 2025"
description: "The paper addresses Arabic image captioning, a task where end-to-end vision–language systems often miss fine-grained and culturally specific content."
authors: ["Passant Elchafei", "Amany Fashwan"]
year: 2025
venue: "arXiv:2510.03295"
source: "https://arxiv.org/abs/2510.03295"
doi: "10.48550/arxiv.2510.03295"
arxiv: "2510.03295"
licence: "CC0 1.0 (arXiv)"
licence_url: "http://creativecommons.org/publicdomain/zero/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/elchafei-2025-multimodal-arabic-captioning-interpretable-visual/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/elchafei-2025-multimodal-arabic-captioning-interpretable-visual/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/) / Multimodal Arabic Captioning with Interpretable Visual Concept Integration

# Multimodal Arabic Captioning with Interpretable Visual Concept Integration

**Authors:** [Passant Elchafei](https://md.research.elamur.ai/free/en/authors/passant-elchafei/), Amany Fashwan · **Year:** 2025 · **Venue:** arXiv:2510.03295 · **Licence:** [CC0 1.0 (arXiv)](http://creativecommons.org/publicdomain/zero/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2510.03295) · [PDF](https://arxiv.org/pdf/2510.03295) · [DOI 10.48550/arxiv.2510.03295](https://doi.org/10.48550/arxiv.2510.03295) · [arXiv 2510.03295](https://arxiv.org/abs/2510.03295) · [OpenAlex](https://openalex.org/W4414968675)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 2 in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper addresses Arabic image captioning, a task where end-to-end vision–language systems often miss fine-grained and culturally specific content. For the ImageEval 2025 shared task (Subtask 2), the authors built VLCAP, a two-stage pipeline. The task data has 2,718 training images, 76 validation images and 753 test images. In the first stage, a multilingual CLIP-style encoder (mCLIP, AraCLIP or Jina V4) ranks Arabic labels from a fixed vocabulary by image–text cosine similarity. That vocabulary mixes frequent content words from the training captions with roughly 21K Visual Genome concepts translated into Arabic. In the second stage, the best-ranked labels go into an Arabic prompt, which is sent with the image to Qwen-VL or Gemini Pro Vision. This gives six encoder–generator combinations. Gemini + mCLIP produced the best cosine similarity, 60.01, which ranked first among the shared-task teams. VLCAP also placed second on the GPT-4o judge and led the manual ratings for completeness and accuracy. The work shows that an explicit, inspectable concept layer can steer a multimodal generator in a lower-resource language.

## Key points
- The label vocabulary combines two sources. The first is frequent training-caption words, filtered to remove stopwords, numbers and very short tokens. The second is about 21,000 Visual Genome objects, attributes and scenes, translated and adapted to Arabic.
- About 25–30 top-ranked labels per image are placed into a fixed Arabic prompt template. They are given to Qwen-VL or Gemini Pro Vision together with the image.
- Gemini + mCLIP scored best among the six configurations on two measures: BLEU-1 of 5.34 and cosine similarity of 60.01.
  - The top LLM-judge score of 36.33 came from an AraCLIP configuration. The table assigns it to Gemini, while the abstract assigns it to Qwen.
  - The organizers' base model kept a much higher BLEU-1 of 16.98.
- Official evaluation used three methods:
  - TF–IDF n-gram cosine similarity after Arabic normalization.
  - GPT-4o as a judge.
  - Human rating of 5% of the test set on cultural relevance, conciseness, completeness and accuracy.
- In the human rating, VLCAP ranked first on completeness (2.67) and accuracy (2.97). It ranked second on conciseness (3.17), and its cultural relevance score (2.57) was below that of several teams. The authors conclude that the choice of label extractor mattered more than the choice of generator.

## Interesting ideas
- The system makes "what is seen" an explicit, readable list of concepts before any text is generated, so every caption can be traced to its anchors. — For Mova (back): Generation from the meaning graph could likewise first emit an inspectable list of entities and attributes to verbalize, and errors could then be blamed on content selection or on surface realization.
- The candidate vocabulary is closed and hybrid: corpus-frequent in-domain words are merged with a large general-domain concept inventory, so retrieval covers both local jargon and broad world knowledge. — For Mova (world): Ontology labels drawn from WordNet/Wikidata could be merged with lemmas frequent in our target corpora (fables, school texts), giving a compact concept list that grounds entities in a text.
- Every retriever is crossed with every generator in a small grid, which shows which stage really drives quality; here the retriever outweighed the generator. — For Mova (learn): Witness parsers and LLM teacher tiers could be crossed with downstream readers in the same way, to find which upstream component limits answer accuracy before effort goes into the wrong part.
- Rankings flipped across metrics: the configuration with the best n-gram and cosine scores was not the one the LLM judge or humans preferred, and the strongest BLEU-1 system was weak elsewhere. — For Mova (learn): Evaluation should report several independent metrics side by side, plus a human spot-check of a small fixed sample, rather than trusting one overlap score.
- Retrieved concepts are handed to a strong generative model as hints inside the prompt, which constrains the model toward grounded content without any fine-tuning. — For Mova (learn): When querying an LLM teacher, we could include the frames and entities Mova has already extracted, so the teacher's silver labels stay anchored to Mova's own analysis and disagreements become easy to inspect.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · [Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{elchafei2025multimodal,
  title = {{Multimodal Arabic Captioning with Interpretable Visual Concept Integration}},
  author = {Elchafei, Passant and Fashwan, Amany},
  year = {2025},
  doi = {10.48550/arxiv.2510.03295},
  eprint = {2510.03295},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2510.03295}
}
```

- [← previous Multi-task Learning with Active Learning for Arabic Offensive Speech Detection](https://md.research.elamur.ai/free/en/papers/alansari-2025-multi-task-learning-active-learning/)
- [next → Negation in Universal Dependencies](https://md.research.elamur.ai/free/en/papers/findlay-2025-negation-universal-dependencies/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/)

## References

20 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Muhammad Al-Barham, Imad Afyouni, Khalid Almubarak, Ashraf Elnagar, Ayad Turky, and Ibrahim Hashem. 2024. Araclip: Cross-lingual learning for effective arabic image retrieval. In Proceedings of The Second Arabic Natural Language Processing Conference, pages 102–110. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Muhammad+Al%2DBarham%2C+Imad+Afyouni%2C+Khalid+Almubarak%2C+Ashraf+Elnagar%2C+Ayad+Turky%2C+and+Ibrahim+Hashem%2E+2024%2E+Araclip%3A+Cross%2Dlingual+learning+for+effective+arabic+image+retrieval%2E+In+Proceedings+of+The+Se)
2. Rasha Saleh Al-Malki and Arwa Yousuf Al-Aama. 2023. Arabic captioning for images of clothing using deep learning. Sensors, 23(8). [find](https://scholar.google.com/scholar?q=Rasha+Saleh+Al%2DMalki+and+Arwa+Yousuf+Al%2DAama%2E+2023%2E+Arabic+captioning+for+images+of+clothing+using+deep+learning%2E+Sensors%2C+23%288%29%2E)
3. Huda Al-muzaini, Tasniem N., and Hafida Benhidour. 2018. Automatic arabic image captioning using rnn-lstm-based language model and cnn. International Journal of Advanced Computer Science and Applications, 9. [find](https://scholar.google.com/scholar?q=Huda+Al%2Dmuzaini%2C+Tasniem+N%2E%2C+and+Hafida+Benhidour%2E+2018%2E+Automatic+arabic+image+captioning+using+rnn%2Dlstm%2Dbased+language+model+and+cnn%2E+International+Journal+of+Advanced+Computer+Science+and+Applicati)
4. Rabee Al-Qasem and Mohannad Hendi. 2025. Impactai at imageeval 2025 shared task: Region-aware transformers for arabic image captioning: A case study on the palestinian narrative. In Proceedings of the Third Arabic NLP Conference (ArabicNLP 2025). Co-located with EMNLP 2025, Nov 5–9. [find](https://scholar.google.com/scholar?q=Rabee+Al%2DQasem+and+Mohannad+Hendi%2E+2025%2E+Impactai+at+imageeval+2025+shared+task%3A+Region%2Daware+transformers+for+arabic+image+captioning%3A+A+case+study+on+the+palestinian+narrative%2E+In+Proceedings+of+the)
5. Ashwaq Alsayed, Thamir M. Qadah, and Muhammad Arif. 2023. A performance analysis of transformer-based deep learning models for arabic image captioning. J. King Saud Univ. Comput. Inf. Sci., 35(9). [find](https://scholar.google.com/scholar?q=Ashwaq+Alsayed%2C+Thamir+M%2E+Qadah%2C+and+Muhammad+Arif%2E+2023%2E+A+performance+analysis+of+transformer%2Dbased+deep+learning+models+for+arabic+image+captioning%2E+J%2E+King+Saud+Univ%2E+Comput%2E+Inf%2E+Sci%2E%2C+35%289%29%2E)
6. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. Ahlam Bashiti, Alaa Aljabari, Hadi Hamoud, Md. [find](https://scholar.google.com/scholar?q=Jinze+Bai%2C+Shuai+Bai%2C+Shusheng+Yang%2C+Shijie+Wang%2C+Sinan+Tan%2C+Peng+Wang%2C+Junyang+Lin%2C+Chang+Zhou%2C+and+Jingren+Zhou%2E+2023%2E+Qwen%2Dvl%3A+A+versatile+vision%2Dlanguage+model+for+understanding%2C+localization%2C+tex) [our card](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/)
7. Rafiul Biswas, Bilal Shalash, Mustafa Jarrar, Fadi Zaraket, George Mikros, Ehsaneddin Asgari, and Wajdi Zaghouani. 2025. ImageEval 2025: The First Arabic Image Captioning Shared Task. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025). Co-located with EMNLP 2025, November 5–9. [find](https://scholar.google.com/scholar?q=Rafiul+Biswas%2C+Bilal+Shalash%2C+Mustafa+Jarrar%2C+Fadi+Zaraket%2C+George+Mikros%2C+Ehsaneddin+Asgari%2C+and+Wajdi+Zaghouani%2E+2025%2E+ImageEval+2025%3A+The+First+Arabic+Image+Captioning+Shared+Task%2E+In+Proceedings+o)
8. Abdulkadir Shehu Bichi, Ismail Dauda Abubakar, Fatima Muhammad Adam, Aminu Musa, Auwal Umar Ahmed, Abubakar Ibrahim, Khadija Salihu Aua, Aisha Mustapha Ahmed, and Mahmud Said Ahmed. 2025. Codezone research group at imageeval 2025 shared task: Arabic image captioning using blip and m2m100: A two-stage translation approach for imageeval 2025. In Proceedings of the Third Arabic NLP Conference (ArabicNLP 2025). Co-located with EMNLP 2025, Nov 5–9. [find](https://scholar.google.com/scholar?q=Abdulkadir+Shehu+Bichi%2C+Ismail+Dauda+Abubakar%2C+Fatima+Muhammad+Adam%2C+Aminu+Musa%2C+Auwal+Umar+Ahmed%2C+Abubakar+Ibrahim%2C+Khadija+Salihu+Aua%2C+Aisha+Mustapha+Ahmed%2C+and+Mahmud+Said+Ahmed%2E+2025%2E+Codezone+res)
9. Guanhua Chen, Lu Hou, Yun Chen, Wenliang Dai, Lifeng Shang, Xin Jiang, Qun Liu, Jia Pan, and Wenping Wang. 2023. mCLIP: Multilingual CLIP via cross-lingual transfer. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13028– 13043, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Guanhua+Chen%2C+Lu+Hou%2C+Yun+Chen%2C+Wenliang+Dai%2C+Lifeng+Shang%2C+Xin+Jiang%2C+Qun+Liu%2C+Jia+Pan%2C+and+Wenping+Wang%2E+2023%2E+mCLIP%3A+Multilingual+CLIP+via+cross%2Dlingual+transfer%2E+In+Proceedings+of+the+61st+Annual+)
10. Samar Elbedwehy and T. Medhat. 2023. Improved arabic image captioning model using feature concatenation with pre-trained word embedding. Neural Comput. Appl., 35(26):19051–19067. [find](https://scholar.google.com/scholar?q=Samar+Elbedwehy+and+T%2E+Medhat%2E+2023%2E+Improved+arabic+image+captioning+model+using+feature+concatenation+with+pre%2Dtrained+word+embedding%2E+Neural+Comput%2E+Appl%2E%2C+35%2826%29%3A19051%E2%80%9319067%2E)
11. Obeida Eljundi, Mohamad Dhaybi, Kotaiba Mokadam, Hazem Hajj, and Daniel Asmar. 2020. Resources and end-to-end neural network models for arabic image captioning. pages 233–241. [find](https://scholar.google.com/scholar?q=Obeida+Eljundi%2C+Mohamad+Dhaybi%2C+Kotaiba+Mokadam%2C+Hazem+Hajj%2C+and+Daniel+Asmar%2E+2020%2E+Resources+and+end%2Dto%2Dend+neural+network+models+for+arabic+image+captioning%2E+pages+233%E2%80%93241%2E)
12. Jonathan Emami, Pierre Nugues, Ashraf Elnagar, and Imad Afyouni. 2022. Arabic image captioning using pre-training of deep bidirectional transformers. In Proceedings of the 15th International Conference on Natural Language Generation, pages 40–51, Waterville, Maine, USA and virtual meeting. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jonathan+Emami%2C+Pierre+Nugues%2C+Ashraf+Elnagar%2C+and+Imad+Afyouni%2E+2022%2E+Arabic+image+captioning+using+pre%2Dtraining+of+deep+bidirectional+transformers%2E+In+Proceedings+of+the+15th+International+Conferenc)
13. Wen Gendy and Dularia Patel. 2024. Advancements in computer vision: A comprehensive survey of image processing and interdisciplinary applications. Academic Journal of Science and Technology, 13(2):28–34. [find](https://scholar.google.com/scholar?q=Wen+Gendy+and+Dularia+Patel%2E+2024%2E+Advancements+in+computer+vision%3A+A+comprehensive+survey+of+image+processing+and+interdisciplinary+applications%2E+Academic+Journal+of+Science+and+Technology%2C+13%282%29%3A28%E2%80%93)
14. Michael Günther, Saba Sturua, Mohammad Akram, Isabelle Mohr, Andrei Ungureanu, Sedigheh Eslami, Scott Martens, Bo Wang, Nan Wang, and Han Xiao. 2025. jina-embeddings-v4: Universal embeddings for multimodal multilingual retrieval. Muhammad Abu Horaira, Farhan Amin, Sakibul Hasan, Md. [find](https://scholar.google.com/scholar?q=Michael+G%C3%BCnther%2C+Saba+Sturua%2C+Mohammad+Akram%2C+Isabelle+Mohr%2C+Andrei+Ungureanu%2C+Sedigheh+Eslami%2C+Scott+Martens%2C+Bo+Wang%2C+Nan+Wang%2C+and+Han+Xiao%2E+2025%2E+jina%2Dembeddings%2Dv4%3A+Universal+embeddings+for+multi)
15. Tanvir Ahammed Shawon, and Muhammad Ibrahim Khan. 2025. Phantomtroupe at imageeval shared task: Multimodal arabic image captioning through translation-based fine-tuning of llm models. In Proceedings of the Third Arabic NLP Conference (ArabicNLP 2025). Co-located with EMNLP 2025, Nov 5–9. [find](https://scholar.google.com/scholar?q=Tanvir+Ahammed+Shawon%2C+and+Muhammad+Ibrahim+Khan%2E+2025%2E+Phantomtroupe+at+imageeval+shared+task%3A+Multimodal+arabic+image+captioning+through+translation%2Dbased+fine%2Dtuning+of+llm+models%2E+In+Proceedings+o)
16. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, Michael Bernstein, and Fei-Fei Li. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123. [find](https://scholar.google.com/scholar?q=Ranjay+Krishna%2C+Yuke+Zhu%2C+Oliver+Groth%2C+Justin+Johnson%2C+Kenji+Hata%2C+Joshua+Kravitz%2C+Stephanie+Chen%2C+Yannis+Kalantidis%2C+Li%2DJia+Li%2C+David+Shamma%2C+Michael+Bernstein%2C+and+Fei%2DFei+Li%2E+2017%2E+Visual+genome%3A+) [our card](https://md.research.elamur.ai/free/en/papers/krishna-2016-visual-genome-connecting-language-vision/)
17. Abdelrahman Mohamed, Fakhraddin Alwajih, El Moatez Billah Nagoudi, Alcides Alcoba, and Muhammad Abdul-Mageed. 2023. Violet: A vision-language model for arabic image captioning with gemini decoder. pages 1–11. [find](https://scholar.google.com/scholar?q=Abdelrahman+Mohamed%2C+Fakhraddin+Alwajih%2C+El+Moatez+Billah+Nagoudi%2C+Alcides+Alcoba%2C+and+Muhammad+Abdul%2DMageed%2E+2023%2E+Violet%3A+A+vision%2Dlanguage+model+for+arabic+image+captioning+with+gemini+decoder%2E+pag)
18. Mariam Saeed, Sarah Elshabrawy, Abdelrahman Ha-grass, Mazen Yasser, and Ayman Khalafallah. 2025. Averroes at imageeval 2025 shared task: Advancing arabic image captioning with augmentation and two-stage generation. In Proceedings of the Third Arabic NLP Conference (ArabicNLP 2025). Co-located with EMNLP 2025, Nov 5–9. [find](https://scholar.google.com/scholar?q=Mariam+Saeed%2C+Sarah+Elshabrawy%2C+Abdelrahman+Ha%2Dgrass%2C+Mazen+Yasser%2C+and+Ayman+Khalafallah%2E+2025%2E+Averroes+at+imageeval+2025+shared+task%3A+Advancing+arabic+image+captioning+with+augmentation+and+two%2Dsta)
19. Muhy Eddin Za’ter and Bashar Talafha. 2022. Benchmarking and improving arabic automatic image captioning through the use of multi-task learning paradigm. [find](https://scholar.google.com/scholar?q=Muhy+Eddin+Za%E2%80%99ter+and+Bashar+Talafha%2E+2022%2E+Benchmarking+and+improving+arabic+automatic+image+captioning+through+the+use+of+multi%2Dtask+learning+paradigm%2E)
20. Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. 2024. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8):5625–5644. [find](https://scholar.google.com/scholar?q=Jingyi+Zhang%2C+Jiaxing+Huang%2C+Sheng+Jin%2C+and+Shijian+Lu%2E+2024%2E+Vision%2Dlanguage+models+for+vision+tasks%3A+A+survey%2E+IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intelligence%2C+46%288%29%3A5625%E2%80%935644%2E)