---
title: "Violet — paper by Mohamed et al. (2023) — summary and ideas"
description: "The authors build Violet, an encoder-decoder captioner. 43 references, cited by 9 papers here."
authors: ["Abdelrahman Mohamed", "Fakhraddin Alwajih", "El Moatez Billah Nagoudi", "Alcides Inciarte", "Muhammad Abdul-Mageed"]
year: 2023
venue: "Proceedings of ArabicNLP 2023"
source: "https://aclanthology.org/2023.arabicnlp-1.1/"
doi: "10.18653/v1/2023.arabicnlp-1.1"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mohamed-2023-violet-vision-language-model-arabic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mohamed-2023-violet-vision-language-model-arabic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/13/) / Violet: A Vision-Language Model for Arabic Image Captioning with Gemini Decoder

# Violet: A Vision-Language Model for Arabic Image Captioning with Gemini Decoder

**Authors:** [Abdelrahman Mohamed](https://md.research.elamur.ai/free/en/authors/abdelrahman-mohamed/), [Fakhraddin Alwajih](https://md.research.elamur.ai/free/en/authors/fakhraddin-alwajih/), [El Moatez Billah Nagoudi](https://md.research.elamur.ai/free/en/authors/el-moatez-billah-nagoudi/), [Alcides Inciarte](https://md.research.elamur.ai/free/en/authors/alcides-alcoba-inciarte/) and [Muhammad Abdul-Mageed](https://md.research.elamur.ai/free/en/authors/muhammad-abdul-mageed/) · **Year:** 2023 · **Venue:** [Proceedings of ArabicNLP 2023](https://md.research.elamur.ai/free/en/venues/arabicnlp/) ([ArabicNLP 2023](https://md.research.elamur.ai/free/en/venues/arabicnlp/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.arabicnlp-1.1/) · [PDF](https://aclanthology.org/2023.arabicnlp-1.1.pdf) · [DOI 10.18653/v1/2023.arabicnlp-1.1](https://doi.org/10.18653/v1/2023.arabicnlp-1.1) · [OpenAlex](https://openalex.org/W4389518385)

**Citations:** 9 citations (OpenAlex, 2026-10-14) · [9](https://md.research.elamur.ai/free/en/papers/mohamed-2023-violet-vision-language-model-arabic/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Arabic image captioning lags far behind English because native captioned images are scarce and strong Arabic generators have rarely been connected to vision models. The authors build Violet, an encoder-decoder captioner. Object features from a detector go through a small transformer encoder. Their output feeds a split ("Gemini") version of the Arabic generative model JASMINE. The lower half of JASMINE stays frozen and works as a plain language model. The upper half receives inserted cross-attention layers and acts as a vision-text fusion decoder.

For training data, they machine-translate English MSCOCO captions and filter out poor translations with a quality check. For a cleaner test, annotators hand-wrote Arabic captions for part of the MSCOCO test split, producing AraCOCO.

Violet beats a baseline that captions in English with VisualGPT and then translates into Arabic. It also outperforms earlier Arabic systems on Flickr8k by roughly 13 CIDEr points. The work shows that pairing a fluent pretrained native-language decoder with light fusion layers works better than translating English captions.

## Key points
- Architecture:
  - A FasterRCNN-style detector extracts K = 50 object features.
  - The features are projected down and passed to a three-layer transformer encoder.
  - JASMINE is the decoder. Cross-attention is added only in its second half.
  - Gates (π t, π m) balance the visual and linguistic signals.
- A meshed connection lets every encoder layer feed the cross-attention. Learned sigmoid weights (α) decide how much each layer contributes.
- Freezing the first six decoder layers scored better than leaving them trainable and also needed less compute. Adding cross-attention to every decoder layer gave clearly worse results.
- On AraCOCO, Violet versus translated VisualGPT output:

  | Model | CIDEr | BLEU-4 |
  |---|---|---|
  | Violet | 61.2 | 19.0 |
  | Translated VisualGPT | 58.5 | 17.6 |

- On the translated Flickr8k test set:
  - Violet reaches CIDEr 60.1, versus 46.9 for Elbedwehy and Medhat (2023) and 42.3 for Emami et al. (2022).
  - Violet's BLEU-1 is lower than Elbedwehy and Medhat's.
  - The authors argue CIDEr is the more relevant metric for captioning.
- Released resources: a quality-filtered Arabic translation of MSCOCO and the manual AraCOCO set. Stated limitations:
  - The system is not trained end to end, because it depends on an external detector.
  - Training covers only MSCOCO's 80 object classes.

## Interesting ideas
- Freeze the lower half of a pretrained decoder and add new cross-attention only in the upper half; this kept fluency, cut compute, and beat full fine-tuning, which seemed to damage the pretrained weights.
- A meshed connection lets every encoder layer contribute through learned gates, instead of relying only on the final layer, so the model picks the depth that matters.
- Training data comes from translating an existing English resource and then discarding low-quality outputs with an automatic check, rather than trusting every translation.
- The system is compared with a strong "just translate the English system's output" baseline, which shows whether a native model is really needed.
- Training uses automatically acquired data, but a small separate human-written test set (AraCOCO) gives cleaner evaluation, because metrics on translated test data may hide errors.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tree): gparse could keep a pretrained encoder's lower layers frozen and train only the upper layers plus the arc/label heads, which protects general knowledge while adapting to UD.
- **I2** (tree): gparse could mix all transformer layers with learned per-layer weights before arc scoring, and the learned weights would also show which layers carry syntax.
- **I3** (learn): silver data from LLM teachers or converted treebanks could pass the same kind of filter, keeping only items that pass agreement or quality checks before training.
- **I4** (learn): evaluations could include a cheap shortcut baseline as a negative control, for example a pipeline without the frames stage, to prove each stage adds value.
- **I5** (answer): keep a small hand-checked set of fable QA and word problems apart from silver-derived tests, so scores on the answer readers are not inflated by teacher artifacts.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of British Columbia](https://md.research.elamur.ai/free/en/universities/university-of-british-columbia/) — University, Vancouver, Canada · 104 works · 37,402 citations
- [Mohamed bin Zayed University of Artificial Intelligence](https://md.research.elamur.ai/free/en/universities/mohamed-bin-zayed-university-of-artificial-intelligence/) — University, Abu Dhabi, United Arab Emirates · 138 works · 6,590 citations

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works

## Cite

Cite as: Abdelrahman Mohamed, Fakhraddin Alwajih, El Moatez Billah Nagoudi, Alcides Inciarte and Muhammad Abdul-Mageed. 2023. Violet: A Vision-Language Model for Arabic Image Captioning with Gemini Decoder. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mohamed2023violet,
  title = {{Violet: A Vision-Language Model for Arabic Image Captioning with Gemini Decoder}},
  author = {Abdelrahman Mohamed, Fakhraddin Alwajih, El Moatez Billah Nagoudi, Alcides Inciarte and Muhammad Abdul-Mageed},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.1},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.1}
}
```

- [← previous Prompting and Fine-Tuning Open-Sourced Large Language Models for Stance Classification](https://md.research.elamur.ai/free/en/papers/cruickshank-2023-prompting-fine-tuning-open-sourced/)
- [next → 2*n is better than n²: Decomposing Event Coreference Resolution into Two Tractable Problems](https://md.research.elamur.ai/free/en/papers/ahmed-2023-2-n-is-better-than/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/13/)

## References

43 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Imad Afyouni, Imtinan Azhar, and Ashraf Elnagar. 2021. Aracap: A hybrid deep learning architecture for arabic image captioning. Procedia Computer Science, 189:382–389. [find](https://scholar.google.com/scholar?q=Imad+Afyouni%2C+Imtinan+Azhar%2C+and+Ashraf+Elnagar%2E+2021%2E+Aracap%3A+A+hybrid+deep+learning+architecture+for+arabic+image+captioning%2E+Procedia+Computer+Science%2C+189%3A382%E2%80%93389%2E)
2. Huda A Al-Muzaini, Tasniem N Al-Yahya, and Hafida Benhidour. 2018. Automatic arabic image captioning using rnn-lstm-based language model and cnn. International Journal of Advanced Computer Science and Applications, 9(6). [https://alliancecan.ca](https://alliancecan.ca) https://arc.ubc.ca/ubc-arc-sockeye [↗](https://alliancecan.ca)
3. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. [Flamingo: a visual language model for few-shot learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/). Advances in Neural Information Processing Systems, 35:23716–23736.
4. Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, Jianfeng Gao, et al. 2022. Vision-language pre-training: Basics, recent advances, and future trends. Foundations and Trends® in Computer Graphics and Vision, 14(3–4):163–352. [find](https://scholar.google.com/scholar?q=Zhe+Gan%2C+Linjie+Li%2C+Chunyuan+Li%2C+Lijuan+Wang%2C+Zicheng+Liu%2C+Jianfeng+Gao%2C+et+al%2E+2022%2E+Vision%2Dlanguage+pre%2Dtraining%3A+Basics%2C+recent+advances%2C+and+future+trends%2E+Foundations+and+Trends%C2%AE+in+Computer+Grap)
5. Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077–6086. Google translate api. Accessed: Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778.
6. Anfal Attai and Ashraf Elnagar. 2020. A survey on arabic image captioning systems using deep learning models. In 2020 14th International Conference on Innovations in Information Technology (IIT), pages 114–119. IEEE. [find](https://scholar.google.com/scholar?q=Anfal+Attai+and+Ashraf+Elnagar%2E+2020%2E+A+survey+on+arabic+image+captioning+systems+using+deep+learning+models%2E+In+2020+14th+International+Conference+on+Innovations+in+Information+Technology+%28IIT%29%2C+page)
7. Lun Huang, Wenmin Wang, Jie Chen, and Xiao-Yong Wei. 2019. Attention on attention for image captioning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4634–4643. [find](https://scholar.google.com/scholar?q=Lun+Huang%2C+Wenmin+Wang%2C+Jie+Chen%2C+and+Xiao%2DYong+Wei%2E+2019%2E+Attention+on+attention+for+image+captioning%2E+In+Proceedings+of+the+IEEE%2FCVF+international+conference+on+computer+vision%2C+pages+4634%E2%80%934643%2E)
8. Muhammad Awais, Muzammal Naseer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, and Fahad Shahbaz Khan. 2023. Foundational models defining a new era in vision: A survey and outlook. arXiv preprint arXiv:2307.13721. [↗](https://arxiv.org/abs/2307.13721)
9. Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. 2023. [Language is not all you need: Aligning perception with language models](https://md.research.elamur.ai/free/en/papers/huang-2023-language-all-need-aligning-perception/). arXiv preprint arXiv:2302.14045. [↗](https://arxiv.org/abs/2302.14045)
10. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large scale autoregressive language modeling with meshtensorflow. [find](https://scholar.google.com/scholar?q=Sid+Black%2C+Leo+Gao%2C+Phil+Wang%2C+Connor+Leahy%2C+and+Stella+Biderman%2E+2021%2E+GPT%2DNeo%3A+Large+scale+autoregressive+language+modeling+with+meshtensorflow%2E)
11. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. 2014. [Caffe: Convolutional architecture for fast feature embeding](https://md.research.elamur.ai/free/en/papers/jia-2014-caffe-convolutional-architecture-fast-feature/). arXiv preprint, arXiv:1408.5093. [↗](https://arxiv.org/abs/1408.5093)
12. Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny. 2022. Visualgpt: Data-efficient adaptation of pretrained language models for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18030–18040. [find](https://scholar.google.com/scholar?q=Jun+Chen%2C+Han+Guo%2C+Kai+Yi%2C+Boyang+Li%2C+and+Mohamed+Elhoseiny%2E+2022%2E+Visualgpt%3A+Data%2Defficient+adaptation+of+pretrained+language+models+for+image+captioning%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on)
13. Vasu Jindal. 2017. A deep learning approach for arabic caption generation using roots-words. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31. [find](https://scholar.google.com/scholar?q=Vasu+Jindal%2E+2017%2E+A+deep+learning+approach+for+arabic+caption+generation+using+roots%2Dwords%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volume+31%2E)
14. Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. 2020. Meshed-memory transformer for image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10578–10587. [find](https://scholar.google.com/scholar?q=Marcella+Cornia%2C+Matteo+Stefanini%2C+Lorenzo+Baraldi%2C+and+Rita+Cucchiara%2E+2020%2E+Meshed%2Dmemory+transformer+for+image+captioning%2E+In+Proceedings+of+the+IEEE%2FCVF+conference+on+computer+vision+and+pattern+r)
15. Andrej Karpathy, Armand Joulin, and Li F Fei-Fei. 2014. Deep fragment embeddings for bidirectional image sentence mapping. Advances in neural information processing systems, 27. [find](https://scholar.google.com/scholar?q=Andrej+Karpathy%2C+Armand+Joulin%2C+and+Li+F+Fei%2DFei%2E+2014%2E+Deep+fragment+embeddings+for+bidirectional+image+sentence+mapping%2E+Advances+in+neural+information+processing+systems%2C+27%2E)
16. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. [↗](https://arxiv.org/abs/2207.04672)
17. Moaz T Lasheen and Nahla H Barakat. 2022. Arabic image captioning: the effect of text pre-processing on the attention weights and the bleu-n scores. Int J Adv Comput Sci Appl, 13(7):11. [find](https://scholar.google.com/scholar?q=Moaz+T+Lasheen+and+Nahla+H+Barakat%2E+2022%2E+Arabic+image+captioning%3A+the+effect+of+text+pre%2Dprocessing+on+the+attention+weights+and+the+bleu%2Dn+scores%2E+Int+J+Adv+Comput+Sci+Appl%2C+13%287%29%3A11%2E)
18. Siming Li, Girish Kulkarni, Tamara Berg, Alexander Berg, and Yejin Choi. 2011. Composing simple image descriptions using web-scale n-grams. In Proceedings of the fifteenth conference on computational natural language learning, pages 220–228. [find](https://scholar.google.com/scholar?q=Siming+Li%2C+Girish+Kulkarni%2C+Tamara+Berg%2C+Alexander+Berg%2C+and+Yejin+Choi%2E+2011%2E+Composing+simple+image+descriptions+using+web%2Dscale+n%2Dgrams%2E+In+Proceedings+of+the+fifteenth+conference+on+computational+)
19. Samar Elbedwehy and T Medhat. 2023. Improved arabic image captioning model using feature concatenation with pre-trained word embedding. Neural Computing and Applications, pages 1–17. [find](https://scholar.google.com/scholar?q=Samar+Elbedwehy+and+T+Medhat%2E+2023%2E+Improved+arabic+image+captioning+model+using+feature+concatenation+with+pre%2Dtrained+word+embedding%2E+Neural+Computing+and+Applications%2C+pages+1%E2%80%9317%2E)
20. Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020. [Oscar: Object-semantics aligned pre-training for vision-language tasks](https://md.research.elamur.ai/free/en/papers/li-2020-oscar-object-semantics-aligned-pre/). In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX 16, pages 121–137. Springer.
21. Obeida ElJundi, Mohamad Dhaybi, Kotaiba Mokadam, Hazem M Hajj, and Daniel C Asmar. 2020. Resources and end-to-end neural network models for arabic image captioning. In VISIGRAPP (5: VIS-APP), pages 233–241. [find](https://scholar.google.com/scholar?q=Obeida+ElJundi%2C+Mohamad+Dhaybi%2C+Kotaiba+Mokadam%2C+Hazem+M+Hajj%2C+and+Daniel+C+Asmar%2E+2020%2E+Resources+and+end%2Dto%2Dend+neural+network+models+for+arabic+image+captioning%2E+In+VISIGRAPP+%285%3A+VIS%2DAPP%29%2C+pages+23)
22. Jonathan Emami, Pierre Nugues, Ashraf Elnagar, and Imad Afyouni. 2022. [Arabic image captioning using pre-training of deep bidirectional transformers](https://md.research.elamur.ai/free/en/papers/emami-2022-arabic-image-captioning-pre-training/). In Proceedings of the 15th International Conference on Natural Language Generation, pages 40–51.
23. Chin-Yew Lin. 2004. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out, pages 74–81.
24. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. [Microsoft coco: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In Computer Vision– ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer.
25. Sabri Monaf Sabri. 2021. Arabic image captioning using deep learning with attention. Ph.D. thesis, University of Georgia. [find](https://scholar.google.com/scholar?q=Sabri+Monaf+Sabri%2E+2021%2E+Arabic+image+captioning+using+deep+learning+with+attention%2E+Ph%2ED%2E+thesis%2C+University+of+Georgia%2E)
26. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+In+ICLR%2E)
27. Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Silvia Cascianelli, Giuseppe Fiameni, and Rita Cucchiara. 2022. From show to tell: A survey on deep learning-based image captioning. IEEE transactions on pattern analysis and machine intelligence, 45(1):539–559. [find](https://scholar.google.com/scholar?q=Matteo+Stefanini%2C+Marcella+Cornia%2C+Lorenzo+Baraldi%2C+Silvia+Cascianelli%2C+Giuseppe+Fiameni%2C+and+Rita+Cucchiara%2E+2022%2E+From+show+to+tell%3A+A+survey+on+deep+learning%2Dbased+image+captioning%2E+IEEE+transactio)
28. El Moatez Billah Nagoudi, Muhammad Abdul-Mageed, AbdelRahim Elmadany, Alcides Alcoba Inciarte, and Md Tawkat Islam Khondaker. 2022. [Jasmine: Arabic gpt models for few-shot learning](https://md.research.elamur.ai/free/en/papers/nagoudi-2023-jasmine-arabic-gpt-models-few/). arXiv preprint arXiv:2212.10755. [↗](https://arxiv.org/abs/2212.10755)
29. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. [Cider: Consensus-based image description evaluation](https://md.research.elamur.ai/free/en/papers/vedantam-2014-cider-consensus-based-image-description/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575.
30. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. [Show and tell: A neural image caption generator](https://md.research.elamur.ai/free/en/papers/vinyals-2014-show-tell-neural-image-caption/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156–3164.
31. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744. [↗](https://doi.org/10.52202/068431-2011)
32. Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. 2022. [Git: A generative image-to-text transformer for vision and language](https://md.research.elamur.ai/free/en/papers/wang-2022-git-generative-image-text-transformer/). arXiv preprint arXiv:2205.14100. [↗](https://arxiv.org/abs/2205.14100)
33. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318. [↗](https://doi.org/10.3115/1073083.1073135)
34. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. [Show, attend and tell: Neural image caption generation with visual attention](https://md.research.elamur.ai/free/en/papers/xu-2015-show-attend-tell-neural-image/). In International conference on machine learning, pages 2048–2057. PMLR.
35. Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641–2649. [find](https://scholar.google.com/scholar?q=Bryan+A+Plummer%2C+Liwei+Wang%2C+Chris+M+Cervantes%2C+Juan+C+Caicedo%2C+Julia+Hockenmaier%2C+and+Svetlana+Lazebnik%2E+2015%2E+Flickr30k+entities%3A+Collecting+region%2Dto%2Dphrase+correspondences+for+richer+image%2Dto%2Dsent)
36. Yezhou Yang, Ching Teo, Hal Daumé III, and Yiannis Aloimonos. 2011. Corpus-guided sentence generation of natural images. In Proceedings of the 2011 conference on empirical methods in natural language processing, pages 444–454. [find](https://scholar.google.com/scholar?q=Yezhou+Yang%2C+Ching+Teo%2C+Hal+Daum%C3%A9+III%2C+and+Yiannis+Aloimonos%2E+2011%2E+Corpus%2Dguided+sentence+generation+of+natural+images%2E+In+Proceedings+of+the+2011+conference+on+empirical+methods+in+natural+language+)
37. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In International conference on machine learning, pages 8748–8763. PMLR.
38. Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. 2016. Image captioning with semantic attention. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4651–4659. [find](https://scholar.google.com/scholar?q=Quanzeng+You%2C+Hailin+Jin%2C+Zhaowen+Wang%2C+Chen+Fang%2C+and+Jiebo+Luo%2E+2016%2E+Image+captioning+with+semantic+attention%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+page)
39. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
40. Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. 2022. [Coca: Contrastive captioners are image-text foundation models](https://md.research.elamur.ai/free/en/papers/yu-2022-coca-contrastive-captioners-image-text/). arXiv preprint arXiv:2205.01917. [↗](https://arxiv.org/abs/2205.01917)
41. Nils Reimers and Iryna Gurevych. 2019. [Sentence-bert: Sentence embeddings using siamese bert-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). arXiv preprint arXiv:1908.10084. [↗](https://arxiv.org/abs/1908.10084)
42. Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. [Faster r-cnn: Towards real-time object detection with region proposal networks](https://md.research.elamur.ai/free/en/papers/ren-2015-faster-r-cnn-towards-real/). Advances in neural information processing systems, 28.
43. Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. [Self-critical sequence training for image captioning](https://md.research.elamur.ai/free/en/papers/rennie-2016-self-critical-sequence-training-image/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7008–7024.

*Updated 2 November 2026*