---
title: "Why do LLaVA Vision-Language Models Reply to Images in"
description: "LLaVA-style vision-language models often ignore the language of a question and answer in English… 49 references, cited by 7 papers here."
authors: ["Musashi Hinck", "Carolin Holtermann", "Matthew Lyle Olson", "Florian Schneider", "Sungduk Yu", "Anahita Bhiwandiwalla", "Anne Lauscher", "Shao-Yen Tseng", "Vasudev Lal"]
year: 2024
venue: "Findings of the Association for Computational Linguistics: EMNLP 2024"
source: "https://aclanthology.org/2024.findings-emnlp.783/"
doi: "10.18653/v1/2024.findings-emnlp.783"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hinck-2024-why-do-llava-vision-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hinck-2024-why-do-llava-vision-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/) / Why do LLaVA Vision-Language Models Reply to Images in English?

# Why do LLaVA Vision-Language Models Reply to Images in English?

**Authors:** [Musashi Hinck](https://md.research.elamur.ai/free/en/authors/musashi-hinck/), [Carolin Holtermann](https://md.research.elamur.ai/free/en/authors/carolin-holtermann/), [Matthew Lyle Olson](https://md.research.elamur.ai/free/en/authors/matthew-lyle-olson/), [Florian Schneider](https://md.research.elamur.ai/free/en/authors/florian-schneider/), Sungduk Yu, Anahita Bhiwandiwalla, [Anne Lauscher](https://md.research.elamur.ai/free/en/authors/anne-lauscher/), Shao-Yen Tseng and Vasudev Lal · **Year:** 2024 · **Venue:** [Findings of the Association for Computational Linguistics: EMNLP 2024](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2024](https://md.research.elamur.ai/free/en/venues/findings/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.findings-emnlp.783/) · [PDF](https://aclanthology.org/2024.findings-emnlp.783.pdf) · [DOI 10.18653/v1/2024.findings-emnlp.783](https://doi.org/10.18653/v1/2024.findings-emnlp.783) · [OpenAlex](https://openalex.org/W4404780795)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/hinck-2024-why-do-llava-vision-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLaVA-style vision-language models often ignore the language of a question and answer in English when an image is attached. The authors call this Image-induced Fidelity Loss (IFL). They measure it by running each item twice: once with the real image and once with a textual description of that image in its place. Because the two inputs carry the same meaning, any drop in "replies in the query's language" can be attributed to the image itself.

Using three multilingual VQA benchmarks covering fourteen languages, they show that adding the picture lowers the chance of a correct-language reply by roughly 6 to 53 points. To find the cause, they train many LLaVA variants that differ in one design choice at a time and estimate each choice's effect statistically. They also inspect and edit the models' hidden representations. Both lines of evidence place the problem in the language backbone, not the vision side. Swapping in a bilingual LLM helps most, and steering intermediate attention layers also reduces IFL. The work argues that language fidelity should be evaluated as a goal in its own right.

## Key points
- The evaluation set has 7740 tasks drawn from MaXM, PALO-LLaVAW and ViSIT across 14 languages. Each model produces 15,480 responses: every task is run with the image and again with its description.
- Four existing models were tested: LLaVA-v1.5-7b, LLaVA-v1.5-13b, BakLLaVA and LLaVA-Gemma-2B.
  - The worst case was LLaVA-Gemma-2B on ViSIT, with a 52.9 percentage point drop.
  - Almost every model–benchmark effect was significant; BakLLaVA on ViSIT was the exception.
- Output language was identified with GlotLID. Because GlotLID makes systematic errors, especially on short texts, the estimates were corrected with design-based supervised learning (DSL). The correction used a stratified sample of 1000 manually labelled outputs.
- In the ablation study, replacing the English-centric LLM backbone with a bilingual one had the largest effect against IFL.
- UMAP views show that visual tokens do not cluster with any particular language. Intervening on intermediate attention layers lowered IFL, which suggests the issue sits inside the LLM.

## Interesting ideas
- Swap one input component for a meaning-equivalent stand-in (image versus written description) and attribute the change in output to the form of the input, not its content. This is a clean counterfactual control.
- Treat an automatic labeller as a biased proxy, then correct the downstream statistics with a small stratified, hand-checked sample (DSL), so the results read as if everything had been expert-annotated.
- Pair a large statistical ablation, varying one design choice per trained model and estimating each choice's effect, with a look inside the model. Each kind of evidence checks the other.
- Score whether the output keeps a property of the input (here, its language) as a separate metric from answer accuracy. Correct-but-mismatched answers then do not hide inside accuracy figures.
- Cheap, targeted edits to intermediate attention states at inference time can fix a specific bias without retraining.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Build negative-control pairs where a sentence is replaced by a paraphrase or an explicit-pronoun version with identical meaning. Any change in the resulting frames would then expose form-driven errors.
- **I2** (learn): Silver trees agreed by witness parsers could be debiased the same way. A few hundred gold-checked sentences would correct estimates of attachment or LAS-like error rates without annotating the whole corpus.
- **I3** (tree): For gparse, train variants that each change one choice (embedding, treebank mix, dialect conversion) and regress error rates on those choices, then confirm the leading suspects by inspecting intermediate states.
- **I4** (back): Generation and the text → graph → text compressor could be checked for preserved surface properties such as tense, person and register, alongside meaning fidelity.
- **I5** (tree): Known systematic gparse errors could be probed and patched with small layer-level interventions, logged so each correction stays explainable.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Universität Hamburg](https://md.research.elamur.ai/free/en/universities/universitat-hamburg/) — University, Hamburg, Germany · 79 works · 5,422 citations
- [Intel](https://md.research.elamur.ai/free/en/companies/intel/) — Company, Santa Clara, United States · 15 works · 41,257 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 657 works

## Cite

Cite as: Musashi Hinck, Carolin Holtermann, Matthew Lyle Olson, Florian Schneider, Sungduk Yu, Anahita Bhiwandiwalla, Anne Lauscher, Shao-Yen Tseng and Vasudev Lal. 2024. Why do LLaVA Vision-Language Models Reply to Images in English?. Findings of the Association for Computational Linguistics: EMNLP 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{hinck2024why,
  title = {{Why do LLaVA Vision-Language Models Reply to Images in English?}},
  author = {Musashi Hinck, Carolin Holtermann, Matthew Lyle Olson, Florian Schneider, Sungduk Yu, Anahita Bhiwandiwalla, Anne Lauscher, Shao-Yen Tseng and Vasudev Lal},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.783},
  url = {https://doi.org/10.18653/v1/2024.findings-emnlp.783}
}
```

- [← previous What Causes the Failure of Explicit to Implicit Discourse Relation Recognition?](https://md.research.elamur.ai/free/en/papers/liu-2024-what-causes-failure-explicit-implicit/)
- [next → WorryWords: Norms of Anxiety Association for over 44k English Words](https://md.research.elamur.ai/free/en/papers/mohammad-2024-worrywords-norms-anxiety-association-over/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/9/)

## References

49 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. 2024. [Yi: Open foundation models by 01.ai](https://md.research.elamur.ai/free/en/papers/ai-2024-yi-open-foundation-models-01/). Preprint, arXiv:2403.04652. [↗](https://arxiv.org/abs/2403.04652)
2. Michael Andersland. 2024. [Amharic llama and llava: Multimodal llms for low resource languages](https://md.research.elamur.ai/free/en/papers/andersland-2024-amharic-llama-llava-multimodal-llms/). arXiv preprint arXiv:2403.06354. [↗](https://arxiv.org/abs/2403.06354)
3. Andy Arditi, Oscar Balcells Obeso, Aaquib111, Wes Gurnee, and Neel Nanda. 2024. Refusal in llms is mediated by a single direction. AI Alignment Forum. [find](https://scholar.google.com/scholar?q=Andy+Arditi%2C+Oscar+Balcells+Obeso%2C+Aaquib111%2C+Wes+Gurnee%2C+and+Neel+Nanda%2E+2024%2E+Refusal+in+llms+is+mediated+by+a+single+direction%2E+AI+Alignment+Forum%2E)
4. Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, and Ludwig Schimdt. 2023. VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use. arXiv preprint arXiv:2308.06595. [↗](https://arxiv.org/abs/2308.06595)
5. Terra Blevins and Luke Zettlemoyer. 2022. Language contamination helps explains the cross-lingual capabilities of English pretrained models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3563–3574, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [↗](https://doi.org/10.48448/668z-qr43)
6. Tim Brooks, Aleksander Holynski, and Alexei A. Efros. 2023. InstructPix2Pix: Learning to Follow Image Editing Instructions. arXiv preprint. ArXiv:2211.09800 [cs]. [↗](https://arxiv.org/abs/2211.09800)
7. Soravit Changpinyo, Linting Xue, Michal Yarom, Ashish Thapliyal, Idan Szpektor, Julien Amelot, Xi Chen, and Radu Soricut. 2023. MaXM: Towards Multilingual Visual Question Answering. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2667–2682, Singapore. [find](https://scholar.google.com/scholar?q=Soravit+Changpinyo%2C+Linting+Xue%2C+Michal+Yarom%2C+Ashish+Thapliyal%2C+Idan+Szpektor%2C+Julien+Amelot%2C+Xi+Chen%2C+and+Radu+Soricut%2E+2023%2E+MaXM%3A+Towards+Multilingual+Visual+Question+Answering%2E+In+Findings+of+the)
8. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.acl-main.747)
9. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d18-1269)
10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423)
11. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). In International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.2010.11929)
12. Naoki Egami, Musashi Hinck, Brandon Stewart, and Hanying Wei. 2023. Using imperfect surrogates for downstream inference: Design-based supervised learning for social science applications of large language models. In Advances in Neural Information Processing Systems, volume 36, pages 68589–68601. Curran Associates, Inc. [↗](https://doi.org/10.17605/osf.io/gjt87)
13. Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank. 2022. MAGMA – multimodal augmentation of generative models through adapter-based finetuning. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2416–2428, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Constantin+Eichenberg%2C+Sidney+Black%2C+Samuel+Weinbach%2C+Letitia+Parcalabescu%2C+and+Anette+Frank%2E+2022%2E+MAGMA+%E2%80%93+multimodal+augmentation+of+generative+models+through+adapter%2Dbased+finetuning%2E+In+Findings+o)
14. Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. 2016. [Multi30K: Multilingual English-German image descriptions](https://md.research.elamur.ai/free/en/papers/elliott-2016-multi30k-multilingual-english-german-image/). In Proceedings of the 5th Workshop on Vision and Language, pages 70– 74, Berlin, Germany. Association for Computational Linguistics.
15. Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2024. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093. [↗](https://arxiv.org/abs/2406.04093)
16. Musashi Hinck, Matthew L. Olson, David Cobbley, Shao-Yen Tseng, and Vasudev Lal. 2024. Llava-gemma: Accelerating multimodal foundation models with a compact language model. Preprint, arXiv:2404.01331. [↗](https://arxiv.org/abs/2404.01331)
17. Carolin Holtermann, Paul Röttger, Timm Dill, and Anne Lauscher. 2024. Evaluating the elementary multilingual capabilities of large language models with multiq. Preprint, arXiv:2403.03814. [↗](https://arxiv.org/abs/2403.03814)
18. Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. 2024. Prismatic vlms: Investigating the design space of visually-conditioned language models. In International Conference on Machine Learning (ICML). [↗](https://doi.org/10.48550/arxiv.2402.07865)
19. Amir Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schuetze. 2023. [GlotLID: Language identification for low-resource languages](https://md.research.elamur.ai/free/en/papers/kargaran-2023-glotlid-language-identification-low-resource/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6155–6218, Singapore. Association for Computational Linguistics.
20. Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://md.research.elamur.ai/free/en/papers/kim-2021-vilt-vision-language-transformer-convolution/). arXiv preprint. ArXiv:2102.03334 [cs, stat]. [↗](https://arxiv.org/abs/2102.03334)
21. Leland McInnes, John Healy, and James Melville. 2018. [Umap: Uniform manifold approximation and projection for dimension reduction](https://md.research.elamur.ai/free/en/papers/mcinnes-2018-umap-uniform-manifold-approximation-projection/). arXiv preprint arXiv:1802.03426. [↗](https://arxiv.org/abs/1802.03426)
22. Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. [Similarity of neural network representations revisited](https://md.research.elamur.ai/free/en/papers/kornblith-2019-similarity-neural-network-representations-revisited/). In International conference on machine learning, pages 3519–3529. PMLR. [↗](https://doi.org/10.48550/arxiv.1905.00414)
23. Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. 2023. Linearly Mapping from Image to Text Space. arXiv preprint. ArXiv:2209.15162 [cs]. [↗](https://arxiv.org/abs/2209.15162)
24. Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. 2020. The Open Images Dataset v4: Unified Image Classification, Object Detection, and Visual Relationship Detection at Scale. International journal of computer vision, 128(7):1956–1981. [find](https://scholar.google.com/scholar?q=Alina+Kuznetsova%2C+Hassan+Rom%2C+Neil+Alldrin%2C+Jasper+Uijlings%2C+Ivan+Krasin%2C+Jordi+Pont%2DTuset%2C+Shahab+Kamali%2C+Stefan+Popov%2C+Matteo+Malloci%2C+Alexander+Kolesnikov%2C+et+al%2E+2020%2E+The+Open+Images+Dataset+v4%3A+)
25. Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen. 2023. [Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback](https://md.research.elamur.ai/free/en/papers/lai-2023-okapi-instruction-tuned-large-language/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 318–327, Singapore. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2307.16039)
26. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). arXiv preprint. ArXiv:2301.12597 [cs]. [↗](https://arxiv.org/abs/2301.12597)
27. Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021. Visually grounded reasoning across languages and cultures. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10467–10485, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2109.13238)
28. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. [Visual Instruction Tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). In Advances in Neural Information Processing Systems, volume 36, pages 34892–34916, New Orleans, LT, USA. Curran Associates, Inc. [↗](https://doi.org/10.48550/arxiv.2304.08485)
29. Muhammad Maaz, Hanoona Rasheed, Abdelrahman Shaker, Salman Khan, Hisham Cholakal, Rao M Anwer, Tim Baldwin, Michael Felsberg, and Fahad S Khan. 2024. PALO: A Polyglot Large Multimodal Model for 5B People. arXiv preprint arXiv:2402.14818. [↗](https://arxiv.org/abs/2402.14818)
30. Oscar Mañas, Pau Rodriguez Lopez, Saba Ahmadi, Aida Nematzadeh, Yash Goyal, and Aishwarya Agrawal. 2023. MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2523–2548, Dubrovnik, Croatia. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2210.07179)
31. OpenAI. 2024. Gpt-4o: Documentation. Accessed: 2024-06-15. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2024%2E+Gpt%2D4o%3A+Documentation%2E+Accessed%3A+2024%2D06%2D15%2E)
32. Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. 2024. [Dinov2: Learning robust visual features without supervision](https://md.research.elamur.ai/free/en/papers/oquab-2023-dinov2-learning-robust-visual-features/). Preprint, arXiv:2304.07193. [↗](https://arxiv.org/abs/2304.07193)
33. Björn Plüster. 2023. LeoLM: Igniting German-Language LLM Research. Accessed: 2024-06-15. [find](https://scholar.google.com/scholar?q=Bj%C3%B6rn+Pl%C3%BCster%2E+2023%2E+LeoLM%3A+Igniting+German%2DLanguage+LLM+Research%2E+Accessed%3A+2024%2D06%2D15%2E)
34. Chen Qiu, Dan Oneat , ă, Emanuele Bugliarello, Stella Frank, and Desmond Elliott. 2022. Multilingual multimodal learning with machine translated text. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4178–4193, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chen+Qiu%2C+Dan+Oneat+%2C+a%CC%86%2C+Emanuele+Bugliarello%2C+Stella+Frank%2C+and+Desmond+Elliott%2E+2022%2E+Multilingual+multimodal+learning+with+machine+translated+text%2E+In+Findings+of+the+Association+for+Computational)
35. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. [Learning Transferable Visual Models From Natural Language Supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In Proceedings of the 38th International Conference on Machine Learning, pages 8748–8763. PMLR. ISSN: 2640-3498. [↗](https://doi.org/10.48550/arxiv.2103.00020)
36. Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2019. A Survey of Cross-lingual Word Embedding Models. Journal of Artificial Intelligence Research, 65:569– 631. [↗](https://doi.org/10.1613/jair.1.11640)
37. Florian Schneider and Sunayana Sitaram. 2024. M5 – a diverse benchmark to assess the performance of large multimodal models across multilingual and multicultural vision-language tasks. Preprint, arXiv:2407.03791. [↗](https://arxiv.org/abs/2407.03791)
38. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. [Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, Melbourne, Australia. Association for Computational Linguistics.
39. Dongjae Shin, Hyunseok Lim, Inho Won, Changsu Choi, Minjun Kim, Seungwoo Song, Hangyeol Yoo, Sangmin Kim, and Kyungtae Lim. 2024. X-llava: Optimizing bilingual large vision-language alignment. arXiv preprint arXiv:2403.11399. [↗](https://arxiv.org/abs/2403.11399)
40. SkunkworksAI. 2023. Bakllava-1. Hugging Face model repository. [find](https://scholar.google.com/scholar?q=SkunkworksAI%2E+2023%2E+Bakllava%2D1%2E+Hugging+Face+model+repository%2E)
41. Yueqi Song, Simran Khanuja, and Graham Neubig. 2024. What is missing in multilingual visual reasoning and how to fix it. arXiv preprint arXiv:2403.01404. [↗](https://arxiv.org/abs/2403.01404)
42. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. Preprint, arXiv:2207.04672. [↗](https://arxiv.org/abs/2207.04672)
43. Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). Preprint, arXiv:2307.09288. [↗](https://arxiv.org/abs/2307.09288)
44. Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. 2022. [SimVLM: Simple Visual Language Model Pretraining with Weak Supervision](https://md.research.elamur.ai/free/en/papers/wang-2021-simvlm-simple-visual-language-model/). arXiv preprint. ArXiv:2108.10904 [cs]. [↗](https://arxiv.org/abs/2108.10904)
45. Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. [Do llamas work in english? on the latent language of multilingual transformers](https://md.research.elamur.ai/free/en/papers/wendler-2024-do-llamas-work-english-latent/). Preprint, arXiv:2402.10588. [↗](https://arxiv.org/abs/2402.10588)
46. Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. [Wildchat: 1m chatgpt interaction logs in the wild](https://md.research.elamur.ai/free/en/papers/zhao-2024-wildchat-1m-chatgpt-interaction-logs/). arXiv preprint arXiv:2405.01470. [↗](https://arxiv.org/abs/2405.01470)
47. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Preprint, arXiv:2306.05685. [↗](https://arxiv.org/abs/2306.05685)
48. Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, et al. 2024. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Transformer Circuits Thread. [find](https://scholar.google.com/scholar?q=Adly+Templeton%2C+Tom+Conerly%2C+Jonathan+Marcus%2C+Jack+Lindsey%2C+Trenton+Bricken%2C+Brian+Chen%2C+Adam+Pearce%2C+Craig+Citro%2C+Emmanuel+Ameisen%2C+Andy+Jones%2C+et+al%2E+2024%2E+Scaling+monosemanticity%3A+Extracting+interp)
49. Ashish V. Thapliyal, Jordi Pont Tuset, Xi Chen, and Radu Soricut. 2022. [Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset](https://md.research.elamur.ai/free/en/papers/thapliyal-2022-crossmodal-3600-massively-multilingual-multimodal/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 715–729, Abu Dhabi, United Arab Emirates. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan,

*Updated 1 November 2026*