--- title: "Evaluating Cross-Lingual Behavior and Consistency of" description: "The authors build two benchmarks from Google Landmarks Dataset v2 (GLDv2). 2 references." authors: ["Hao Wang", "Pinzhi Huang", "Daisuke Kawahara"] year: 2026 venue: "Proceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026)" source: "https://aclanthology.org/2026.trustnlp-main.1/" doi: "10.18653/v1/2026.trustnlp-main.1" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/wang-2026-evaluating-cross-lingual-behavior-consistency/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2026-evaluating-cross-lingual-behavior-consistency/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/29/) / Evaluating Cross-Lingual Behavior and Consistency of Multimodal Large Language Models # Evaluating Cross-Lingual Behavior and Consistency of Multimodal Large Language Models **Authors:** [Hao Wang](https://md.research.elamur.ai/free/en/authors/hao-wang/), Pinzhi Huang and [Daisuke Kawahara](https://md.research.elamur.ai/free/en/authors/daisuke-kawahara/) · **Year:** 2026 · **Venue:** [Proceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026)](https://md.research.elamur.ai/free/en/venues/trustnlp/) ([Workshop on Trustworthy NLP 2026](https://md.research.elamur.ai/free/en/venues/trustnlp/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.trustnlp-main.1/) · [PDF](https://aclanthology.org/2026.trustnlp-main.1.pdf) · [DOI 10.18653/v1/2026.trustnlp-main.1](https://doi.org/10.18653/v1/2026.trustnlp-main.1) · [OpenAlex](https://openalex.org/W7166841967) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary The paper examines whether multimodal LLMs give equally correct answers regardless of the language of the question, using travel to famous landmarks as the scenario. The authors build two benchmarks from Google Landmarks Dataset v2 (GLDv2). KnowRecall is a multiple-choice VQA set: a landmark photo is paired with cultural and historical questions, which Gemini-1.5-Pro generated from English Wikipedia and translated into 15 languages. VisRecall gives a model only a landmark's name in one of 9 languages and asks it to describe how the place looks. Descriptions are scored with CLIPScore against reference photos. Three settings are compared: English, the landmark's local language, and all other languages. Consistency is measured by how much the correct answers, or the scores, overlap across language pairs. Every open-weight and proprietary model tested does best in English and worst in "other" languages. Structured chain-of-thought prompting narrows this gap. Text-only backbones often describe landmarks better than the multimodal models built on them. This matters for any multilingual or culturally grounded deployment, including for less-represented languages. ## Key points - **KnowRecall** has 3,000 questions about 1,500 landmarks, with 100 landmarks per language and 2 questions each, in 15 languages. Only landmarks in countries with a single official language were included. Ukrainian is not among the languages. - **VisRecall** covers 450 landmarks (50 per language) in 9 languages. Descriptions are translated to English and scored by CLIPScore against up to 20 GLDv2 images. - **KnowRecall accuracy (EN / LOC / GLO):** GPT-4o leads with 68.3 / 69.2 / 64.4. Gemini-2.0-Flash has the highest consistency (86.3). The best open-weight model is Qwen2.5-VL-7B-IT, at 56.6 / 55.2 / 51.0 with consistency 80.9. - **Structured CoT** asks the model to identify the landmark, translate the question, and then reason. It raises GPT-4o's GLO accuracy to about 69 and its consistency to about 89–90. - **Language patterns:** consistency is high within the Romance languages and between Chinese and Japanese, but low for Greek and Hebrew. In VisRecall, base LLMs beat their MLLM versions except for the Qwen2.5 pair. Llama-3-8B-IT, however, answered in the prompt's language only 30.9% of the time. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) — University, New York, United States · 201 works · 100,180 citations - [Waseda University](https://md.research.elamur.ai/free/en/universities/waseda-university/) — University, Tokyo, Japan · 20 works · 584 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works 🧭 Topics (4): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 828 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 657 works ## Cite Cite as: Hao Wang, Pinzhi Huang and Daisuke Kawahara. 2026. Evaluating Cross-Lingual Behavior and Consistency of Multimodal Large Language Models. Proceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{wang2026evaluating, title = {{Evaluating Cross-Lingual Behavior and Consistency of Multimodal Large Language Models}}, author = {Hao Wang, Pinzhi Huang and Daisuke Kawahara}, booktitle = {Proceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026)}, year = {2026}, doi = {10.18653/v1/2026.trustnlp-main.1}, url = {https://doi.org/10.18653/v1/2026.trustnlp-main.1} } ``` - [← previous Evaluating Cost-Efficiency of LLMs in a RAG Setup on Polish Wikipedia: Quality vs. Energy Consumption](https://md.research.elamur.ai/free/en/papers/smits-2026-evaluating-cost-efficiency-llms-rag/) - [next → Evaluating Data Augmentation Strategies for Training Spanish Misspelling Detection Models](https://md.research.elamur.ai/free/en/papers/sancho-2026-evaluating-data-augmentation-strategies-training/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/29/) ## References 2 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425–2433. [find](https://scholar.google.com/scholar?q=Stanislaw+Antol%2C+Aishwarya+Agrawal%2C+Jiasen+Lu%2C+Margaret+Mitchell%2C+Dhruv+Batra%2C+C+Lawrence+Zitnick%2C+and+Devi+Parikh%2E+2015%2E+Vqa%3A+Visual+question+answering%2E+In+Proceedings+of+the+IEEE+international+confe) 2. We constructed the KnowRecall dataset using Gemini for translation. Given the extensive number of language variants and the large volume of VQA questions, it was impractical to double-verify every translation. Consequently, some translation errors may be present. We provide further discussion on Gem… — [Qwen2.5-VL Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2025-qwen2-5-vl-technical/) *Updated 1 November 2026*