--- title: "Visual Grounding of Inter-lingual Word-Embeddings (2022)" description: "The authors present this as a first baseline for inter-lingual visual grounding. 4 references." authors: ["Wafaa Mohammed", "Hassan Shahmohammadi", "Hendrik P. A. Lensch", "R. Harald Baayen"] year: 2022 venue: "Proceedings of the Workshop on Unimodal and Multimodal Induction of Linguistic Structures (UM-IoS)" source: "https://aclanthology.org/2022.umios-1.3/" doi: "10.18653/v1/2022.umios-1.3" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/mohammed-2022-visual-grounding-inter-lingual-word/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mohammed-2022-visual-grounding-inter-lingual-word/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/9/) / Visual Grounding of Inter-lingual Word-Embeddings # Visual Grounding of Inter-lingual Word-Embeddings **Authors:** [Wafaa Mohammed](https://md.research.elamur.ai/free/en/authors/wafaa-mohammed/), Hassan Shahmohammadi, Hendrik P. A. Lensch and [R. Harald Baayen](https://md.research.elamur.ai/free/en/authors/r-harald-baayen/) · **Year:** 2022 · **Venue:** Proceedings of the Workshop on Unimodal and Multimodal Induction of Linguistic Structures (UM-IoS) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2022.umios-1.3/) · [PDF](https://aclanthology.org/2022.umios-1.3.pdf) · [DOI 10.18653/v1/2022.umios-1.3](https://doi.org/10.18653/v1/2022.umios-1.3) · [OpenAlex](https://openalex.org/W4385573193) **Citations:** 1 citations (OpenAlex, 2026-10-14) ## Summary Most work on visually grounding word embeddings targets English alone, so it is unclear how grounding behaves when several languages share the process. This paper extends an existing monolingual grounding model (Shahmohammadi et al., 2022) to English, German and Arabic. Frozen GloVe vectors for caption words pass through one shared linear "alignment" layer into a grounded space. Per-language LSTMs then encode each caption, and training pushes every caption encoding toward the same Inception-V3 image vector from MS COCO 2017. The German and Arabic captions are machine translations of the English ones. After training, the alignment matrix alone converts any textual word vector into a grounded one, including words not seen in the captions. On word-similarity benchmarks, grounded vectors clearly beat text-only ones. Pairing English with German helps further, while adding Arabic slightly lowers similarity scores. On categorization (BLESS), the pattern reverses: Arabic gives English the largest gain. The authors present this as a first baseline for inter-lingual visual grounding. ## Key points - Data: MS COCO 2017 (123,287 images, 5 captions each). German captions come from Biswas et al. (2021), translated with Fairseq. Arabic captions come from Hashim (2020), made with the Google cloud translation API, and cover only about 82k images (77k train / 5k validation). All language combinations were rerun on this 82k subset for fairness. - Setup: - Encoders: GloVe input vectors and the Inception-V3 image encoder, both frozen. - Vector sizes: grounded vectors have 1024 dimensions; LSTM outputs and image vectors have 2048. - Training: NAdam with learning rate 0.001, batch size 256, 20 epochs, early-stopping patience of 5. - Vocabulary: 10k words for English and 30k for the more inflected German and Arabic. - Similarity benchmarks: - English: WordSim353, MEN, RW, MTurk, SimVerb and SimLex999. - German: multilingual WordSim353 and SimLex999. - Arabic: Almarsoomi, MC30, Saif40 and WordSim. - Similarity results: grounding improved over text-only embeddings in all three languages. Adding Arabic caused a small drop but still beat the text baseline. - Statistical analysis: a Gaussian location-scale GAMM on BLESS relation scores (word as random effect) showed a trade-off. Monolingual grounding gave the highest mean. Inter-lingual grounding gave the lowest variance, meaning tighter semantic clusters. ## Interesting ideas - Restricting the text-to-grounded mapping to a single linear matrix only nudges the original vectors, so the learned matrix can be applied afterward to any vocabulary item, giving zero-shot coverage. - Several languages are trained against one shared, language-neutral target (the image vector), with one loss per language summed, so each language keeps its own encoder but shares the alignment. - The evaluation models both the mean and the variance of scores (location-scale GAMM with per-word random effects), which revealed that a method can lose on average quality yet win on consistency. - All models were retrained on the subset where every language has data, so differences reflect the method rather than the amount of data. - Similarity and categorization benchmarks ranked the same embeddings in opposite orders, which shows that a single benchmark family can mislead. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (world): A small linear map from word vectors to compiled ontology features (WordNet classes, measures) could extend that knowledge to out-of-lexicon words while staying inspectable as one matrix. - **I2** (back): Translation through the meta-world could use the meaning graph as the shared pivot, scoring each language's realization separately against the same graph. - **I3** (learn): Evaluation reports could track per-relation or per-construction variance across items, not just mean accuracy, to catch repairs that make results more erratic. - **I4** (learn): When comparing witness parsers or teacher tiers, restrict scoring to the sentences all of them cover so that coverage gaps do not distort agreement figures. - **I5** (answer): Readers should be judged on several test types at once (school tests, word problems, fable QA), and improvements should be accepted only when they do not reverse on another type. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Tübingen](https://md.research.elamur.ai/free/en/universities/university-of-tubingen/) — University, Tübingen, Germany · 78 works · 41,280 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works 🧭 Topics (2): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 3,066 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works ## Cite Cite as: Wafaa Mohammed, Hassan Shahmohammadi, Hendrik P. A. Lensch and R. Harald Baayen. 2022. Visual Grounding of Inter-lingual Word-Embeddings. Proceedings of the Workshop on Unimodal and Multimodal Induction of Linguistic Structures (UM-IoS). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{mohammed2022visual, title = {{Visual Grounding of Inter-lingual Word-Embeddings}}, author = {Wafaa Mohammed, Hassan Shahmohammadi, Hendrik P. A. Lensch and R. Harald Baayen}, booktitle = {Proceedings of the Workshop on Unimodal and Multimodal Induction of Linguistic Structures (UM-IoS)}, year = {2022}, doi = {10.18653/v1/2022.umios-1.3}, url = {https://doi.org/10.18653/v1/2022.umios-1.3} } ``` - [← previous Universal Dependencies and Semantics for English and Hebrew Child-directed Speech](https://md.research.elamur.ai/free/en/papers/szubert-2022-universal-dependencies-semantics-english-hebrew/) - [next → Word-Embeddings Distinguish Denominal and Root-Derived Verbs in Semitic](https://md.research.elamur.ai/free/en/papers/benbaji-2022-word-embeddings-distinguish-denominal-root/) - [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/9/) ## References 4 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. 2022. [Towards a Cleaner Document-Oriented Multilingual Crawled Corpus](https://md.research.elamur.ai/free/en/papers/abadji-2022-towards-cleaner-document-oriented-multilingual/). arXiv eprints, page arXiv:2201.06642. [↗](https://arxiv.org/abs/2201.06642) 2. Faaza A Almarsoomi, James D OShea, Zuhair Bandar, and Keeley Crockett. 2013. Awss: An algorithm for measuring arabic word semantic similarity. In 2013 IEEE international conference on systems, man, and cybernetics, pages 504–509. IEEE. We hope that these initial steps towards interlingual visual grounding inspire further research. Low-resourced languages might benefit from joint processing with high-resourced languages in multilingual models but one has to make sure that their unique characteristics are not overwhelmed and masked by datasets acquired in different cultural settings. [find](https://scholar.google.com/scholar?q=Faaza+A+Almarsoomi%2C+James+D+OShea%2C+Zuhair+Bandar%2C+and+Keeley+Crockett%2E+2013%2E+Awss%3A+An+algorithm+for+measuring+arabic+word+semantic+similarity%2E+In+2013+IEEE+international+conference+on+systems%2C+man%2C+an) 3. Abdulrahman Almuhareb and Massimo Poesio. 2005. Concept learning and categorization from the web. In proceedings of the annual meeting of the Cognitive Science society, volume 27. [find](https://scholar.google.com/scholar?q=Abdulrahman+Almuhareb+and+Massimo+Poesio%2E+2005%2E+Concept+learning+and+categorization+from+the+web%2E+In+proceedings+of+the+annual+meeting+of+the+Cognitive+Science+society%2C+volume+27%2E) 4. Jordi Armengol-Estapé, Casimiro Pio Carrino, Carlos Rodriguez-Penagos, Ona de Gibert Bonet, Carme Armentano-Oller, Aitor Gonzalez-Agirre, Maite Melero, and Marta Villegas. 2021. Are multilingual models the best choice for moderately under-resourced languages? a comprehensive assessment for catalan. arXiv preprint arXiv:2107.07903. [↗](https://arxiv.org/abs/2107.07903) *Updated 1 November 2026*