--- title: "LaMI — paper by Yariv et al. (2026) — summary and ideas" description: "Text-only LLMs often fail at visual commonsense questions, such as the colour of an animal's belly. 1 reference." authors: ["Guy Yariv", "Idan Schwartz", "Yossi Adi", "Sagie Benaim"] year: 2026 venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)" source: "https://aclanthology.org/2026.acl-short.3/" doi: "10.18653/v1/2026.acl-short.3" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/yariv-2026-lami-augmenting-large-language-models/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yariv-2026-lami-augmenting-large-language-models/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/38/) / LaMI: Augmenting Large Language Models via Late Multi-Image Fusion # LaMI: Augmenting Large Language Models via Late Multi-Image Fusion **Authors:** Guy Yariv, Idan Schwartz, [Yossi Adi](https://md.research.elamur.ai/free/en/authors/yossi-adi/) and Sagie Benaim · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.acl-short.3/) · [PDF](https://aclanthology.org/2026.acl-short.3.pdf) · [DOI 10.18653/v1/2026.acl-short.3](https://doi.org/10.18653/v1/2026.acl-short.3) · [OpenAlex](https://openalex.org/W4399913030) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary Text-only LLMs often fail at visual commonsense questions, such as the colour of an animal's belly. VLMs handle these questions better, but they need expensive multimodal training and often lose ground on purely textual reasoning. LaMI (Late Multi-Image fusion) adds visual knowledge to an existing LLM while keeping it unchanged. The LLM and a vision encoder both stay frozen. Only two small parts are trained: a Visual Token Projector, which turns image patch features into pseudo-text embeddings, and a Late Fusion Attention Layer. In that layer, the LLM's final hidden states attend to the visual tokens once, just before the prediction head. Real images are not available at test time, so a distilled text-to-image model generates several images from the prompt in one parallel batch. Each image produces its own prediction, and these are blended with the text-only prediction. The weighting uses CLIP image–text alignment, so poorly matching images give way to the text-only path. The method beats earlier visually-augmented LMs on object commonsense, rivals VLMs on visual tasks, and also slightly improves text benchmarks for models such as Llama3. This makes it a cheap way to add visual grounding to newly released LLMs. ## Key points - **Object commonsense with GPT-2:** For a fair comparison, LaMI was trained only on Visual Genome. It reaches 72.5 on Memory Color, 69.2 on Color Terms, 66.8 on shape and 85.5 on relative size, compared with 32.4, 34.6, 44.5 and 43.1 for plain GPT-2. On relative size it only marginally beats VaLM (85.0). - **Ablation:** Late fusion outperforms both early and intermediate fusion. Using multiple images helps under every fusion strategy, and combining late fusion with multiple images gives the best results. - **Comparison with VLMs (average over VC/CR/RC):** - Llama3-8B-Instruct improves from 61.2 to 62.7, ahead of Llava-Next at 60.7. - Qwen-2.5-7B-Instruct improves from 64.4 to 66.0, ahead of Qwen2-VL-7B-Instruct at 63.7. - Vicuna-7B improves from 53.4 to 55.1, ahead of InstructBLIP at 52.1. - **Number of images:** Gains level off at around k≈6, and k=3 already helps. With Gemma-2B, inference time is 389 ms with generated images, 58 ms with CLIP text embeddings and 20 ms for the baseline. Generated images score better than CLIP embeddings. - **Compute versus visual evidence:** On visual commonsense with Gemma-2B, a Best-of-N baseline with a matched compute budget reaches 47.8, while LaMI reaches 50.1. This suggests the gains come from visual evidence rather than extra compute. - **Failure mode:** When generated images are plausible but wrong, they can override a correct text prior. This is most likely for abstract or legendary concepts. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Bar-Ilan University](https://md.research.elamur.ai/free/en/universities/bar-ilan-university/) — University, Ramat Gan, Israel · 146 works · 24,845 citations - [Hebrew University of Jerusalem](https://md.research.elamur.ai/free/en/universities/hebrew-university-of-jerusalem/) — University, Jerusalem, Israel · 69 works · 9,594 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works 🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works ## Cite Cite as: Guy Yariv, Idan Schwartz, Yossi Adi and Sagie Benaim. 2026. LaMI: Augmenting Large Language Models via Late Multi-Image Fusion. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{yariv2026lami, title = {{LaMI: Augmenting Large Language Models via Late Multi-Image Fusion}}, author = {Guy Yariv, Idan Schwartz, Yossi Adi and Sagie Benaim}, booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)}, year = {2026}, doi = {10.18653/v1/2026.acl-short.3}, url = {https://doi.org/10.18653/v1/2026.acl-short.3} } ``` - [← previous LAMCL: A Length-aware Momentum Contrastive Learning Framework for Multiscale Machine-Revised Text Detection](https://md.research.elamur.ai/free/en/papers/zhou-2026-lamcl-length-aware-momentum-contrastive/) - [next → Lang2Act: Fine-Grained Visual Reasoning through Self-Emergent Linguistic Toolchains](https://md.research.elamur.ai/free/en/papers/xiong-2026-lang2act-fine-grained-visual-reasoning/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/38/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work. 1. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, and 8 others… — [Flamingo: a Visual Language Model for Few-Shot Learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/) *Updated 2 November 2026*