---
title: "When Seeing Overrides Knowing — paper by Ortu et al. (2026)"
description: "The authors build WHOOPS-AHA!, derived from the WHOOPS! collection of implausible scenes. 58 references, cited by 2 papers here."
authors: ["Francesco Ortu", "Zhijing Jin", "Diego Doimo", "Alberto Cazzaniga"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.642/"
doi: "10.18653/v1/2026.acl-long.642"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ortu-2026-when-seeing-overrides-knowing-disentangling/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ortu-2026-when-seeing-overrides-knowing-disentangling/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/9/) / When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Models

# When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Models

**Authors:** [Francesco Ortu](https://md.research.elamur.ai/free/en/authors/francesco-ortu/), [Zhijing Jin](https://md.research.elamur.ai/free/en/authors/zhijing-jin/), [Diego Doimo](https://md.research.elamur.ai/free/en/authors/diego-doimo/) and [Alberto Cazzaniga](https://md.research.elamur.ai/free/en/authors/alberto-cazzaniga/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.642/) · [PDF](https://aclanthology.org/2026.acl-long.642.pdf) · [DOI 10.18653/v1/2026.acl-long.642](https://doi.org/10.18653/v1/2026.acl-long.642) · [OpenAlex](https://openalex.org/W4416834506)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/ortu-2026-when-seeing-overrides-knowing-disentangling/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Vision-language models sometimes face images that contradict what they learned in pre-training, and such clashes can cause hallucinations. This paper asks how such models internally decide between the image and their stored knowledge. The authors build WHOOPS-AHA!, derived from the WHOOPS! collection of implausible scenes. Each image comes with a GPT-4o-written sentence prefix and two sets of continuations: one matching commonsense and one matching the odd picture. For example, a wolf howling at the sun could be completed with "moon" or "sun". Using logit attribution, they rank attention heads in LLaVA-NeXT-7B and Gemma3-12B by whether they push toward factual or counterfactual answers.

A small set of upper-layer heads drives this competition. Rescaling these heads' attention to image tokens shifts the model toward either visual evidence or prior knowledge. The heads are engaged specifically under conflict, not during ordinary visual processing. Their attention maps also locate the anomalous image regions more precisely than gradient-based attribution. These findings point toward VLMs that can better calibrate how much they trust vision versus memory.

## Key points
- **Dataset construction:** WHOOPS-AHA! builds on the 500 scenes of WHOOPS!. GPT-4.1 and Gemini-2.5-Flash acted as judges of quality.
  - Grammaticality averaged about 2.94 out of 3.
  - Alignment averaged 4.43 (knowledge-based completions) and 4.69 (image-based completions) out of 5.
  - Two human raters checked a 20-item subset against the LLM ratings.
- **Models analysed:** LLaVA-NeXT-7B (32 layers × 32 heads) and Gemma3-12B (48 layers × 16 heads). Both are late-fusion architectures.
- **Intervention method:** At the final token position, the post-softmax attention to image tokens in the selected heads is scaled by a factor involving λ. This causally steers outputs in either direction.
- **Region attribution:** Attention from the conflict-resolution heads localizes counterfactual image regions better than gradient-based methods.
- **Limitations and release:** The analysis relies on the approximate Logit Lens, uses single-token probes, and ignores MLP layers and visual encoders. Code and data are available at GitHub "francescortu/Seeing-Knowing" for non-commercial research use.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Toronto](https://md.research.elamur.ai/free/en/universities/university-of-toronto/) — University, Toronto, Canada · 96 works · 142,007 citations
- [Vector Institute](https://md.research.elamur.ai/free/en/labs/vector-institute/) — Research institute, Toronto, Canada · 33 works · 4,535 citations
- [University of Trieste](https://md.research.elamur.ai/free/en/universities/university-of-trieste/) — University, Trieste, Italy · 3 works · 48 citations
- [AREA Science Park](https://md.research.elamur.ai/free/en/labs/area-science-park/) — Research institute, Trieste, Italy · 2 works · 25 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 657 works

## Cite

Cite as: Francesco Ortu, Zhijing Jin, Diego Doimo and Alberto Cazzaniga. 2026. When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ortu2026when,
  title = {{When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Models}},
  author = {Francesco Ortu, Zhijing Jin, Diego Doimo and Alberto Cazzaniga},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.642},
  url = {https://doi.org/10.18653/v1/2026.acl-long.642}
}
```

- [← previous When More Thinking Hurts: Overthinking in LLM Test-Time Compute Scaling](https://md.research.elamur.ai/free/en/papers/zhou-2026-when-more-thinking-hurts-overthinking/)
- [next → Which Works Best for Vietnamese? A Practical Study of Information Retrieval Methods across Domains](https://md.research.elamur.ai/free/en/papers/nguyen-2026-which-works-best-vietnamese-practical/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/9/)

## References

58 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy B… [find](https://scholar.google.com/scholar?q=Jean%2DBaptiste+Alayrac%2C+Jeff+Donahue%2C+Pauline+Luc%2C+Antoine+Miech%2C+Iain+Barr%2C+Yana+Hasson%2C+Karel+Lenc%2C+Arthur+Mensch%2C+Katherine+Millican%2C+Malcolm+Reynolds%2C+Roman+Ring%2C+Eliza+Rutherford%2C+Serkan+Cabi%2C+Ten)
2. [Flamingo: a visual language model for few-shot learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/). In Advances in Neural Information Processing Systems, volume 35, pages 23716–23736. Curran Associates, Inc.
3. Lorenzo Basile, Valentino Maiorca, Diego Doimo, Francesco Locatello, and Alberto Cazzaniga. 2025. Head pursuit: Probing attention specialization in multimodal transformers. In Advances in Neural Information Processing Systems (NeurIPS). [find](https://scholar.google.com/scholar?q=Lorenzo+Basile%2C+Valentino+Maiorca%2C+Diego+Doimo%2C+Francesco+Locatello%2C+and+Alberto+Cazzaniga%2E+2025%2E+Head+pursuit%3A+Probing+attention+specialization+in+multimodal+transformers%2E+In+Advances+in+Neural+Infor)
4. Samyadeep Basu, Martin Grayson, Cecily Morrison, Besmira Nushi, Soheil Feizi, and Daniela Massiceti. 2024. Understanding Information Storage and Transfer in Multi-modal Large Language Models. arXiv preprint. ArXiv:2406.04236. — [arXiv:2406.04236](https://arxiv.org/abs/2406.04236) [find](https://scholar.google.com/scholar?q=Samyadeep+Basu%2C+Martin+Grayson%2C+Cecily+Morrison%2C+Besmira+Nushi%2C+Soheil+Feizi%2C+and+Daniela+Massiceti%2E+2024%2E+Understanding+Information+Storage+and+Transfer+in+Multi%2Dmodal+Large+Language+Models%2E+arXiv+pr)
5. Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. 2023. [Eliciting latent predictions from transformers with the tuned lens](https://md.research.elamur.ai/free/en/papers/belrose-2023-eliciting-latent-predictions-transformers-tuned/). CoRR, abs/2303.08112.
6. Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, Jie Lei, Tengyu Ma, Baishan Guo, Arpit Kalla, Markus Marks, Joseph Greer, Meng Wang, Peize Sun, Roman Rädle, Triantafyllos Afouras, Effrosyn… — [arXiv:2511.16719](https://arxiv.org/abs/2511.16719) [find](https://scholar.google.com/scholar?q=Nicolas+Carion%2C+Laura+Gustafson%2C+Yuan%2DTing+Hu%2C+Shoubhik+Debnath%2C+Ronghang+Hu%2C+Didac+Suris%2C+Chaitanya+Ryali%2C+Kalyan+Vasudev+Alwala%2C+Haitham+Khedr%2C+Andrew+Huang%2C+Jie+Lei%2C+Tengyu+Ma%2C+Baishan+Guo%2C+Arpit+K)
7. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit S. Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan-Jiang Jiang, Krishna Har… — [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://md.research.elamur.ai/free/en/papers/comanici-2025-gemini-2-5-pushing-frontier/)
8. Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Linjun Zhang, James Zou, and Huaxiu Yao. 2023. Holistic analysis of hallucination in gpt-4v(ision): Bias and interference challenges. CoRR, abs/2311.03287. — [arXiv:2311.03287](https://arxiv.org/abs/2311.03287) [find](https://scholar.google.com/scholar?q=Chenhang+Cui%2C+Yiyang+Zhou%2C+Xinyu+Yang%2C+Shirley+Wu%2C+Linjun+Zhang%2C+James+Zou%2C+and+Huaxiu+Yao%2E+2023%2E+Holistic+analysis+of+hallucination+in+gpt%2D4v%28ision%29%3A+Bias+and+interference+challenges%2E+CoRR%2C+abs%2F2311%2E)
9. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8493–8502.… [find](https://scholar.google.com/scholar?q=Damai+Dai%2C+Li+Dong%2C+Yaru+Hao%2C+Zhifang+Sui%2C+Baobao+Chang%2C+and+Furu+Wei%2E+2022%2E+Knowledge+neurons+in+pretrained+transformers%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computationa)
10. Nitzan Bitton Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, and Roy Schwartz. 2023. Breaking common sense: Whoops! A vision-and-language benchmark of synthetic and compositional images. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris… [find](https://scholar.google.com/scholar?q=Nitzan+Bitton+Guetta%2C+Yonatan+Bitton%2C+Jack+Hessel%2C+Ludwig+Schmidt%2C+Yuval+Elovici%2C+Gabriel+Stanovsky%2C+and+Roy+Schwartz%2E+2023%2E+Breaking+common+sense%3A+Whoops%21+A+vision%2Dand%2Dlanguage+benchmark+of+synthetic)
11. Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hend… — [Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models](https://md.research.elamur.ai/free/en/papers/deitke-2024-molmo-pixmo-open-weights-open/)
12. Danny Halawi, Jean-Stanislas Denain, and Jacob Steinhardt. 2023. Overthinking the truth: Understanding how language models process false demonstrations. CoRR, abs/2307.09476. [find](https://scholar.google.com/scholar?q=Danny+Halawi%2C+Jean%2DStanislas+Denain%2C+and+Jacob+Steinhardt%2E+2023%2E+Overthinking+the+truth%3A+Understanding+how+language+models+process+false+demonstrations%2E+CoRR%2C+abs%2F2307%2E09476%2E)
13. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Ja… [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Neel+Nanda%2C+Catherine+Olsson%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Tom+Conerly%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Zac+Hatfield%2DDodds%2C+Da)
14. Jinghan He, Kuan Zhu, Haiyun Guo, Junfeng Fang, Zhenglin Hua, Yuheng Jia, Ming Tang, Tat-Seng Chua, and Jinqiao Wang. 2025. [Cracking the code of hallucination in lvlms with vision-aware head divergence](https://md.research.elamur.ai/free/en/papers/he-2025-cracking-code-hallucination-lvlms-vision/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volum…
15. Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023. Dissecting recall of factual associations in auto-regressive language models. CoRR, abs/2304.14767. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Jasmijn+Bastings%2C+Katja+Filippova%2C+and+Amir+Globerson%2E+2023%2E+Dissecting+recall+of+factual+associations+in+auto%2Dregressive+language+models%2E+CoRR%2C+abs%2F2304%2E14767%2E)
16. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 548… [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Roei+Schuster%2C+Jonathan+Berant%2C+and+Omer+Levy%2E+2021%2E+Transformer+feed%2Dforward+layers+are+key%2Dvalue+memories%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Pr)
17. Michal Golovanevsky, William Rudman, Michael Lepori, Amir Bar, Ritambhara Singh, and Carsten Eickhoff. 2025a. [Pixels versus priors: Controlling knowledge priors in vision-language models through visual counterfacts](https://md.research.elamur.ai/free/en/papers/golovanevsky-2025-pixels-versus-priors-controlling-knowledge/). CoRR, abs/2505.17127.
18. Michal Golovanevsky, William Rudman, Vedant Palit, Carsten Eickhoff, and Ritambhara Singh. 2025b. What do VLMs NOTICE? a mechanistic interpretability pipeline for Gaussian-noise-free text-image corruption and evaluation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of… [find](https://scholar.google.com/scholar?q=Michal+Golovanevsky%2C+William+Rudman%2C+Vedant+Palit%2C+Carsten+Eickhoff%2C+and+Ritambhara+Singh%2E+2025b%2E+What+do+VLMs+NOTICE%3F+a+mechanistic+interpretability+pipeline+for+Gaussian%2Dnoise%2Dfree+text%2Dimage+corrup)
19. Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. 2024. [Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models](https://md.research.elamur.ai/free/en/papers/guan-2023-hallusionbench-advanced-diagnostic-suite-entangled/).…
20. Tianyang Han, Qing Lian, Rui Pan, Renjie Pi, Jipeng Zhang, Shizhe Diao, Yong Lin, and Tong Zhang. 2024. The instinctive bias: Spurious images lead to illusion in MLLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16163–16177, Miami, Florida, USA.… [find](https://scholar.google.com/scholar?q=Tianyang+Han%2C+Qing+Lian%2C+Rui+Pan%2C+Renjie+Pi%2C+Jipeng+Zhang%2C+Shizhe+Diao%2C+Yong+Lin%2C+and+Tong+Zhang%2E+2024%2E+The+instinctive+bias%3A+Spurious+images+lead+to+illusion+in+MLLMs%2E+In+Proceedings+of+the+2024+Conf)
21. Michael Hanna, Ollie Liu, and Alexandre Variengien. 2023. [How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model](https://md.research.elamur.ai/free/en/papers/hanna-2023-gpt-2-compute-greater-than/). CoRR, abs/2305.00586.
22. Tianze Hua, Tian Yun, and Ellie Pavlick. 2025. How do vision-language models process conflicting information across modalities? CoRR, abs/2507.01790. [find](https://scholar.google.com/scholar?q=Tianze+Hua%2C+Tian+Yun%2C+and+Ellie+Pavlick%2E+2025%2E+How+do+vision%2Dlanguage+models+process+conflicting+information+across+modalities%3F+CoRR%2C+abs%2F2507%2E01790%2E)
23. Nick Jiang, Anish Kachinthaya, Suzanne Petryk, and Yossi Gandelsman. 2025. Interpreting and editing vision-language representations to mitigate hallucinations. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Nick+Jiang%2C+Anish+Kachinthaya%2C+Suzanne+Petryk%2C+and+Yossi+Gandelsman%2E+2025%2E+Interpreting+and+editing+vision%2Dlanguage+representations+to+mitigate+hallucinations%2E+In+The+Thirteenth+International+Conferen)
24. Zhuoran Jin, Pengfei Cao, Hongbang Yuan, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, and Jun Zhao. 2024. Cutting off the head ends the conflict: A mechanism for interpreting and mitigating knowledge conflicts in language models. In Findings of the Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Zhuoran+Jin%2C+Pengfei+Cao%2C+Hongbang+Yuan%2C+Yubo+Chen%2C+Jiexin+Xu%2C+Huaijun+Li%2C+Xiaojian+Jiang%2C+Kang+Liu%2C+and+Jun+Zhao%2E+2024%2E+Cutting+off+the+head+ends+the+conflict%3A+A+mechanism+for+interpreting+and+mitiga)
25. Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean-Bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu,… [find](https://scholar.google.com/scholar?q=Aishwarya+Kamath%2C+Johan+Ferret%2C+Shreya+Pathak%2C+Nino+Vieillard%2C+Ramona+Merhej%2C+Sarah+Perrin%2C+Tatiana+Matejovicova%2C+Alexandre+Ram%C3%A9%2C+Morgane+Rivi%C3%A8re%2C+Louis+Rouillard%2C+Thomas+Mesnard%2C+Geoffrey+Cideron%2C+Je)
26. Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Tomáš Kočiský, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. 2021. Mind the gap: Assessing temporal generalization in neural lang… [find](https://scholar.google.com/scholar?q=Angeliki+Lazaridou%2C+Adhiguna+Kuncoro%2C+Elena+Gribovskaya%2C+Devang+Agrawal%2C+Adam+Liska%2C+Tayfun+Terzi%2C+Mai+Gimenez%2C+Cyprien+de+Masson+d%E2%80%99Autume%2C+Tom%C3%A1%C5%A1+Koc%CC%8Cisk%C3%BD%2C+Sebastian+Ruder%2C+Dani+Yogatama%2C+Kris+Cao%2C+Su)
27. Tiep Le, Vasudev Lal, and Phillip Howard. 2023. Coco-counterfactuals: Automatically constructed counterfactual examples for image-text pairs. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA,… [find](https://scholar.google.com/scholar?q=Tiep+Le%2C+Vasudev+Lal%2C+and+Phillip+Howard%2E+2023%2E+Coco%2Dcounterfactuals%3A+Automatically+constructed+counterfactual+examples+for+image%2Dtext+pairs%2E+In+Advances+in+Neural+Information+Processing+Systems+36%3A+A)
28. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. [BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation](https://md.research.elamur.ai/free/en/papers/li-2022-blip-bootstrapping-language-image-pre/). In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages…
29. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. [Evaluating object hallucination in large vision-language models](https://md.research.elamur.ai/free/en/papers/li-2023-evaluating-object-hallucination-large-vision/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 292–305. As…
30. Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. 2024a. [Mitigating hallucination in large multi-modal models via robust instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-mitigating-hallucination-large-multi-modal/). In The Twelfth International Conference on Learning Representations.
31. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024b. Llava-next: Improved reasoning, ocr, and world knowledge. [find](https://scholar.google.com/scholar?q=Haotian+Liu%2C+Chunyuan+Li%2C+Yuheng+Li%2C+Bo+Li%2C+Yuanhan+Zhang%2C+Sheng+Shen%2C+and+Yong+Jae+Lee%2E+2024b%2E+Llava%2Dnext%3A+Improved+reasoning%2C+ocr%2C+and+world+knowledge%2E)
32. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). In Advances in Neural Information Processing Systems, volume 36, pages 34892– 34916. Curran Associates, Inc.
33. Xiaoyuan Liu, Wenxuan Wang, Youliang Yuan, Jen tse Huang, Qiuzhi Liu, Pinjia He, and Zhaopeng Tu. 2024c. [Insight over sight?](https://md.research.elamur.ai/free/en/papers/liu-2025-insight-over-sight-exploring-vision/) exploring the vision-knowledge conflicts in multimodal llms. Preprint, arXiv:2410.08145.
34. Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, and Yang Liu. 2024d. [Prompt injection attack against llm-integrated applications](https://md.research.elamur.ai/free/en/papers/liu-2023-prompt-injection-attack-against-llm/). Preprint, arXiv:2306.05499.
35. Kelvin Luu, Daniel Khashabi, Suchin Gururangan, Karishma Mandyam, and Noah A. Smith. 2022. [Time waits for no one! analysis and challenges of temporal misalignment](https://md.research.elamur.ai/free/en/papers/luu-2022-time-waits-no-one-analysis/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tec…
36. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. In NeurIPS. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+David+Bau%2C+Alex+Andonian%2C+and+Yonatan+Belinkov%2E+2022%2E+Locating+and+editing+factual+associations+in+GPT%2E+In+NeurIPS%2E)
37. Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. 2023. [Progress measures for grokking via mechanistic interpretability](https://md.research.elamur.ai/free/en/papers/nanda-2023-progress-measures-grokking-mechanistic-interpretability/). In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
38. Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. 2025. Towards interpreting visual information processing in vision-language models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. [find](https://scholar.google.com/scholar?q=Clement+Neo%2C+Luke+Ong%2C+Philip+Torr%2C+Mor+Geva%2C+David+Krueger%2C+and+Fazl+Barez%2E+2025%2E+Towards+interpreting+visual+information+processing+in+vision%2Dlanguage+models%2E+In+The+Thirteenth+International+Confere)
39. Yaniv Nikankin, Dana Arad, Yossi Gandelsman, and Yonatan Belinkov. 2025. Same task, different circuits: Disentangling modality-specific mechanisms in VLMs. In Advances in Neural Information Processing Systems, volume 38. [find](https://scholar.google.com/scholar?q=Yaniv+Nikankin%2C+Dana+Arad%2C+Yossi+Gandelsman%2C+and+Yonatan+Belinkov%2E+2025%2E+Same+task%2C+different+circuits%3A+Disentangling+modality%2Dspecific+mechanisms+in+VLMs%2E+In+Advances+in+Neural+Information+Processing)
40. Nostalgebraist. 2020. interpreting gpt: the logit lens. Accessed: Nov 2023. [find](https://scholar.google.com/scholar?q=Nostalgebraist%2E+2020%2E+interpreting+gpt%3A+the+logit+lens%2E+Accessed%3A+Nov+2023%2E)
41. OpenAI. 2025. Introducing gpt-4.1 in the api. [https://openai.com/index/gpt-4-1/](https://openai.com/index/gpt-4-1/). Accessed: 2025-09-22. [↗](https://openai.com/index/gpt-4-1/)
42. OpenRouter, Inc. 2025. Openrouter: The unified interface for llms. [find](https://scholar.google.com/scholar?q=OpenRouter%2C+Inc%2E+2025%2E+Openrouter%3A+The+unified+interface+for+llms%2E)
43. Francesco Ortu, Zhijing Jin, Diego Doimo, Mrinmaya Sachan, Alberto Cazzaniga, and Bernhard Schölkopf. 2024. [Competition of mechanisms: Tracing how language models handle facts and counterfactuals](https://md.research.elamur.ai/free/en/papers/ortu-2024-competition-mechanisms-tracing-how-language/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: L…
44. Vedant Palit, Rohan Pandey, Aryaman Arora, and Paul Pu Liang. 2023. Towards vision-language mechanistic interpretability: A causal tracing tool for BLIP. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 2856–2861. [find](https://scholar.google.com/scholar?q=Vedant+Palit%2C+Rohan+Pandey%2C+Aryaman+Arora%2C+and+Paul+Pu+Liang%2E+2023%2E+Towards+vision%2Dlanguage+mechanistic+interpretability%3A+A+causal+tracing+tool+for+BLIP%2E+In+Proceedings+of+the+IEEE%2FCVF+International+C)
45. Anirudh Phukan, Divyansh, Harshit Kumar Morj, Vaishnavi, Apoorv Saxena, and Koustava Goswami. 2025. Beyond logit lens: Contextual embeddings for robust hallucination detection & grounding in vlms. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Com… [find](https://scholar.google.com/scholar?q=Anirudh+Phukan%2C+Divyansh%2C+Harshit+Kumar+Morj%2C+Vaishnavi%2C+Apoorv+Saxena%2C+and+Koustava+Goswami%2E+2025%2E+Beyond+logit+lens%3A+Contextual+embeddings+for+robust+hallucination+detection+%26+grounding+in+vlms%2E+In+)
46. Sreetama Sarkar, Yue Che, Alex Gavin, Peter Anthony Beerel, and Souvik Kundu. 2025. Mitigating hallucinations in vision-language models through image-guided head suppression. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, Novembe… [find](https://scholar.google.com/scholar?q=Sreetama+Sarkar%2C+Yue+Che%2C+Alex+Gavin%2C+Peter+Anthony+Beerel%2C+and+Souvik+Kundu%2E+2025%2E+Mitigating+hallucinations+in+vision%2Dlanguage+models+through+image%2Dguided+head+suppression%2E+In+Proceedings+of+the+202)
47. Sarah Schwettmann, Neil Chowdhury, Samuel Klein, David Bau, and Antonio Torralba. 2023. Multimodal neurons in pretrained text-only transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 2862–2867. [find](https://scholar.google.com/scholar?q=Sarah+Schwettmann%2C+Neil+Chowdhury%2C+Samuel+Klein%2C+David+Bau%2C+and+Antonio+Torralba%2E+2023%2E+Multimodal+neurons+in+pretrained+text%2Donly+transformers%2E+In+Proceedings+of+the+IEEE%2FCVF+International+Conference)
48. Alessandro Serra, Francesco Ortu, Emanuele Panizon, Lucrezia Valeriani, Lorenzo Basile, Alessio Ansuini, Diego Doimo, and Alberto Cazzaniga. 2024. The narrow gate: Localized image-text communication in vision-language models. CoRR, abs/2412.06646. mt-bench and chatbot arena. In Advances in Neural In… — [arXiv:2412.06646](https://arxiv.org/abs/2412.06646) [find](https://scholar.google.com/scholar?q=Alessandro+Serra%2C+Francesco+Ortu%2C+Emanuele+Panizon%2C+Lucrezia+Valeriani%2C+Lorenzo+Basile%2C+Alessio+Ansuini%2C+Diego+Doimo%2C+and+Alberto+Cazzaniga%2E+2024%2E+The+narrow+gate%3A+Localized+image%2Dtext+communication+i)
49. Tinghui Zhu, Qin Liu, Fei Wang, Zhengzhong Tu, and Muhao Chen. 2024. Unraveling cross-modality knowledge conflicts in large vision-language models. CoRR, abs/2410.03659. — [arXiv:2410.03659](https://arxiv.org/abs/2410.03659) [find](https://scholar.google.com/scholar?q=Tinghui+Zhu%2C+Qin+Liu%2C+Fei+Wang%2C+Zhengzhong+Tu%2C+and+Muhao+Chen%2E+2024%2E+Unraveling+cross%2Dmodality+knowledge+conflicts+in+large+vision%2Dlanguage+models%2E+CoRR%2C+abs%2F2410%2E03659%2E+%E2%80%94+%5BarXiv%3A2410%2E03659%5D%28https%3A%2F%2Fa)
50. Chameleon Team. 2024. [Chameleon: Mixed-modal early-fusion foundation models](https://md.research.elamur.ai/free/en/papers/team-2024-chameleon-mixed-modal-early-fusion/). arXiv preprint arXiv:2405.09818.
51. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
52. Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. 2025. [Retrieval head mechanistically explains long-context factuality](https://md.research.elamur.ai/free/en/papers/wu-2024-retrieval-head-mechanistically-explains-long/). In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.
53. Rongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei Xu. 2024. [Knowledge conflicts for LLMs: A survey](https://md.research.elamur.ai/free/en/papers/xu-2024-knowledge-conflicts-llms-survey/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8541–8565, Miami, Florida, USA. Association for Computational Linguisti…
54. Le Yang, Ziwei Zheng, Boxu Chen, Zhengyu Zhao, Chenhao Lin, and Chao Shen. 2025a. Nullu: Mitigating object hallucinations in large vision-language models via halluspace projection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025, pag… [find](https://scholar.google.com/scholar?q=Le+Yang%2C+Ziwei+Zheng%2C+Boxu+Chen%2C+Zhengyu+Zhao%2C+Chenhao+Lin%2C+and+Chao+Shen%2E+2025a%2E+Nullu%3A+Mitigating+object+hallucinations+in+large+vision%2Dlanguage+models+via+halluspace+projection%2E+In+IEEE%2FCVF+Confere)
55. Tianyun Yang, Ziniu Li, Juan Cao, and Chang Xu. 2025b. Understanding and mitigating hallucination in large vision-language models via modular attribution and intervention. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. [find](https://scholar.google.com/scholar?q=Tianyun+Yang%2C+Ziniu+Li%2C+Juan+Cao%2C+and+Chang+Xu%2E+2025b%2E+Understanding+and+mitigating+hallucination+in+large+vision%2Dlanguage+models+via+modular+attribution+and+intervention%2E+In+The+Thirteenth+Internatio)
56. Qinan Yu, Jack Merullo, and Ellie Pavlick. 2023. Characterizing mechanisms for factual recall in language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9924–9959, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Qinan+Yu%2C+Jack+Merullo%2C+and+Ellie+Pavlick%2E+2023%2E+Characterizing+mechanisms+for+factual+recall+in+language+models%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+Natural+Language+Process)
57. Zeping Yu and Sophia Ananiadou. 2024. Understanding multimodal LLMs: the mechanistic interpretability of LLaVA in visual question answering. arXiv preprint arXiv:2411.10950. — [arXiv:2411.10950](https://arxiv.org/abs/2411.10950) [find](https://scholar.google.com/scholar?q=Zeping+Yu+and+Sophia+Ananiadou%2E+2024%2E+Understanding+multimodal+LLMs%3A+the+mechanistic+interpretability+of+LLaVA+in+visual+question+answering%2E+arXiv+preprint+arXiv%3A2411%2E10950%2E+%E2%80%94+%5BarXiv%3A2411%2E10950%5D%28https)
58. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+P%2E+Xing%2C+Hao+Zhang%2C+Joseph+E%2E+Gonzalez%2C+and+Ion+Stoica%2E+2023%2E+Judging+llm%2Das)

*Updated 31 October 2026*