---
title: "When Relations Break — paper by Shin et al. (2026)"
description: "The authors conclude that VLMs need more geometry-aware designs, which matters for… 99.66% accuracy on Reefknot MCQ. 27 references."
authors: ["Philip Wootaek Shin", "Ajay Narayanan Sridhar", "Lakshmi Sivani Devarapalli", "Rui Zhang", "Jack Sampson", "Vijaykrishnan Narayanan"]
year: 2026
venue: "Proceedings of the 4th Workshop on Advances in Language and Vision Research (ALVR)"
source: "https://aclanthology.org/2026.alvr-main.14/"
doi: "10.18653/v1/2026.alvr-main.14"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shin-2026-when-relations-break-analyzing-relation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shin-2026-when-relations-break-analyzing-relation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/48/) / When Relations Break: Analyzing Relation Hallucination in Vision-Language Model Under Rotation and Noise

# When Relations Break: Analyzing Relation Hallucination in Vision-Language Model Under Rotation and Noise

**Authors:** Philip Wootaek Shin, Ajay Narayanan Sridhar, Lakshmi Sivani Devarapalli, [Rui Zhang](https://md.research.elamur.ai/free/en/authors/rui-zhang/), Jack Sampson and Vijaykrishnan Narayanan · **Year:** 2026 · **Venue:** [Proceedings of the 4th Workshop on Advances in Language and Vision Research (ALVR)](https://md.research.elamur.ai/free/en/venues/alvr/) ([ALVR 2026](https://md.research.elamur.ai/free/en/venues/alvr/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.alvr-main.14/) · [PDF](https://aclanthology.org/2026.alvr-main.14.pdf) · [DOI 10.18653/v1/2026.alvr-main.14](https://doi.org/10.18653/v1/2026.alvr-main.14) · [OpenAlex](https://openalex.org/W7166858381)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper asks how fragile vision–language models are at relational reasoning, meaning how objects interact or are positioned, when input images are rotated or degraded. Humans still read relations correctly in a turned or noisy photo, but VLMs often do not. The authors built non-overlapping evaluation subsets from three relation-hallucination benchmarks: MMRel, R-Bench and Reefknot.

They tested open-source and closed-source models under 90°/270° rotations and four corruptions that keep scene geometry intact:
- Impulse Noise
- Gaussian Blur
- Pixelate
- Saturate

They then compared two kinds of fixes. The first tells the model about the distortion in the prompt. The second cleans the image before inference, using an orientation detector or restoration models.

Even mild distortions consistently lowered accuracy. Rotation hurt most on MMRel. Prompt hints gave almost no benefit. Orientation correction worked well. Denoising helped only in some conditions. Better perceptual image quality did not reliably lead to better relational answers. The authors conclude that VLMs need more geometry-aware designs, which matters for real-world, imperfect imagery.

## Key points
- **Evaluation sets:**
  - MMRel: 1,632 image–question pairs
  - R-Bench: 3,466 pairs
  - Reefknot: 1,185 multiple-choice and 2,922 yes/no questions
- **Models tested:**
  - Open-source, for the rotation direction check: Qwen2-VL 7B, InternVL2 8B, LLaVA-Next 8B, DeepSeek-Janus 7B, LLaMA-3.2 11B (Vision)
  - Closed-source, for the main study: GPT-5.1, Gemini 2.5 Pro, Claude Sonnet 4.5
- **Effect of clockwise rotation on MMRel yes/no accuracy:**
  - GPT-5.1: 89.83% → 58.82%
  - Gemini 2.5 Pro: 90.87% → 53.25%
  - Claude Sonnet 4.5: 64.03% → 30.45%
  - Gemini 2.5 Pro also fell from 75.97% to 48.93% on R-Bench.
- **Orientation detector (Barbosa, 2025):**
  - It recognized rotation with 99.66% accuracy on Reefknot MCQ and 99.38% on Y/N.
  - After correction, GPT-5.1 reached 95.04% on cognitive MCQ.
  - Adding rotation metadata to the prompt barely changed accuracy.
- **Denoising with GPT-5.1 (severity 2):**
  - On MMRel, the accuracy drop shrank from roughly 2.5–4.3 pp to about 0.5–2.2 pp.
  - On R-Bench, the drop stayed at roughly 9–12 pp.
  - Saturate sometimes raised accuracy as severity increased.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Pennsylvania State University](https://md.research.elamur.ai/free/en/universities/pennsylvania-state-university/) — University, State College, United States · 56 works · 5,360 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 657 works

## Cite

Cite as: Philip Wootaek Shin, Ajay Narayanan Sridhar, Lakshmi Sivani Devarapalli, Rui Zhang, Jack Sampson and Vijaykrishnan Narayanan. 2026. When Relations Break: Analyzing Relation Hallucination in Vision-Language Model Under Rotation and Noise. Proceedings of the 4th Workshop on Advances in Language and Vision Research (ALVR).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shin2026when,
  title = {{When Relations Break: Analyzing Relation Hallucination in Vision-Language Model Under Rotation and Noise}},
  author = {Philip Wootaek Shin, Ajay Narayanan Sridhar, Lakshmi Sivani Devarapalli, Rui Zhang, Jack Sampson and Vijaykrishnan Narayanan},
  booktitle = {Proceedings of the 4th Workshop on Advances in Language and Vision Research (ALVR)},
  year = {2026},
  doi = {10.18653/v1/2026.alvr-main.14},
  url = {https://doi.org/10.18653/v1/2026.alvr-main.14}
}
```

- [← previous When Prompt Optimization Becomes Jailbreaking: Adaptive Red-Teaming of Large Language Models](https://md.research.elamur.ai/free/en/papers/shamsi-2026-when-prompt-optimization-becomes-jailbreaking/)
- [next → When Rules Learn: A Self-Evolving Agent for Legal Case Retrieval](https://md.research.elamur.ai/free/en/papers/tao-2026-when-rules-learn-self-evolving/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/48/)

## References

27 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anthropic. 2026. Claude sonnet 4.5. [https://claude.ai](https://claude.ai). Large language model. [↗](https://claude.ai)
2. Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. 2025. Hallucination of multimodal [large language models: A survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/).
3. Duarte Barbosa. 2025. Deep image orientation detection. [https://github.com/duartebarbosadev/](https://github.com/duartebarbosadev/) deep-image-orientation-detection. GitHub repository. [↗](https://github.com/duartebarbosadev/)
4. I Chen, Wei-Ting Chen, Yu-Wei Liu, Yuan-Chun Chiang, Sy-Yen Kuo, Ming-Hsuan Yang, and 1 others. 2025. Unirestore: Unified perceptual and task-oriented image restoration model using diffusion prior. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 17969–17979. [find](https://scholar.google.com/scholar?q=I+Chen%2C+Wei%2DTing+Chen%2C+Yu%2DWei+Liu%2C+Yuan%2DChun+Chiang%2C+Sy%2DYen+Kuo%2C+Ming%2DHsuan+Yang%2C+and+1+others%2E+2025%2E+Unirestore%3A+Unified+perceptual+and+task%2Doriented+image+restoration+model+using+diffusion+prior%2E+In)
5. Zhe Chen, Wenhai Wang, Yue Cao, and 1 others. 2024. [Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks](https://md.research.elamur.ai/free/en/papers/chen-2023-internvl-scaling-up-vision/). arXiv preprint arXiv:2312.14238.
6. Google DeepMind. 2026. Gemini 2.5 pro. [https://deepmind.google](https://deepmind.google). Large language model. [↗](https://deepmind.google)
7. DeepSeek-AI. 2024. [Janus: Decoupling visual encoding for unified multimodal understanding and generation](https://md.research.elamur.ai/free/en/papers/wu-2024-janus-decoupling-visual-encoding/). arXiv preprint arXiv:2410.13848.
8. Yongchao Feng, Yajie Liu, Shuai Yang, Wenrui Cai, Jinqing Zhang, Qiqi Zhan, Ziyue Huang, Hongxi Yan, Qiao Wan, Chenguang Liu, and 1 others. 2025. Vision-language model for object detection and segmentation: A review and evaluation. arXiv preprint arXiv:2504.09480. — [arXiv:2504.09480](https://arxiv.org/abs/2504.09480) [find](https://scholar.google.com/scholar?q=Yongchao+Feng%2C+Yajie+Liu%2C+Shuai+Yang%2C+Wenrui+Cai%2C+Jinqing+Zhang%2C+Qiqi+Zhan%2C+Ziyue+Huang%2C+Hongxi+Yan%2C+Qiao+Wan%2C+Chenguang+Liu%2C+and+1+others%2E+2025%2E+Vision%2Dlanguage+model+for+object+detection+and+segment)
9. Dan Hendrycks and Thomas Dietterich. 2019. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks+and+Thomas+Dietterich%2E+2019%2E+Benchmarking+neural+network+robustness+to+common+corruptions+and+perturbations%2E+arXiv+preprint+arXiv%3A1903%2E12261%2E)
10. Chaoya Jiang, Yongrui Heng, Wei Ye, Han Yang, Haiyang Xu, Ming Yan, Ji Zhang, Fei Huang, and Shikun Zhang. 2025. Vlm-r3: Region recognition, reasoning, and refinement for enhanced multimodal chain-of-thought. arXiv preprint arXiv:2505.16192. — [arXiv:2505.16192](https://arxiv.org/abs/2505.16192) [find](https://scholar.google.com/scholar?q=Chaoya+Jiang%2C+Yongrui+Heng%2C+Wei+Ye%2C+Han+Yang%2C+Haiyang+Xu%2C+Ming+Yan%2C+Ji+Zhang%2C+Fei+Huang%2C+and+Shikun+Zhang%2E+2025%2E+Vlm%2Dr3%3A+Region+recognition%2C+reasoning%2C+and+refinement+for+enhanced+multimodal+chain%2Dof%2D)
11. Sheng Jin, Xueying Jiang, Jiaxing Huang, Lewei Lu, and Shijian Lu. 2024. Llms meet vlms: Boost open vocabulary object detection with fine-grained descriptors. arXiv preprint arXiv:2402.04630. — [arXiv:2402.04630](https://arxiv.org/abs/2402.04630) [find](https://scholar.google.com/scholar?q=Sheng+Jin%2C+Xueying+Jiang%2C+Jiaxing+Huang%2C+Lewei+Lu%2C+and+Shijian+Lu%2E+2024%2E+Llms+meet+vlms%3A+Boost+open+vocabulary+object+detection+with+fine%2Dgrained+descriptors%2E+arXiv+preprint+arXiv%3A2402%2E04630%2E+%E2%80%94+%5BarXiv)
12. Guibiao Liao, Jiankun Li, and Xiaoqing Ye. 2024. Vlm2scene: Self-supervised image-text-lidar learning with foundation models for autonomous driving scene understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 3351–3359. [find](https://scholar.google.com/scholar?q=Guibiao+Liao%2C+Jiankun+Li%2C+and+Xiaoqing+Ye%2E+2024%2E+Vlm2scene%3A+Self%2Dsupervised+image%2Dtext%2Dlidar+learning+with+foundation+models+for+autonomous+driving+scene+understanding%2E+In+Proceedings+of+the+AAAI+Conf)
13. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. [Improved baselines with visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-improved-baselines-visual-instruction-tuning/). arXiv preprint arXiv:2310.03744.
14. Meta AI. 2024. [Llama 3.2: Open multimodal foundation models](https://md.research.elamur.ai/free/en/papers/deitke-2024-molmo-pixmo-open-weights-open/). arXiv preprint arXiv:2409.17146.
15. Jiahao Nie, Gongjie Zhang, Wenbin An, Yun Xing, Yap-Peng Tan, Alex C. Kot, and Shijian Lu. 2025. Mmrel: Benchmarking relation understanding in multi-modal large language models. Preprint, arXiv:2406.09121. — [arXiv:2406.09121](https://arxiv.org/abs/2406.09121) [find](https://scholar.google.com/scholar?q=Jiahao+Nie%2C+Gongjie+Zhang%2C+Wenbin+An%2C+Yun+Xing%2C+Yap%2DPeng+Tan%2C+Alex+C%2E+Kot%2C+and+Shijian+Lu%2E+2025%2E+Mmrel%3A+Benchmarking+relation+understanding+in+multi%2Dmodal+large+language+models%2E+Preprint%2C+arXiv%3A2406%2E0)
16. Tianyi Niu, Jaemin Cho, Elias Stengel-Eskin, and Mohit Bansal. 2026. Rotbench: Evaluating multimodal large language models on identifying image rotation. Preprint, arXiv:2508.13968. — [arXiv:2508.13968](https://arxiv.org/abs/2508.13968) [find](https://scholar.google.com/scholar?q=Tianyi+Niu%2C+Jaemin+Cho%2C+Elias+Stengel%2DEskin%2C+and+Mohit+Bansal%2E+2026%2E+Rotbench%3A+Evaluating+multimodal+large+language+models+on+identifying+image+rotation%2E+Preprint%2C+arXiv%3A2508%2E13968%2E+%E2%80%94+%5BarXiv%3A2508%2E1396)
17. OpenAI. 2026. Chatgpt-5.1. [https://chat.openai.com](https://chat.openai.com). Large language model. [↗](https://chat.openai.com)
18. Surya Selvam, Ravi K Rajendran, Murugan Sankaradas, Anand Raghunathan, and Srimat T Chakradhar. 2025. Simcache: similarity caching for efficient vlm-based scene understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 3327–3336. [find](https://scholar.google.com/scholar?q=Surya+Selvam%2C+Ravi+K+Rajendran%2C+Murugan+Sankaradas%2C+Anand+Raghunathan%2C+and+Srimat+T+Chakradhar%2E+2025%2E+Simcache%3A+similarity+caching+for+efficient+vlm%2Dbased+scene+understanding%2E+In+Proceedings+of+the+Co)
19. Philip Wootaek Shin, Jack Sampson, Vijaykrishnan Narayanan, Andres Marquez, and Mahantesh Halappanavar. 2025. Losing the plot: How vlm responses degrade on imperfect charts. Preprint, arXiv:2509.18425. — [arXiv:2509.18425](https://arxiv.org/abs/2509.18425) [find](https://scholar.google.com/scholar?q=Philip+Wootaek+Shin%2C+Jack+Sampson%2C+Vijaykrishnan+Narayanan%2C+Andres+Marquez%2C+and+Mahantesh+Halappanavar%2E+2025%2E+Losing+the+plot%3A+How+vlm+responses+degrade+on+imperfect+charts%2E+Preprint%2C+arXiv%3A2509%2E18425)
20. Peng Wang, Shuai Bai, Shengbang Tan, Shuai Wang, Zhihao Fan, and 1 others. 2024. [Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution](https://md.research.elamur.ai/free/en/papers/wang-2024-qwen2-vl-enhancing-vision/). arXiv preprint arXiv:2409.12191.
21. Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612. [find](https://scholar.google.com/scholar?q=Zhou+Wang%2C+Alan+C+Bovik%2C+Hamid+R+Sheikh%2C+and+Eero+P+Simoncelli%2E+2004%2E+Image+quality+assessment%3A+from+error+visibility+to+structural+similarity%2E+IEEE+transactions+on+image+processing%2C+13%284%29%3A600%E2%80%93612%2E)
22. Mingrui Wu, Jiayi Ji, Oucheng Huang, Jiale Li, Yuhang Wu, Xiaoshuai Sun, and Rongrong Ji. 2024. Evaluating and analyzing relationship hallucinations in large vision-language models. Preprint, arXiv:2406.16449. — [arXiv:2406.16449](https://arxiv.org/abs/2406.16449) [find](https://scholar.google.com/scholar?q=Mingrui+Wu%2C+Jiayi+Ji%2C+Oucheng+Huang%2C+Jiale+Li%2C+Yuhang+Wu%2C+Xiaoshuai+Sun%2C+and+Rongrong+Ji%2E+2024%2E+Evaluating+and+analyzing+relationship+hallucinations+in+large+vision%2Dlanguage+models%2E+Preprint%2C+arXiv%3A24)
23. Weiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang, Xizhou Zhu, and 1 others. 2025. Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models. arXiv preprint arXiv:2504.15279. — [arXiv:2504.15279](https://arxiv.org/abs/2504.15279) [find](https://scholar.google.com/scholar?q=Weiye+Xu%2C+Jiahao+Wang%2C+Weiyun+Wang%2C+Zhe+Chen%2C+Wengang+Zhou%2C+Aijun+Yang%2C+Lewei+Lu%2C+Houqiang+Li%2C+Xiaohua+Wang%2C+Xizhou+Zhu%2C+and+1+others%2E+2025%2E+Visulogic%3A+A+benchmark+for+evaluating+visual+reasoning+in+m)
24. Fanghua Yu, Jinjin Gu, Zheyuan Li, Jinfan Hu, Xiangtao Kong, Xintao Wang, Jingwen He, Yu Qiao, and Chao Dong. 2024. Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild. Preprint, arXiv:2401.13627. — [arXiv:2401.13627](https://arxiv.org/abs/2401.13627) [find](https://scholar.google.com/scholar?q=Fanghua+Yu%2C+Jinjin+Gu%2C+Zheyuan+Li%2C+Jinfan+Hu%2C+Xiangtao+Kong%2C+Xintao+Wang%2C+Jingwen+He%2C+Yu+Qiao%2C+and+Chao+Dong%2E+2024%2E+Scaling+up+to+excellence%3A+Practicing+model+scaling+for+photo%2Drealistic+image+restora)
25. Hao Zhang, Wenqi Shao, Hong Liu, Yongqiang Ma, Ping Luo, Yu Qiao, Nanning Zheng, and Kaipeng Zhang. 2025. B-avibench: Toward evaluating the robustness of large vision-language model on black-box adversarial visual-instructions. Trans. Info. For. Sec., 20:1434–1446. [find](https://scholar.google.com/scholar?q=Hao+Zhang%2C+Wenqi+Shao%2C+Hong+Liu%2C+Yongqiang+Ma%2C+Ping+Luo%2C+Yu+Qiao%2C+Nanning+Zheng%2C+and+Kaipeng+Zhang%2E+2025%2E+B%2Davibench%3A+Toward+evaluating+the+robustness+of+large+vision%2Dlanguage+model+on+black%2Dbox+adver)
26. Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595. [find](https://scholar.google.com/scholar?q=Richard+Zhang%2C+Phillip+Isola%2C+Alexei+A+Efros%2C+Eli+Shechtman%2C+and+Oliver+Wang%2E+2018%2E+The+unreasonable+effectiveness+of+deep+features+as+a+perceptual+metric%2E+In+Proceedings+of+the+IEEE+conference+on+com)
27. Kening Zheng, Junkai Chen, Yibo Yan, Xin Zou, and Xuming Hu. 2025. Reefknot: A comprehensive benchmark for relation hallucination evaluation, analysis and mitigation in multimodal large language models. Preprint, arXiv:2408.09429. [find](https://scholar.google.com/scholar?q=Kening+Zheng%2C+Junkai+Chen%2C+Yibo+Yan%2C+Xin+Zou%2C+and+Xuming+Hu%2E+2025%2E+Reefknot%3A+A+comprehensive+benchmark+for+relation+hallucination+evaluation%2C+analysis+and+mitigation+in+multimodal+large+language+model)

*Updated 1 November 2026*