---
title: "Analyzing and Mitigating Object Hallucination in Large"
description: "The authors propose LURE (LVLM hallucination REvisor), a lightweight post-hoc corrector. 57 references, cited by 105 papers here."
authors: ["Yiyang Zhou", "Chenhang Cui", "Jaehong Yoon", "Linjun Zhang", "Zhun Deng", "Chelsea Finn", "Mohit Bansal", "Huaxiu Yao"]
year: 2023
venue: "arXiv:2310.00754"
source: "https://arxiv.org/abs/2310.00754"
doi: "10.48550/arxiv.2310.00754"
arxiv: "2310.00754"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhou-2023-analyzing-mitigating-object-hallucination-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhou-2023-analyzing-mitigating-object-hallucination-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / Analyzing and Mitigating Object Hallucination in Large Vision-Language Models

# Analyzing and Mitigating Object Hallucination in Large Vision-Language Models

**Authors:** [Yiyang Zhou](https://md.research.elamur.ai/free/en/authors/yiyang-zhou/), Chenhang Cui, [Jaehong Yoon](https://md.research.elamur.ai/free/en/authors/jaehong-yoon/), Linjun Zhang, Zhun Deng, [Chelsea Finn](https://md.research.elamur.ai/free/en/authors/chelsea-finn/), [Mohit Bansal](https://md.research.elamur.ai/free/en/authors/mohit-bansal/) and [Huaxiu Yao](https://md.research.elamur.ai/free/en/authors/huaxiu-yao/) · **Year:** 2023 · **Venue:** arXiv:2310.00754 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2310.00754) · [PDF](https://arxiv.org/pdf/2310.00754) · [DOI 10.48550/arxiv.2310.00754](https://doi.org/10.48550/arxiv.2310.00754) · [arXiv 2310.00754](https://arxiv.org/abs/2310.00754) · [OpenAlex](https://openalex.org/W4387323833)

**Citations:** 29 citations (OpenAlex, 2026-10-14) · [105](https://md.research.elamur.ai/free/en/papers/zhou-2023-analyzing-mitigating-object-hallucination-large/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large vision-language models (LVLMs) often describe objects that are not present in the image. This object hallucination undermines downstream uses such as robotics, medical imaging and human-computer interaction. Earlier fixes either targeted small multimodal models or required expensive curated fine-tuning data. The authors propose LURE (LVLM hallucination REvisor), a lightweight post-hoc corrector. It rewrites a model's output into a more faithful description.

The design rests on a statistical and theoretical study that links hallucination to three factors:
- objects that frequently co-occur in training data,
- high decoding uncertainty,
- late position in the generated text.

To train the revisor, GPT-3.5 corrupts correct captions in two ways. It adds plausible co-occurring objects, and it masks uncertain or late-appearing objects with a placeholder tag. The revisor then learns to undo these errors. Because it works on text output, it can be attached to any LVLM. Across six open-source LVLMs, LURE beat the strongest earlier approach on CHAIR, GPT-based scoring and human judgments. This offers a cheap, model-agnostic way to improve caption reliability.

## Key points
- The analysis defines three diagnostic measures. CoScore captures the co-occurrence overlap of hallucinated objects with other mentioned objects. UnScore is the negative log-probability of an object token. PoScore is an object's relative index within the description.
- Hallucinated objects cluster at higher co-occurrence and uncertainty values and toward the end of descriptions. The authors attribute the positional effect to errors accumulating during autoregressive decoding.
- The training data for the revisor is synthetic. GPT-3.5 inserts likely co-occurring objects into correct captions and swaps uncertain or tail objects for a placeholder tag.
- LURE was tested on six open-source LVLMs against baselines that include Teacher (BLIP-2 reference captions), CoT and GPT-Teacher. Evaluation used CHAIR, GPT evaluation and human evaluation.
- The code and data are released at github.com/YiyangZhou/LURE (arXiv:2310.00754).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 506 works · 330,052 citations
- [Columbia University](https://md.research.elamur.ai/free/en/universities/columbia-university/) — University, New York, United States · 254 works · 56,747 citations
- [University of North Carolina at Chapel Hill](https://md.research.elamur.ai/free/en/universities/university-of-north-carolina-at-chapel-hill/) — University, Chapel Hill, United States · 74 works · 23,278 citations
- [Rutgers, The State University of New Jersey](https://md.research.elamur.ai/free/en/universities/rutgers-the-state-university-of-new-jersey/) — University, New Brunswick, United States · 40 works · 12,486 citations

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works

## Cite

Cite as: Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal and Huaxiu Yao. 2023. Analyzing and Mitigating Object Hallucination in Large Vision-Language Models. arXiv:2310.00754.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhou2023analyzing,
  title = {{Analyzing and Mitigating Object Hallucination in Large Vision-Language Models}},
  author = {Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal and Huaxiu Yao},
  year = {2023},
  doi = {10.48550/arxiv.2310.00754},
  eprint = {2310.00754},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.00754}
}
```

- [← previous SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects](https://md.research.elamur.ai/free/en/papers/adelani-2023-sib-200-simple-inclusive-big/)
- [next → FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation](https://md.research.elamur.ai/free/en/papers/vu-2023-freshllms-refreshing-large-language-models/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

57 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. [Spice: Semantic propositional image caption evaluation](https://md.research.elamur.ai/free/en/papers/anderson-2016-spice-semantic-propositional-image-caption/). In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pp. 382–398.
2. Satanjeev Banerjee and Alon Lavie. [Meteor: An automatic metric for mt evaluation with improved correlation with human judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/). In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65–72, 2005.
3. Ali Furkan Biten, Lluı́s Gómez, and Dimosthenis Karatzas. Let there be a clock on the beach: Reducing object hallucination in image captioning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1381–1390, 2022. [find](https://scholar.google.com/scholar?q=Ali+Furkan+Biten%2C+Llu%C4%B1%CC%81s+Go%CC%81mez%2C+and+Dimosthenis+Karatzas%2E+Let+there+be+a+clock+on+the+beach%3A+Reducing+object+hallucination+in+image+captioning%2E+In+Proceedings+of+the+IEEE%2FCVF+Winter+Conference+on+App)
4. Paul Brie, Nicolas Burny, Arthur Sluÿters, and Jean Vanderdonckt. Evaluating a large language model on searching for gui layouts. Proceedings of the ACM on Human-Computer Interaction, 7 (EICS):1–37, 2023. [find](https://scholar.google.com/scholar?q=Paul+Brie%2C+Nicolas+Burny%2C+Arthur+Sluy%CC%88ters%2C+and+Jean+Vanderdonckt%2E+Evaluating+a+large+language+model+on+searching+for+gui+layouts%2E+Proceedings+of+the+ACM+on+Human%2DComputer+Interaction%2C+7+%28EICS%29%3A1%E2%80%9337%2C+)
5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020.
6. Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang. Unlabeled data improves adversarial robustness. Advances in neural information processing systems, 32, 2019. [find](https://scholar.google.com/scholar?q=Yair+Carmon%2C+Aditi+Raghunathan%2C+Ludwig+Schmidt%2C+John+C+Duchi%2C+and+Percy+S+Liang%2E+Unlabeled+data+improves+adversarial+robustness%2E+Advances+in+neural+information+processing+systems%2C+32%2C+2019%2E)
7. Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3558–3568, 2021. [find](https://scholar.google.com/scholar?q=Soravit+Changpinyo%2C+Piyush+Sharma%2C+Nan+Ding%2C+and+Radu+Soricut%2E+Conceptual+12m%3A+Pushing+web%2Dscale+image%2Dtext+pre%2Dtraining+to+recognize+long%2Dtail+visual+concepts%2E+In+Proceedings+of+the+IEEE%2FCVF+Conferen)
8. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL [https://lmsys.org/blog/2023-03-30-vicuna…](https://lmsys.org/blog/2023-03-30-vicuna…) [↗](https://lmsys.org/blog/2023-03-30-vicuna…)
9. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311, 2022.
10. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. [Instructblip: Towards general-purpose vision-language models with instruction tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/), 2023.
11. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. [find](https://scholar.google.com/scholar?q=Jia+Deng%2C+Wei+Dong%2C+Richard+Socher%2C+Li%2DJia+Li%2C+Kai+Li%2C+and+Li+Fei%2DFei%2E+Imagenet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+2009+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pp%2E+248%E2%80%93)
12. Jie Ding, Vahid Tarokh, and Yuhong Yang. Bridging aic and bic: a new criterion for autoregression. IEEE Transactions on Information Theory, 64(6):4024–4043, 2017. [find](https://scholar.google.com/scholar?q=Jie+Ding%2C+Vahid+Tarokh%2C+and+Yuhong+Yang%2E+Bridging+aic+and+bic%3A+a+new+criterion+for+autoregression%2E+IEEE+Transactions+on+Information+Theory%2C+64%286%29%3A4024%E2%80%934043%2C+2017%2E)
13. Markus Freitag and Yaser Al-Onaizan. [Beam search strategies for neural machine translation](https://md.research.elamur.ai/free/en/papers/freitag-2017-beam-search-strategies-neural-machine/). arXiv preprint arXiv:1702.01806, 2017.
14. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. [Mme: A comprehensive evaluation benchmark for multimodal large language models](https://md.research.elamur.ai/free/en/papers/fu-2023-mme-comprehensive-evaluation-benchmark-multimodal/). arXiv preprint arXiv:2306.13394, 2023.
15. Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: A vision and language model for dialogue with humans, 2023. [find](https://scholar.google.com/scholar?q=Tao+Gong%2C+Chengqi+Lyu%2C+Shilong+Zhang%2C+Yudong+Wang%2C+Miao+Zheng%2C+Qian+Zhao%2C+Kuikun+Liu%2C+Wenwei+Zhang%2C+Ping+Luo%2C+and+Kai+Chen%2E+Multimodal%2Dgpt%3A+A+vision+and+language+model+for+dialogue+with+humans%2C+2023%2E)
16. Anisha Gunjal, Jihan Yin, and Erhan Bas. [Detecting and preventing hallucinations in large vision language models](https://md.research.elamur.ai/free/en/papers/gunjal-2023-detecting-preventing-hallucinations-large-vision/). arXiv preprint arXiv:2308.06394, 2023.
17. Edward James Hannan, AJ McDougall, and Don Stephen Poskitt. Recursive estimation of autoregressions. Journal of the Royal Statistical Society: Series B (Methodological), 51(2):217–233, 1989. [find](https://scholar.google.com/scholar?q=Edward+James+Hannan%2C+AJ+McDougall%2C+and+Don+Stephen+Poskitt%2E+Recursive+estimation+of+autoregressions%2E+Journal+of+the+Royal+Statistical+Society%3A+Series+B+%28Methodological%29%2C+51%282%29%3A217%E2%80%93233%2C+1989%2E)
18. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. [The curious case of neural text degeneration](https://md.research.elamur.ai/free/en/papers/holtzman-2019-curious-case-neural-text-degeneration/). arXiv preprint arXiv:1904.09751, 2019.
19. Mingzhe Hu, Shaoyan Pan, Yuheng Li, and Xiaofeng Yang. Advancing medical imaging with language models: A journey from n-grams to chatgpt. arXiv preprint arXiv:2304.04920, 2023. — [arXiv:2304.04920](https://arxiv.org/abs/2304.04920) [find](https://scholar.google.com/scholar?q=Mingzhe+Hu%2C+Shaoyan+Pan%2C+Yuheng+Li%2C+and+Xiaofeng+Yang%2E+Advancing+medical+imaging+with+language+models%3A+A+journey+from+n%2Dgrams+to+chatgpt%2E+arXiv+preprint+arXiv%3A2304%2E04920%2C+2023%2E+%E2%80%94+%5BarXiv%3A2304%2E04920%5D%28ht)
20. Ching-Kang Ing. Accumulated prediction errors, information criteria and optimal forecasting for autoregressive time series. The Annals of Statistics, 35(3):1238–1277, 2007. [find](https://scholar.google.com/scholar?q=Ching%2DKang+Ing%2E+Accumulated+prediction+errors%2C+information+criteria+and+optimal+forecasting+for+autoregressive+time+series%2E+The+Annals+of+Statistics%2C+35%283%29%3A1238%E2%80%931277%2C+2007%2E)
21. Jae Myung Kim, A Koepke, Cordelia Schmid, and Zeynep Akata. Exposing and mitigating spurious correlations for cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2584–2594, 2023. [find](https://scholar.google.com/scholar?q=Jae+Myung+Kim%2C+A+Koepke%2C+Cordelia+Schmid%2C+and+Zeynep+Akata%2E+Exposing+and+mitigating+spurious+correlations+for+cross%2Dmodal+retrieval%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pa)
22. Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. [Otter: A multi-modal model with in-context instruction tuning](https://md.research.elamur.ai/free/en/papers/li-2023-otter-multi-modal-model-context/). arXiv preprint arXiv:2305.03726, 2023a.
23. Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694–9705, 2021. [find](https://scholar.google.com/scholar?q=Junnan+Li%2C+Ramprasaath+Selvaraju%2C+Akhilesh+Gotmare%2C+Shafiq+Joty%2C+Caiming+Xiong%2C+and+Steven+Chu+Hong+Hoi%2E+Align+before+fuse%3A+Vision+and+language+representation+learning+with+momentum+distillation%2E+Adva)
24. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. [Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). arXiv preprint arXiv:2301.12597, 2023b.
25. Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, et al. M 3 it: A large-scale dataset towards multi-modal multilingual instruction tuning. arXiv preprint arXiv:2306.04387, 2023c. — [arXiv:2306.04387](https://arxiv.org/abs/2306.04387) [find](https://scholar.google.com/scholar?q=Lei+Li%2C+Yuwei+Yin%2C+Shicheng+Li%2C+Liang+Chen%2C+Peiyi+Wang%2C+Shuhuai+Ren%2C+Mukai+Li%2C+Yazheng+Yang%2C+Jingjing+Xu%2C+Xu+Sun%2C+et+al%2E+M+3+it%3A+A+large%2Dscale+dataset+towards+multi%2Dmodal+multilingual+instruction+tuni)
26. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. [Evaluating object hallucination in large vision-language models](https://md.research.elamur.ai/free/en/papers/li-2023-evaluating-object-hallucination-large-vision/). arXiv preprint arXiv:2305.10355, 2023d.
27. Chin-Yew Lin. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out, pp. 74–81, 2004.
28. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. [Microsoft coco: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740–7…
29. Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306.[14565, 2023a](https://md.research.elamur.ai/free/en/papers/liu-2023-mitigating-hallucination-large-multi-modal/).
30. Haokun Liu, Yaonan Zhu, Kenji Kato, Izumi Kondo, Tadayoshi Aoyama, and Yasuhisa Hasegawa. Llm-based human-robot collaboration framework for manipulation tasks. arXiv preprint arXiv:2308.14972, 2023b. — [arXiv:2308.14972](https://arxiv.org/abs/2308.14972) [find](https://scholar.google.com/scholar?q=Haokun+Liu%2C+Yaonan+Zhu%2C+Kenji+Kato%2C+Izumi+Kondo%2C+Tadayoshi+Aoyama%2C+and+Yasuhisa+Hasegawa%2E+Llm%2Dbased+human%2Drobot+collaboration+framework+for+manipulation+tasks%2E+arXiv+preprint+arXiv%3A2308%2E14972%2C+2023b%2E+)
31. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023c. [find](https://scholar.google.com/scholar?q=Haotian+Liu%2C+Chunyuan+Li%2C+Qingyang+Wu%2C+and+Yong+Jae+Lee%2E+Visual+instruction+tuning%2C+2023c%2E)
32. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). arXiv preprint arXiv:2304.08485, 2023d.
33. Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. [Video-chatgpt: Towards detailed video understanding via large vision and language models](https://md.research.elamur.ai/free/en/papers/maaz-2023-video-chatgpt-towards-detailed-video/). arXiv preprint arXiv:2306.05424, 2023.
34. Jinjie Mai, Jun Chen, Bing Li, Guocheng Qian, Mohamed Elhoseiny, and Bernard Ghanem. Llm as a robotic brain: Unifying egocentric memory and control. arXiv preprint arXiv:2304.09349, 2023. — [arXiv:2304.09349](https://arxiv.org/abs/2304.09349) [find](https://scholar.google.com/scholar?q=Jinjie+Mai%2C+Jun+Chen%2C+Bing+Li%2C+Guocheng+Qian%2C+Mohamed+Elhoseiny%2C+and+Bernard+Ghanem%2E+Llm+as+a+robotic+brain%3A+Unifying+egocentric+memory+and+control%2E+arXiv+preprint+arXiv%3A2304%2E09349%2C+2023%2E+%E2%80%94+%5BarXiv%3A230)
35. Gary M Olson, James D Herbsleb, and Henry H Reuter. Characterizing the sequential structure of interactive behaviors through statistical and grammatical techniques. Human–Computer Interaction, 9(3-4):427–472, 1994. [find](https://scholar.google.com/scholar?q=Gary+M+Olson%2C+James+D+Herbsleb%2C+and+Henry+H+Reuter%2E+Characterizing+the+sequential+structure+of+interactive+behaviors+through+statistical+and+grammatical+techniques%2E+Human%E2%80%93Computer+Interaction%2C+9%283%2D4%29%3A)
36. Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems, 24, 2011. [find](https://scholar.google.com/scholar?q=Vicente+Ordonez%2C+Girish+Kulkarni%2C+and+Tamara+Berg%2E+Im2text%3A+Describing+images+using+1+million+captioned+photographs%2E+Advances+in+neural+information+processing+systems%2C+24%2C+2011%2E)
37. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311–318, 2002.
38. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/), 2021.
39. Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. [Object hallucination in image captioning](https://md.research.elamur.ai/free/en/papers/rohrbach-2018-object-hallucination-image-captioning/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4035–4045, 2018.
40. Swarnadeep Saha, Peter Hase, and Mohit Bansal. Can language models teach weaker agents? teacher explanations improve students via theory of mind. arXiv preprint arXiv:2306.09299, 2023. [find](https://scholar.google.com/scholar?q=Swarnadeep+Saha%2C+Peter+Hase%2C+and+Mohit+Bansal%2E+Can+language+models+teach+weaker+agents%3F+teacher+explanations+improve+students+via+theory+of+mind%2E+arXiv+preprint+arXiv%3A2306%2E09299%2C+2023%2E)
41. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. [Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556–2565, 2018.
42. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023. [↗](https://github.com/tatsu-lab/stanford_alpaca)
43. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
44. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. [Cider: Consensus-based image description evaluation](https://md.research.elamur.ai/free/en/papers/vedantam-2014-cider-consensus-based-image-description/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566–4575, 2015.
45. Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096–1103, 2008. [find](https://scholar.google.com/scholar?q=Pascal+Vincent%2C+Hugo+Larochelle%2C+Yoshua+Bengio%2C+and+Pierre%2DAntoine+Manzagol%2E+Extracting+and+composing+robust+features+with+denoising+autoencoders%2E+In+Proceedings+of+the+25th+international+conference+o)
46. Junyang Wang, Yiyang Zhou, Guohai Xu, Pengcheng Shi, Chenlin Zhao, Haiyang Xu, Qinghao Ye, Ming Yan, Ji Zhang, Jihua Zhu, et al. Evaluation and analysis of hallucination in large vision-language models. arXiv preprint arXiv:2308.15126, 2023a. — [arXiv:2308.15126](https://arxiv.org/abs/2308.15126) [find](https://scholar.google.com/scholar?q=Junyang+Wang%2C+Yiyang+Zhou%2C+Guohai+Xu%2C+Pengcheng+Shi%2C+Chenlin+Zhao%2C+Haiyang+Xu%2C+Qinghao+Ye%2C+Ming+Yan%2C+Ji+Zhang%2C+Jihua+Zhu%2C+et+al%2E+Evaluation+and+analysis+of+hallucination+in+large+vision%2Dlanguage+model)
47. Sheng Wang, Zihao Zhao, Xi Ouyang, Qian Wang, and Dinggang Shen. Chatcad: Interactive computer-aided diagnosis on medical image using large language models. arXiv preprint arXiv:2302.07257, 2023b. — [arXiv:2302.07257](https://arxiv.org/abs/2302.07257) [find](https://scholar.google.com/scholar?q=Sheng+Wang%2C+Zihao+Zhao%2C+Xi+Ouyang%2C+Qian+Wang%2C+and+Dinggang+Shen%2E+Chatcad%3A+Interactive+computer%2Daided+diagnosis+on+medical+image+using+large+language+models%2E+arXiv+preprint+arXiv%3A2302%2E07257%2C+2023b%2E+%E2%80%94+%5B)
48. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
49. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. [mplug-owl: Modularization empowers large language models with multimodality](https://md.research.elamur.ai/free/en/papers/ye-2023-mplug-owl-modularization-empowers-large/). arXiv preprint arXiv:2304.14178, 2023.
50. Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training: Aligning texts with visual concepts. arXiv preprint arXiv:2111.08276, 2021. — [arXiv:2111.08276](https://arxiv.org/abs/2111.08276) [find](https://scholar.google.com/scholar?q=Yan+Zeng%2C+Xinsong+Zhang%2C+and+Hang+Li%2E+Multi%2Dgrained+vision+language+pre%2Dtraining%3A+Aligning+texts+with+visual+concepts%2E+arXiv+preprint+arXiv%3A2111%2E08276%2C+2021%2E+%E2%80%94+%5BarXiv%3A2111%2E08276%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
51. Linjun Zhang, Zhun Deng, Kenji Kawaguchi, and James Zou. When and how mixup improves calibration. In International Conference on Machine Learning, pp. 26135–26160. PMLR, 2022a. [find](https://scholar.google.com/scholar?q=Linjun+Zhang%2C+Zhun+Deng%2C+Kenji+Kawaguchi%2C+and+James+Zou%2E+When+and+how+mixup+improves+calibration%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+26135%E2%80%9326160%2E+PMLR%2C+2022a%2E)
52. Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. [How language model hallucinations can snowball](https://md.research.elamur.ai/free/en/papers/zhang-2023-language-model-hallucinations-snowball/). arXiv preprint arXiv:2305.13534, 2023a.
53. Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. [Llama-adapter: Efficient fine-tuning of language models with zero-init attention](https://md.research.elamur.ai/free/en/papers/zhang-2023-llama-adapter-efficient-fine-tuning/). arXiv preprint arXiv:2303.16199, 2023b.
54. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. [Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv preprint arXiv:2205.01068, 2022b.
55. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+arXiv+preprint+arXiv%3A1904%2E09675%2C+2019%2E)
56. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). arXiv preprint arXiv:2306.05685, 2023.
57. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. [Minigpt-4: Enhancing vision-language understanding with advanced large language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/). arXiv preprint arXiv:2304.10592, 2023.

*Updated 2 November 2026*