---
title: "Investigating the Catastrophic Forgetting in Multimodal Large"
description: "They introduce EMT (Evaluating MulTimodality), which treats an MLLM as an image classifier. 61 references, cited by 43 papers here."
authors: ["Yuexiang Zhai", "Shengbang Tong", "Xiao Li", "Mu Cai", "Qing Qu", "Yong Jae Lee", "Yi Ma"]
year: 2023
venue: "arXiv:2309.10313"
source: "https://arxiv.org/abs/2309.10313"
doi: "10.48550/arxiv.2309.10313"
arxiv: "2309.10313"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhai-2023-investigating-catastrophic-forgetting-multimodal-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhai-2023-investigating-catastrophic-forgetting-multimodal-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/) / Investigating the Catastrophic Forgetting in Multimodal Large Language Models

# Investigating the Catastrophic Forgetting in Multimodal Large Language Models

**Authors:** [Yuexiang Zhai](https://md.research.elamur.ai/free/en/authors/yuexiang-zhai/), [Shengbang Tong](https://md.research.elamur.ai/free/en/authors/shengbang-tong/), Xiao Li, Mu Cai, Qing Qu, [Yong Jae Lee](https://md.research.elamur.ai/free/en/authors/yong-jae-lee/) and [Yi Ma](https://md.research.elamur.ai/free/en/authors/yi-ma/) · **Year:** 2023 · **Venue:** arXiv:2309.10313 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2309.10313) · [PDF](https://arxiv.org/pdf/2309.10313) · [DOI 10.48550/arxiv.2309.10313](https://doi.org/10.48550/arxiv.2309.10313) · [arXiv 2309.10313](https://arxiv.org/abs/2309.10313) · [OpenAlex](https://openalex.org/W4386907423)

**Citations:** 16 citations (OpenAlex, 2026-10-14) · [43](https://md.research.elamur.ai/free/en/papers/zhai-2023-investigating-catastrophic-forgetting-multimodal-large/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Multimodal large language models (MLLMs) are typically built by attaching a pre-trained vision encoder to an open-source LLM and then instruction-tuning the combination. The authors ask whether this process erodes the visual recognition ability the encoder originally had, a form of catastrophic forgetting that earlier MLLM benchmarks did not examine. They introduce EMT (Evaluating MulTimodality), which treats an MLLM as an image classifier. EMT prompts the model to name one label from a fixed class list, and a separate LLM then judges whether the answer is correct. Applied to several open-source MLLMs, EMT shows that nearly all of them score below the zero-shot accuracy of their own vision encoders. Hallucinated extra labels and unverifiable descriptions account for much of this gap. Further fine-tuning experiments with LLaVA reveal a two-phase pattern. Brief tuning improves visual–text alignment and transfers to other datasets, while longer tuning leads to overfitting and hallucination, even when the encoder stays frozen. The study indicates that current MLLM fine-tuning practices need better safeguards against forgetting.

## Key points
- **Theoretical framing:** The authors explain forgetting through minority collapse, a concept from Neural Collapse theory. Classes missing from fine-tuning behave like minority classes with zero samples, so their classifiers converge to a single point and become indistinguishable.
- **ResNet18 sanity check:** ResNet18 was pre-trained on half the classes of MNIST, CIFAR-10, CIFAR-100 and miniImagenet, then fine-tuned on the other half, for 100 epochs each. Training accuracy on the original classes fell to roughly zero once fine-tuning began.
- **CLIP experiment:** Fine-tuning the CLIP ViT-L-14 vision encoder (an OpenAI checkpoint loaded through openCLIP) for about 15 epochs raised in-domain accuracy close to 99%. Accuracy on the other datasets declined over the same period.
- **LLaVA-7b experiments:**
  - Tuning only the linear projection adapter for one epoch on an RGB dataset (CIFAR-10, CIFAR-100 or miniImagenet) improved results on the other datasets. In epochs 2–3 the model began to hallucinate.
  - Jointly tuning the LLM and adapter with LoRA hurt non-fine-tuned datasets after a single epoch.
  - Fine-tuning on MNIST harmed every other dataset, which the authors take as evidence that fine-tuning data diversity matters.
- **Proposed extensions:** EMT's answer-checking step relies on the OpenAI API. Alternatives the authors suggest include matching answers via CLIP text embeddings or using hard-coded label matching. Label synonyms remain an open problem, for example whether "dog" should count as correct for African_hunting_dog.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of California, Berkeley](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) — University, Berkeley, United States · 175 works · 227,440 citations
- [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) — University, New York, United States · 201 works · 100,180 citations
- [University of Michigan](https://md.research.elamur.ai/free/en/universities/university-of-michigan/) — University, Ann Arbor, United States · 70 works · 13,693 citations
- [University of Wisconsin–Madison](https://md.research.elamur.ai/free/en/universities/university-of-wisconsin-madison/) — University, Madison, United States · 42 works · 48,486 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 657 works

## Cite

Cite as: Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee and Yi Ma. 2023. Investigating the Catastrophic Forgetting in Multimodal Large Language Models. arXiv:2309.10313.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhai2023investigating,
  title = {{Investigating the Catastrophic Forgetting in Multimodal Large Language Models}},
  author = {Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee and Yi Ma},
  year = {2023},
  doi = {10.48550/arxiv.2309.10313},
  eprint = {2309.10313},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.10313}
}
```

- [← previous EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models](https://md.research.elamur.ai/free/en/papers/paech-2023-eq-bench-emotional-intelligence-benchmark/)
- [next → Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla](https://md.research.elamur.ai/free/en/papers/lieberum-2023-circuit-analysis-interpretability-scale-evidence/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/)

## References

61 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805, 2018. 1, 2
2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901, 2020. 2
3. OpenAI. Chatgpt, 2022. URL [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt). [↗](https://openai.com/blog/chatgpt)
4. R OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv, pages 2303–08774, 2023. 1
5. OpenAI. Gpt-4, 2023. URL [https://openai.com/research/gpt-4](https://openai.com/research/gpt-4). 1 [↗](https://openai.com/research/gpt-4)
6. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. [BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). In ICML, 2023. 1, 3
7. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). arXiv preprint arXiv:2304.08485, 2023. 1, 2, 3, 5, 6, 16, 18, 19
8. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. [Minigpt-4: Enhancing vision-language understanding with advanced large language models](https://md.research.elamur.ai/free/en/papers/zhu-2023-minigpt-4-enhancing-vision-language/). arXiv preprint arXiv:2304.10592, 2023. 2, 9
9. Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. [Otter: A multi-modal model with in-context instruction tuning](https://md.research.elamur.ai/free/en/papers/li-2023-otter-multi-modal-model-context/). arXiv preprint arXiv:2305.03726, 2023. 1, 2, 5, 16, 19
10. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. [Instructblip: Towards general-purpose vision-language models with instruction tuning](https://md.research.elamur.ai/free/en/papers/dai-2023-instructblip-towards-general-purpose-vision/), 2023. 1, 2, 3, 5, 16, 19
11. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In International conference on machine learning, pages 8748–8763. PMLR, 20…
12. Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL https://doi.org/10.5281/ zenodo.5143773. 1, 4, 18, 19 [find](https://scholar.google.com/scholar?q=Gabriel+Ilharco%2C+Mitchell+Wortsman%2C+Ross+Wightman%2C+Cade+Gordon%2C+Nicholas+Carlini%2C+Rohan+Taori%2C+Achal+Dave%2C+Vaishaal+Shankar%2C+Hongseok+Namkoong%2C+John+Miller%2C+Hannaneh+Hajishirzi%2C+Ali+Farhadi%2C+and+Ludwi)
13. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. 1, 6 [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Eric+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+et+al%2E+Scaling+instruction%2Dfinetuned+language+models%2E+arXiv+prep)
14. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023. 3, 6
15. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023. 1, 3
16. Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. [A survey on multimodal large language models](https://md.research.elamur.ai/free/en/papers/yin-2023-survey-multimodal-large-language-models/). arXiv preprint arXiv:2306.13549, 2023. 1, 3
17. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. [Mme: A comprehensive evaluation benchmark for multimodal large language models](https://md.research.elamur.ai/free/en/papers/fu-2023-mme-comprehensive-evaluation-benchmark-multimodal/). arXiv preprint arXiv:2306.13394, 2023. 1, 3, 4
18. Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7870– 7881, Online, November 202… [find](https://scholar.google.com/scholar?q=Sanyuan+Chen%2C+Yutai+Hou%2C+Yiming+Cui%2C+Wanxiang+Che%2C+Ting+Liu%2C+and+Xiangzhan+Yu%2E+Recall+and+learn%3A+Fine%2Dtuning+deep+pretrained+language+models+with+less+forgetting%2E+In+Proceedings+of+the+2020+Conference)
19. Xinshuai Dong, Anh Tuan Luu, Min Lin, Shuicheng Yan, and Hanwang Zhang. How should pre-trained language models be fine-tuned towards adversarial robustness? Advances in Neural Information Processing Systems, 34:4356–4369, 2021. 3 [find](https://scholar.google.com/scholar?q=Xinshuai+Dong%2C+Anh+Tuan+Luu%2C+Min+Lin%2C+Shuicheng+Yan%2C+and+Hanwang+Zhang%2E+How+should+pre%2Dtrained+language+models+be+fine%2Dtuned+towards+adversarial+robustness%3F+Advances+in+Neural+Information+Processing+S)
20. Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. On the stability of fine-tuning {bert}: Misconceptions, explanations, and strong baselines. In International Conference on Learning Representations, 2021. URL [https://openreview.net/forum?id=nzpLWnVAyah](https://openreview.net/forum?id=nzpLWnVAyah). [↗](https://openreview.net/forum?id=nzpLWnVAyah)
21. Tomasz Korbak, Hady Elsahar, German Kruszewski, and Marc Dymetman. Controlling conditional language models without catastrophic forgetting. In International Conference on Machine Learning, pages 11499–11528. PMLR, 2022. 1, 3 [find](https://scholar.google.com/scholar?q=Tomasz+Korbak%2C+Hady+Elsahar%2C+German+Kruszewski%2C+and+Marc+Dymetman%2E+Controlling+conditional+language+models+without+catastrophic+forgetting%2E+In+International+Conference+on+Machine+Learning%2C+pages+11499)
22. Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211, 2013. 1 [find](https://scholar.google.com/scholar?q=Ian+J+Goodfellow%2C+Mehdi+Mirza%2C+Da+Xiao%2C+Aaron+Courville%2C+and+Yoshua+Bengio%2E+An+empirical+investigation+of+catastrophic+forgetting+in+gradient%2Dbased+neural+networks%2E+arXiv+preprint+arXiv%3A1312%2E6211%2C+201)
23. Yibo Yang, Haobo Yuan, Xiangtai Li, Zhouchen Lin, Philip Torr, and Dacheng Tao. Neural collapse inspired feature-classifier alignment for few-shot class-incremental learning. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview. net/forum?id=y5W8tpojhtJ.… [find](https://scholar.google.com/scholar?q=Yibo+Yang%2C+Haobo+Yuan%2C+Xiangtai+Li%2C+Zhouchen+Lin%2C+Philip+Torr%2C+and+Dacheng+Tao%2E+Neural+collapse+inspired+feature%2Dclassifier+alignment+for+few%2Dshot+class%2Dincremental+learning%2E+In+The+Eleventh+Internati)
24. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015. 1 [find](https://scholar.google.com/scholar?q=Stanislaw+Antol%2C+Aishwarya+Agrawal%2C+Jiasen+Lu%2C+Margaret+Mitchell%2C+Dhruv+Batra%2C+C+Lawrence+Zitnick%2C+and+Devi+Parikh%2E+Vqa%3A+Visual+question+answering%2E+In+Proceedings+of+the+IEEE+international+conference+)
25. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. [Evaluating object hallucination in large vision-language models](https://md.research.elamur.ai/free/en/papers/li-2023-evaluating-object-hallucination-large-vision/). arXiv preprint arXiv:2305.10355, 2023. 1, 2, 3, 4, 9
26. William Berrios, Gautam Mittal, Tristan Thrush, Douwe Kiela, and Amanpreet Singh. Towards language models that can see: Computer vision through the lens of natural language. arXiv preprint arXiv:2306.16410, 2023. 2, 3, 5, 19 — [arXiv:2306.16410](https://arxiv.org/abs/2306.16410) [find](https://scholar.google.com/scholar?q=William+Berrios%2C+Gautam+Mittal%2C+Tristan+Thrush%2C+Douwe+Kiela%2C+and+Amanpreet+Singh%2E+Towards+language+models+that+can+see%3A+Computer+vision+through+the+lens+of+natural+language%2E+arXiv+preprint+arXiv%3A2306%2E)
27. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. [Survey of hallucination in natural language generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/). ACM Computing Surveys, 55(12):1–38, 2023. 2, 5
28. Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. [The false promise of imitating proprietary llms](https://md.research.elamur.ai/free/en/papers/gudibande-2023-false-promise-imitating-proprietary-llms/). arXiv preprint arXiv:2305.15717, 2023. 2, 4
29. Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35: 2507–2521, 2022. 2, 3, 18 [find](https://scholar.google.com/scholar?q=Pan+Lu%2C+Swaroop+Mishra%2C+Tanglin+Xia%2C+Liang+Qiu%2C+Kai%2DWei+Chang%2C+Song%2DChun+Zhu%2C+Oyvind+Tafjord%2C+Peter+Clark%2C+and+Ashwin+Kalyan%2E+Learn+to+explain%3A+Multimodal+reasoning+via+thought+chains+for+science+ques)
30. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018. 2 [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+Ilya+Sutskever%2C+et+al%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+2018%2E+2)
31. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2C+2019%2E)
32. Yong Lin, Lu Tan, Hangyu Lin, Zeming Zheng, Renjie Pi, Jipeng Zhang, Shizhe Diao, Haoxiang Wang, Han Zhao, Yuan Yao, et al. Speciality vs generality: An empirical study on catastrophic forgetting in fine-tuning foundation models. arXiv preprint arXiv:2309.06256, 2023. 2 — [arXiv:2309.06256](https://arxiv.org/abs/2309.06256) [find](https://scholar.google.com/scholar?q=Yong+Lin%2C+Lu+Tan%2C+Hangyu+Lin%2C+Zeming+Zheng%2C+Renjie+Pi%2C+Jipeng+Zhang%2C+Shizhe+Diao%2C+Haoxiang+Wang%2C+Han+Zhao%2C+Yuan+Yao%2C+et+al%2E+Speciality+vs+generality%3A+An+empirical+study+on+catastrophic+forgetting+in+f)
33. Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier, 1989. 3 [find](https://scholar.google.com/scholar?q=Michael+McCloskey+and+Neal+J+Cohen%2E+Catastrophic+interference+in+connectionist+networks%3A+The+sequential+learning+problem%2E+In+Psychology+of+learning+and+motivation%2C+volume+24%2C+pages+109%E2%80%93165%2E+Elsevier%2C+)
34. Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146, 2018. 3 [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+arXiv+preprint+arXiv%3A1801%2E06146%2C+2018%2E+3)
35. Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. Mixout: Effective regularization to finetune large-scale pretrained language models. In International Conference on Learning Representations, 2020. URL [https://openreview.net/forum?id=HkgaETNtDB](https://openreview.net/forum?id=HkgaETNtDB). 3 [↗](https://openreview.net/forum?id=HkgaETNtDB)
36. Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. Revisiting few-sample {bert} fine-tuning. In International Conference on Learning Representations, 2021. URL — [Revisiting Few-sample BERT Fine-tuning](https://md.research.elamur.ai/free/en/papers/zhang-2020-revisiting-few-sample-bert-fine/) [https://openreview.net/forum?id=cO1IH43yUF](https://openreview.net/forum?id=cO1IH43yUF). 3 [↗](https://openreview.net/forum?id=cO1IH43yUF)
37. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. [Flamingo: a visual language model for few-shot learning](https://md.research.elamur.ai/free/en/papers/alayrac-2022-flamingo-visual-language-model-few/). Advances in Neural Information Processing Systems, 35:23716– 23736, 2022. 3
38. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023. 3 [find](https://scholar.google.com/scholar?q=Guanzhi+Wang%2C+Yuqi+Xie%2C+Yunfan+Jiang%2C+Ajay+Mandlekar%2C+Chaowei+Xiao%2C+Yuke+Zhu%2C+Linxi+Fan%2C+and+Anima+Anandkumar%2E+Voyager%3A+An+open%2Dended+embodied+agent+with+large+language+models%2E+arXiv+preprint+arXiv%3A23)
39. Anas Awadalla, Irena Gao, Joshua Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo, March 2023. URL https://doi.org/10.5281/zenodo.7733589.… — [doi:10.5281/zenodo.7733589](https://doi.org/10.5281/zenodo.7733589) [↗](https://doi.org/10.5281/zenodo.7733589.…)
40. Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. [Llama-adapter: Efficient fine-tuning of language models with zero-init attention](https://md.research.elamur.ai/free/en/papers/zhang-2023-llama-adapter-efficient-fine-tuning/). arXiv preprint arXiv:2303.16199, 2023.
41. Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. [Language is not all you need: Aligning perception with language models](https://md.research.elamur.ai/free/en/papers/huang-2023-language-all-need-aligning-perception/). arXiv preprint arXiv:2302.14045, 2023.
42. Mu Cai, Zeyi Huang, Yuheng Li, Haohan Wang, and Yong Jae Lee. Leveraging large language models for scalable vector graphics-driven image understanding. arXiv preprint arXiv:2306.06094, 2023. 3 — [arXiv:2306.06094](https://arxiv.org/abs/2306.06094) [find](https://scholar.google.com/scholar?q=Mu+Cai%2C+Zeyi+Huang%2C+Yuheng+Li%2C+Haohan+Wang%2C+and+Yong+Jae+Lee%2E+Leveraging+large+language+models+for+scalable+vector+graphics%2Ddriven+image+understanding%2E+arXiv+preprint+arXiv%3A2306%2E06094%2C+2023%2E+3+%E2%80%94+%5BarXi)
43. Hang Zhang, Xin Li, and Lidong Bing. [Video-llama: An instruction-tuned audio-visual language model for video understanding](https://md.research.elamur.ai/free/en/papers/zhang-2023-video-llama-instruction-tuned-audio/). arXiv preprint arXiv:2306.02858, 2023. 3
44. Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. arXiv preprint arXiv:2307.12981, 2023. 3 — [arXiv:2307.12981](https://arxiv.org/abs/2307.12981) [find](https://scholar.google.com/scholar?q=Yining+Hong%2C+Haoyu+Zhen%2C+Peihao+Chen%2C+Shuhong+Zheng%2C+Yilun+Du%2C+Zhenfang+Chen%2C+and+Chuang+Gan%2E+3d%2Dllm%3A+Injecting+the+3d+world+into+large+language+models%2E+arXiv+preprint+arXiv%3A2307%2E12981%2C+2023%2E+3+%E2%80%94+%5BarX)
45. Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652–24663, 2020. 3, 4, 14 [find](https://scholar.google.com/scholar?q=Vardan+Papyan%2C+XY+Han%2C+and+David+L+Donoho%2E+Prevalence+of+neural+collapse+during+the+terminal+phase+of+deep+learning+training%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+117%2840%29%3A+24652%E2%80%9324663%2C+202)
46. Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34:29820–29834, 2021. 3, 4, 15 [find](https://scholar.google.com/scholar?q=Zhihui+Zhu%2C+Tianyu+Ding%2C+Jinxin+Zhou%2C+Xiao+Li%2C+Chong+You%2C+Jeremias+Sulam%2C+and+Qing+Qu%2E+A+geometric+analysis+of+neural+collapse+with+unconstrained+features%2E+Advances+in+Neural+Information+Processing+Sy)
47. Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences, 118(43):e2103091118, 2021. 3, 4, 14, 15 [find](https://scholar.google.com/scholar?q=Cong+Fang%2C+Hangfeng+He%2C+Qi+Long%2C+and+Weijie+J+Su%2E+Exploring+deep+neural+networks+via+layer%2Dpeeled+model%3A+Minority+collapse+in+imbalanced+training%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+118%28)
48. Christos Thrampoulidis, Ganesh Ramachandra Kini, Vala Vakilian, and Tina Behnia. Imbalance trouble: Revisiting neural-collapse geometry. Advances in Neural Information Processing Systems, 35:27225–27238, 2022. 3, 4, 14 [find](https://scholar.google.com/scholar?q=Christos+Thrampoulidis%2C+Ganesh+Ramachandra+Kini%2C+Vala+Vakilian%2C+and+Tina+Behnia%2E+Imbalance+trouble%3A+Revisiting+neural%2Dcollapse+geometry%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A27225%E2%80%93272)
49. Tina Behnia, Ganesh Ramachandra Kini, Vala Vakilian, and Christos Thrampoulidis. On the implicit geometry of cross-entropy parameterizations for label-imbalanced data. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop), 2022. URL [https://openreview.net/forum?id=1piyfD_ictW](https://openreview.net/forum?id=1piyfD_ictW). [↗](https://openreview.net/forum?id=1piyfD_ictW)
50. Zhisheng Zhong, Jiequan Cui, Yibo Yang, Xiaoyang Wu, Xiaojuan Qi, Xiangyu Zhang, and Jiaya Jia. Understanding imbalanced semantic segmentation through neural collapse. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19550–19560, 2023. 3, 14 [find](https://scholar.google.com/scholar?q=Zhisheng+Zhong%2C+Jiequan+Cui%2C+Yibo+Yang%2C+Xiaoyang+Wu%2C+Xiaojuan+Qi%2C+Xiangyu+Zhang%2C+and+Jiaya+Jia%2E+Understanding+imbalanced+semantic+segmentation+through+neural+collapse%2E+In+Proceedings+of+the+IEEE%2FCVF+C)
51. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 3, 15, 16
52. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). arXiv preprint arXiv:2305.18290, 2023. 4
53. Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306.[14565, 2023](https://md.research.elamur.ai/free/en/papers/liu-2023-mitigating-hallucination-large-multi-modal/). 5
54. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). arXiv preprint arXiv:2106.09685, 2021. 6, 7
55. Yibo Yang, Shixiang Chen, Xiangtai Li, Liang Xie, Zhouchen Lin, and Dacheng Tao. Inducing neural collapse in imbalanced learning: Do we really need a learnable classifier at the end of deep neural network? Advances in Neural Information Processing Systems, 35:37991–38002, 2022. 14 [find](https://scholar.google.com/scholar?q=Yibo+Yang%2C+Shixiang+Chen%2C+Xiangtai+Li%2C+Liang+Xie%2C+Zhouchen+Lin%2C+and+Dacheng+Tao%2E+Inducing+neural+collapse+in+imbalanced+learning%3A+Do+we+really+need+a+learnable+classifier+at+the+end+of+deep+neural+net)
56. Liang Xie, Yibo Yang, Deng Cai, and Xiaofei He. Neural collapse inspired attraction–repulsion-balanced loss for imbalanced learning. Neurocomputing, 527:60–70, 2023. 14 [find](https://scholar.google.com/scholar?q=Liang+Xie%2C+Yibo+Yang%2C+Deng+Cai%2C+and+Xiaofei+He%2E+Neural+collapse+inspired+attraction%E2%80%93repulsion%2Dbalanced+loss+for+imbalanced+learning%2E+Neurocomputing%2C+527%3A60%E2%80%9370%2C+2023%2E+14)
57. Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. arXiv preprint arXiv:2011.11619, 2020. 14 — [arXiv:2011.11619](https://arxiv.org/abs/2011.11619) [find](https://scholar.google.com/scholar?q=Dustin+G+Mixon%2C+Hans+Parshall%2C+and+Jianzong+Pi%2E+Neural+collapse+with+unconstrained+features%2E+arXiv+preprint+arXiv%3A2011%2E11619%2C+2020%2E+14+%E2%80%94+%5BarXiv%3A2011%2E11619%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2011%2E11619%29)
58. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. [A simple framework for contrastive learning of visual representations](https://md.research.elamur.ai/free/en/papers/chen-2020-simple-framework-contrastive-learning-visual/). In International conference on machine learning, pages 1597–1607. PMLR, 2020. 15
59. Diederik P Kingma and Jimmy Ba. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980, 2014. 17
60. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. [Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning](https://md.research.elamur.ai/free/en/papers/sharma-2018-conceptual-captions-cleaned-hypernymed-image/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, 2018.…
61. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. [Laion-5b: An open large-scale dataset for training next generation image-text models](https://md.research.elamur.ai/free/en/papers/schuhmann-2022-laion-5b-open-large-scale/). Advances in Neural Information Processing Systems…

*Updated 29 October 2026*