--- title: "Otter: A Multi-Modal Model with In-Context — Li 2023" description: "Most large multimodal models (LMMs) are tuned to answer one instruction about one image, or at most to draw on earlier dialogue turns." authors: ["Bo Li", "Yuanhan Zhang", "Liangyu Chen", "Jinghao Wang", "Fanyi Pu", "Joshua Adrian Cahyono", "Jingkang Yang", "Ziwei Liu"] year: 2023 venue: "arXiv:2305.03726" source: "https://arxiv.org/abs/2305.03726" doi: "10.48550/arxiv.2305.03726" arxiv: "2305.03726" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/li-2023-otter-multi-modal-model-context/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2023-otter-multi-modal-model-context/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / Otter: A Multi-Modal Model with In-Context Instruction Tuning # Otter: A Multi-Modal Model with In-Context Instruction Tuning **Authors:** [Bo Li](https://md.research.elamur.ai/free/en/authors/bo-li/), [Yuanhan Zhang](https://md.research.elamur.ai/free/en/authors/yuanhan-zhang/), Liangyu Chen, Jinghao Wang, Fanyi Pu, Joshua Adrian Cahyono, Jingkang Yang, [Ziwei Liu](https://md.research.elamur.ai/free/en/authors/ziwei-liu/) · **Year:** 2023 · **Venue:** arXiv:2305.03726 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2305.03726) · [PDF](https://arxiv.org/pdf/2305.03726) · [DOI 10.48550/arxiv.2305.03726](https://doi.org/10.48550/arxiv.2305.03726) · [arXiv 2305.03726](https://arxiv.org/abs/2305.03726) · [OpenAlex](https://openalex.org/W4410536678) **Citations:** 158 citations (openalex-snapshot · 2026-10-14) · [121](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Most large multimodal models (LMMs) are tuned to answer one instruction about one image, or at most to draw on earlier dialogue turns. Few use paired images and text as in-context demonstrations. Otter addresses this gap. It is built on the Flamingo design, via OpenFlamingo, with its Perceiver resampler. The model is instruction-tuned so that each query arrives with related multimodal examples, a format that mirrors the interleaved image-text data Flamingo saw during pretraining. To support this, the authors assemble MIMIC-IT, a resource of more than three million instruction–response pairs (roughly 2.2 million distinct) covering both images and videos. Every entry is linked to curated contextual examples. Otter accepts text, several images or video as input. The authors report that training with in-context examples speeds up convergence and improves generalization. They also find that MIMIC-IT's broad coverage helps Otter reason over video and multiple images. The work matters because it provides a large instruction dataset and shows that matching the tuning format to the pretraining format pays off. ## Key points - Otter extends Flamingo/OpenFlamingo. A Perceiver resampler converts images into visual tokens, and the model is trained with token-level supervised fine-tuning on (image, instruction, response) triples. - MIMIC-IT (MultI-Modal In-Context Instruction Tuning) contains over 3M instruction–response pairs with about 2.2M unique instructions. Each instance has attached multimodal context. - Context types include retrieved image-text examples, task descriptions, multi-turn conversations and related video clips. Sources and subsets include LA-I2I, LA-T2T, Dense Caption, TV Shows, Ego4D, VST, Scene Navigation and Spot Difference. - The design aligns with Flamingo's upstream pretraining on interleaved corpora such as MMC4 and M3W. - The authors are from S-Lab, Nanyang Technological University, and Microsoft Research (arXiv:2305.03726). *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) ## Cited by Cited by 121 papers of our library cite it (of the 140673 whose bibliographies we read); 21 of them have a page here, 100 not yet. - [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-vl-versatile-vision-language/) - [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://md.research.elamur.ai/free/en/papers/chen-2023-internvl-scaling-up-vision/) - [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://md.research.elamur.ai/free/en/papers/chen-2023-sharegpt4v-improving-large-multi-modal/) - [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://md.research.elamur.ai/free/en/papers/chen-2023-shikra-unleashing-multimodal-llm-s/) - [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://md.research.elamur.ai/free/en/papers/chen-2024-far-gpt-4v-closing/) - [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://md.research.elamur.ai/free/en/papers/cheng-2024-seeclick-harnessing-gui-grounding-advanced/) - [Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models](https://md.research.elamur.ai/free/en/papers/deitke-2024-molmo-pixmo-open-weights-open/) - [A Survey on In-context Learning](https://md.research.elamur.ai/free/en/papers/dong-2023-survey-context-learning/) - [InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model](https://md.research.elamur.ai/free/en/papers/dong-2024-internlm-xcomposer2-mastering-free/) - [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://md.research.elamur.ai/free/en/papers/fu-2023-mme-comprehensive-evaluation-benchmark-multimodal/) - [Evaluating Object Hallucination in Large Vision-Language Models](https://md.research.elamur.ai/free/en/papers/li-2023-evaluating-object-hallucination-large-vision/) - [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://md.research.elamur.ai/free/en/papers/li-2023-mvbench-comprehensive-multi-modal-video/) - [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://md.research.elamur.ai/free/en/papers/li-2023-seed-bench-benchmarking-multimodal-llms/) - [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://md.research.elamur.ai/free/en/papers/lin-2023-video-llava-learning-united-visual/) - [Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-mitigating-hallucination-large-multi-modal/) - [Aligning Large Multimodal Models with Factually Augmented RLHF](https://md.research.elamur.ai/free/en/papers/sun-2023-aligning-large-multimodal-models-factually/) - [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://md.research.elamur.ai/free/en/papers/yu-2023-mm-vet-evaluating-large-multimodal/) - [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/) - [Instruction Tuning for Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/zhang-2023-instruction-tuning-large-language-models/) - [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://md.research.elamur.ai/free/en/papers/zhang-2023-llama-adapter-efficient-fine-tuning/) - [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://md.research.elamur.ai/free/en/papers/zhang-2023-video-llama-instruction-tuned-audio/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{li2023otter, title = {{Otter: A Multi-Modal Model with In-Context Instruction Tuning}}, author = {Li, Bo and Zhang, Yuanhan and Chen, Liangyu and Wang, Jinghao and Pu, Fanyi and Cahyono, Joshua Adrian and Yang, Jingkang and Liu, Ziwei}, year = {2023}, doi = {10.48550/arxiv.2305.03726}, eprint = {2305.03726}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2305.03726} } ``` - [← previous OSN-MDAD: Machine Translation Dataset for Arabic Multi-Dialectal Conversations on Online Social Media](https://md.research.elamur.ai/free/en/papers/alzamzami-2023-osn-mdad-machine-translation-dataset/) - [next → Overview of PragTag-2023: Low-Resource Multi-Domain Pragmatic Tagging of Peer Reviews](https://md.research.elamur.ai/free/en/papers/dycke-2023-overview-pragtag-2023-low-resource/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Common crawl. [https://commoncrawl.org/](https://commoncrawl.org/). Accessed: 2023-11-11. 3 Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.15661, 2022. 2 Jean-Baptiste Alayrac, Jeff Donahue, Pa… [↗](https://commoncrawl.org/) [our card](https://md.research.elamur.ai/free/en/papers/akyurek-2022-learning-algorithm-context-learning-investigations/)