--- title: "Retrieval-Augmented Multimodal Language Modeling (2022)" description: "Large multimodal generators such as DALL-E and CM3 keep all of their world… 0.90 accuracy. 1 reference, cited by 76 papers here." authors: ["Michihiro Yasunaga", "Armen Aghajanyan", "Weijia Shi", "Rich James", "Jure Leskovec", "Percy Liang", "Mike Lewis", "Luke Zettlemoyer", "Wen-tau Yih"] year: 2022 venue: "arXiv:2211.12561" source: "https://arxiv.org/abs/2211.12561" doi: "10.48550/arxiv.2211.12561" arxiv: "2211.12561" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/yasunaga-2022-retrieval-augmented-multimodal-language-modeling/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yasunaga-2022-retrieval-augmented-multimodal-language-modeling/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / Retrieval-Augmented Multimodal Language Modeling # Retrieval-Augmented Multimodal Language Modeling **Authors:** [Michihiro Yasunaga](https://md.research.elamur.ai/free/en/authors/michihiro-yasunaga/), [Armen Aghajanyan](https://md.research.elamur.ai/free/en/authors/armen-aghajanyan/), [Weijia Shi](https://md.research.elamur.ai/free/en/authors/weijia-shi/), [Rich James](https://md.research.elamur.ai/free/en/authors/rich-james/), [Jure Leskovec](https://md.research.elamur.ai/free/en/authors/jure-leskovec/), [Percy Liang](https://md.research.elamur.ai/free/en/authors/percy-liang/), [Mike Lewis](https://md.research.elamur.ai/free/en/authors/mike-lewis/), [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/) and [Wen-tau Yih](https://md.research.elamur.ai/free/en/authors/wen-tau-yih/) · **Year:** 2022 · **Venue:** arXiv:2211.12561 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2211.12561) · [PDF](https://arxiv.org/pdf/2211.12561) · [DOI 10.48550/arxiv.2211.12561](https://doi.org/10.48550/arxiv.2211.12561) · [arXiv 2211.12561](https://arxiv.org/abs/2211.12561) · [OpenAlex](https://openalex.org/W4309953147) **Citations:** 28 citations (OpenAlex, 2026-10-14) · [76](https://md.research.elamur.ai/free/en/papers/yasunaga-2022-retrieval-augmented-multimodal-language-modeling/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Large multimodal generators such as DALL-E and CM3 keep all of their world knowledge inside their weights. Covering more entities therefore means ever bigger models and datasets, and the knowledge is hard to update. The authors carry retrieval augmentation, already successful in text-only NLP, over to mixed image–text data. Their system, RA-CM3, has two parts. The retriever is a dense retriever built on frozen CLIP encoders: each document's text and image are embedded separately, and the two vectors are averaged. The generator is a CM3 Transformer that receives the retrieved documents prepended to its input. It is trained on LAION with a token-prediction loss over both the main document and the retrieved ones. RA-CM3 is presented as the first model that can both retrieve and generate text and images. On MS-COCO it clearly beats a CM3 baseline without retrieval and also outperforms DALL-E and Flamingo, while using far fewer resources. The work matters because external memory makes entity-heavy generation more faithful. It also gives the model multimodal in-context learning, including control of image generation through demonstration images. ## Key points - **Training and memory data:** 150M image–text pairs from LAION. Images are converted into 1024 tokens with VQGAN, and documents are formatted as HTML-like sequences in the CM3 style. - **Retriever design:** Dense Retrieval (bi-encoder) using frozen CLIP text and image encoders. The two embeddings are mean-pooled and L2-normalized, and candidates are found with MIPS. The method also selects retrieved documents for diversity. - **MS-COCO results:** RA-CM3 improves on vanilla CM3 by about 12 FID points in text-to-image generation and about 17 CIDEr points in captioning. It outperforms DALL-E and Flamingo with under 30% of their parameters and training compute. - **New capabilities:** - higher-fidelity images of uncommon entities, such as a Ming Dynasty vase or the Oriental Pearl tower, and of unusual combinations (e.g., French flag on the moon); - knowledge-aware image infilling and editing; - style control by placing example images in the context. - **Few-shot classification:** On ImageNet-based binary tasks with non-semantic labels ("animal X/Y"), RA-CM3 reaches 0.78, 0.79, 0.86 and 0.90 accuracy at k = 1, 2, 4 and 8 shots. Baseline CM3 stays at roughly 0.50–0.56. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 506 works · 330,053 citations - [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 402 works · 212,763 citations - [Meta](https://md.research.elamur.ai/free/en/companies/meta/) — Company, Menlo Park, United States · 214 works · 241,492 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works 🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 811 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 552 works ## Cite Cite as: Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer and Wen-tau Yih. 2022. Retrieval-Augmented Multimodal Language Modeling. arXiv:2211.12561. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{yasunaga2022retrieval, title = {{Retrieval-Augmented Multimodal Language Modeling}}, author = {Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer and Wen-tau Yih}, year = {2022}, doi = {10.48550/arxiv.2211.12561}, eprint = {2211.12561}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2211.12561} } ``` - [← previous NTREX-128 – News Test References for MT Evaluation of 128 Languages](https://md.research.elamur.ai/free/en/papers/federmann-2022-ntrex-128-news-test-references/) - [next → Text-Free Prosody-Aware Generative Spoken Language Modeling](https://md.research.elamur.ai/free/en/papers/kharitonov-2022-text-free-prosody-aware-generative/) - [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work. 1. procedure k times, each using a different pair of demonstration examples, and take the average ensemble of the predicted probability (“X” and “Y”) across the k passes. 3 Agarwal, O., Ge, H., Shakeri, S., and Al-Rfou, R. Knowledge graph based synthetic corpus generation for knowledge-enhanced languag… — [Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training](https://md.research.elamur.ai/free/en/papers/agarwal-2021-knowledge-graph-based-synthetic-corpus/) *Updated 2 November 2026*