--- title: "Reliable, Adaptable, and Attributable Language Models with" description: "The proposed roadmap calls for rethinking relevance and datastore content, designing more integrated… 2 references, cited by 39 papers here." authors: ["Akari Asai", "Zexuan Zhong", "Danqi Chen", "Pang Wei Koh", "Luke Zettlemoyer", "Hannaneh Hajishirzi", "Wen-tau Yih"] year: 2024 venue: "arXiv:2403.03187" source: "https://arxiv.org/abs/2403.03187" doi: "10.48550/arxiv.2403.03187" arxiv: "2403.03187" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/asai-2024-reliable-adaptable-attributable-language-models/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/asai-2024-reliable-adaptable-attributable-language-models/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / Reliable, Adaptable, and Attributable Language Models with Retrieval # Reliable, Adaptable, and Attributable Language Models with Retrieval **Authors:** [Akari Asai](https://md.research.elamur.ai/free/en/authors/akari-asai/), [Zexuan Zhong](https://md.research.elamur.ai/free/en/authors/zexuan-zhong/), [Danqi Chen](https://md.research.elamur.ai/free/en/authors/danqi-chen/), [Pang Wei Koh](https://md.research.elamur.ai/free/en/authors/pang-wei-koh/), [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/), [Hannaneh Hajishirzi](https://md.research.elamur.ai/free/en/authors/hannaneh-hajishirzi/) and [Wen-tau Yih](https://md.research.elamur.ai/free/en/authors/wen-tau-yih/) · **Year:** 2024 · **Venue:** arXiv:2403.03187 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2403.03187) · [PDF](https://arxiv.org/pdf/2403.03187) · [DOI 10.48550/arxiv.2403.03187](https://doi.org/10.48550/arxiv.2403.03187) · [arXiv 2403.03187](https://arxiv.org/abs/2403.03187) · [OpenAlex](https://openalex.org/W4392538944) **Citations:** 6 citations (OpenAlex, 2026-10-14) · [39](https://md.research.elamur.ai/free/en/papers/asai-2024-reliable-adaptable-attributable-language-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary This position paper by Asai et al. contends that retrieval-augmented LMs should succeed purely parametric LMs as the main design for future language models. It contains no new experiments, datasets, or models. Instead, it surveys prior work and lists five weaknesses of parametric models: hallucinated facts, outputs that are hard to verify, trouble removing private or copyrighted training data, costly adaptation to new domains or time periods, and very large model sizes. The authors argue that querying an external datastore at inference time reduces these problems, because knowledge can be cited, updated, or removed without retraining. They also explain why adoption remains limited. Current systems work mainly on knowledge-heavy tasks such as QA. Simply prepending retrieved passages to an off-the-shelf LM couples the two components only loosely. Open infrastructure for scaling is also scarce. The proposed roadmap calls for rethinking relevance and datastore content, designing more integrated architectures and training methods, and building scalable open-source systems. ## Key points - The paper names five parametric-LM weaknesses (W1–W5): factual errors, weak verifiability, hard data opt-out, expensive adaptation, and huge parameter counts. It cites evidence that scaling alone barely helps with long-tail knowledge. - It identifies three barriers (C1–C3): - Retrieval that relies narrowly on semantic or lexical similarity. - Shallow retriever–LM interaction in prompt-concatenation RAG. - Insufficient training and inference infrastructure. - It traces the field's history from DrQA through jointly trained systems (REALM, RAG, RETRO) to today's training-free RAG built on top of models like GPT-3. - The roadmap includes: - Going past input augmentation to designs that blend retrieved information into the output distribution or merge it within the model's internal layers. - Training the retriever and LM jointly. - Adding retrieval during pre-training. - Growing datastores toward trillions of tokens. - For infrastructure, it highlights FAISS, embedding compression and binary indexes, and possibly specialized CPU-heavy hardware. It notes that LangChain, LlamaIndex, and DSPy exist but argues the field still lacks a standard pipeline and benchmark beyond basic RAG. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works 🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 473 works ## Cite Cite as: Akari Asai, Zexuan Zhong, Danqi Chen, Pang Wei Koh, Luke Zettlemoyer, Hannaneh Hajishirzi and Wen-tau Yih. 2024. Reliable, Adaptable, and Attributable Language Models with Retrieval. arXiv:2403.03187. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{asai2024reliable, title = {{Reliable, Adaptable, and Attributable Language Models with Retrieval}}, author = {Akari Asai, Zexuan Zhong, Danqi Chen, Pang Wei Koh, Luke Zettlemoyer, Hannaneh Hajishirzi and Wen-tau Yih}, year = {2024}, doi = {10.48550/arxiv.2403.03187}, eprint = {2403.03187}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2403.03187} } ``` - [← previous GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinements](https://md.research.elamur.ai/free/en/papers/havrilla-2024-glore-where-improve-llm-reasoning/) - [next → Augmenting Math Word Problems via Iterative Question Composing](https://md.research.elamur.ai/free/en/papers/liu-2024-augmenting-math-word-problems-iterative/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) ## References 2 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Asai, A. and Choi, E. Challenges in information-seeking QA: Unanswerable questions and paragraph retrieval. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021. URL https://aclanth… [find](https://scholar.google.com/scholar?q=Asai%2C+A%2E+and+Choi%2C+E%2E+Challenges+in+information%2Dseeking+QA%3A+Unanswerable+questions+and+paragraph+retrieval%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+a) 2. Asai, A., Yu, X., Kasai, J., and Hajishirzi, H. One question answering model for many languages with cross-lingual dense passage retrieval. In Advances in Neural Information Processing Systems, 2021. URL [https://proceedings.neurips.cc/paper_files/paper/2021/file/](https://proceedings.neurips.cc/paper_files/paper/2021/file/) 3df07fdae1ab273a967aaa1d355b8bb6-Pa… [↗](https://proceedings.neurips.cc/paper_files/paper/2021/file/) *Updated 31 October 2026*