---
title: "REALM: Retrieval-Augmented Language Model Pre-Training"
description: "The authors from Google Research propose REALM, which augments a language model with a… 42 references, cited by 1,085 papers here."
authors: ["Kelvin Guu", "Kenton Lee", "Zora Tung", "Panupong Pasupat", "Ming-Wei Chang"]
year: 2020
venue: "arXiv:2002.08909"
source: "https://arxiv.org/abs/2002.08909"
doi: "10.48550/arxiv.2002.08909"
arxiv: "2002.08909"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/guu-2020-realm-retrieval-augmented-language-model/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/guu-2020-realm-retrieval-augmented-language-model/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/) / REALM: Retrieval-Augmented Language Model Pre-Training

# REALM: Retrieval-Augmented Language Model Pre-Training

**Authors:** [Kelvin Guu](https://md.research.elamur.ai/free/en/authors/kelvin-guu/), [Kenton Lee](https://md.research.elamur.ai/free/en/authors/kenton-lee/), Zora Tung, [Panupong Pasupat](https://md.research.elamur.ai/free/en/authors/panupong-pasupat/) and [Ming-Wei Chang](https://md.research.elamur.ai/free/en/authors/ming-wei-chang/) · **Year:** 2020 · **Venue:** arXiv:2002.08909 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2002.08909) · [PDF](https://arxiv.org/pdf/2002.08909) · [DOI 10.48550/arxiv.2002.08909](https://doi.org/10.48550/arxiv.2002.08909) · [arXiv 2002.08909](https://arxiv.org/abs/2002.08909) · [OpenAlex](https://openalex.org/W3007672467)

**Citations:** 495 citations (OpenAlex, 2026-10-14) · [1085](https://md.research.elamur.ai/free/en/papers/guu-2020-realm-retrieval-augmented-language-model/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

Large language models such as BERT or T5 absorb a great deal of world knowledge during pre-training. However, this knowledge is scattered across the network's weights: it is difficult to determine what exactly the model knows, and storing more facts requires making the network larger. The authors from Google Research propose REALM, which augments a language model with a neural document retriever. Before each prediction, the retriever finds relevant passages from a corpus (Wikipedia), and the encoder takes them into account when making its prediction. The main novelty is that the retriever is trained without supervision: the training signal comes from the masked language modeling task. A document is considered useful if it helps reconstruct the masked tokens more accurately. To handle millions of documents, the embeddings are cached and asynchronously recomputed, and the top candidates are found via MIPS. After fine-tuning on Open-QA, the model outperformed all previous systems on three benchmarks by 4–16 points of absolute accuracy. Moreover, it remains interpretable, since one can see which sources the answer relies on, and modular.

## Key points

- **Method:** the model first retrieves a document and then predicts the answer. The document is treated as a latent variable, and the model optimizes the marginal likelihood. The retriever and the encoder are separate BERT-style Transformers, and relevance is computed as the inner product of embeddings.
- **Evaluation:** NaturalQuestions-Open, WebQuestions and CuratedTrec. The comparison covers models with implicit knowledge (T5) and retrieval-based systems (ORQA, which is the main baseline, and others). REALM achieved the best result on all three benchmarks, with a margin of 4–16% absolute accuracy.
- **Ablations:** both the retriever and the encoder benefit from REALM training, but only the combination of the two yields the best result. Masking "salient" spans (salient span masking) proved critically important: it works better than random token masking (BERT) and random span masking (SpanBERT).
- **Infrastructure:** the MIPS index is rebuilt in parallel approximately every 500 training steps. If it is refreshed less often, the index becomes stale and quality degrades.
- **Example:** in a sentence about constructing an equilateral triangle, the model must predict "Fermat". BERT assigns this word a probability of about 1.1×10⁻¹⁴, whereas REALM, after marginalizing over 8 retrieved documents, assigns 0.129. Among future directions, the authors name multilingual retrieval, i.e., using knowledge from a high-resource language for low-resource languages, which is also relevant for Ukrainian.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 430 works · 424,363 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 473 works

💡 Concepts (1): [💡 Retrieval-augmented generation](https://md.research.elamur.ai/free/en/concepts/retrieval-augmented-generation/) · 50 works

## Cite

Cite as: Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat and Ming-Wei Chang. 2020. REALM: Retrieval-Augmented Language Model Pre-Training. arXiv:2002.08909.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{guu2020realm,
  title = {{REALM: Retrieval-Augmented Language Model Pre-Training}},
  author = {Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat and Ming-Wei Chang},
  year = {2020},
  doi = {10.48550/arxiv.2002.08909},
  eprint = {2002.08909},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2002.08909}
}
```

- [← previous Score-Based Generative Modeling through Stochastic Differential Equations](https://md.research.elamur.ai/free/en/papers/song-2020-score-based-generative-modeling-through/)
- [next → Efficient Transformers: A Survey](https://md.research.elamur.ai/free/en/papers/tay-2020-efficient-transformers-survey/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/)

## References

42 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Asai, A., Hashimoto, K., Hajishirzi, H., Socher, R., and Xiong, C. [Learning to retrieve reasoning paths over wikipedia graph for question answering](https://md.research.elamur.ai/free/en/papers/asai-2019-learning-retrieve-reasoning-paths-over/). arXiv preprint arXiv:1911.10470, 2019.
2. Bahdanau, D., Cho, K., and Bengio, Y. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). arXiv preprint arXiv:1409.0473, 2014.
3. Berant, J., Chou, A., Frostig, R., and Liang, P. [Semantic parsing on freebase from question-answer pairs](https://md.research.elamur.ai/free/en/papers/berant-2013-semantic-parsing-freebase-question-answer/). In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1533–1544, 2013.
4. Brill, E., Dumais, S., and Banko, M. An analysis of the askmsr question-answering system. In Empirical Methods in Natural Language Processing, 2002. [find](https://scholar.google.com/scholar?q=Brill%2C+E%2E%2C+Dumais%2C+S%2E%2C+and+Banko%2C+M%2E+An+analysis+of+the+askmsr+question%2Danswering+system%2E+In+Empirical+Methods+in+Natural+Language+Processing%2C+2002%2E)
5. Chen, D., Fisch, A., Weston, J., and Bordes, A. [Reading wikipedia to answer open-domain questions](https://md.research.elamur.ai/free/en/papers/chen-2017-reading-wikipedia-answer-open-domain/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pp. 1870–1879, 2017.
6. Clark, C. and Gardner, M. [Simple and effective multi-paragraph reading comprehension](https://md.research.elamur.ai/free/en/papers/clark-2017-simple-effective-multi-paragraph-reading/). In Annual Meeting of the Association for Computational Linguistics, 2017.
7. Dai, A. M. and Le, Q. V. Semi-supervised sequence learning. In Advances in neural information processing systems, pp. 3079–3087, 2015. [find](https://scholar.google.com/scholar?q=Dai%2C+A%2E+M%2E+and+Le%2C+Q%2E+V%2E+Semi%2Dsupervised+sequence+learning%2E+In+Advances+in+neural+information+processing+systems%2C+pp%2E+3079%E2%80%933087%2C+2015%2E)
8. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805, 2018.
9. Graves, A., Wayne, G., and Danihelka, I. Neural turing machines. ArXiv, abs/1410.5401, 2014. [find](https://scholar.google.com/scholar?q=Graves%2C+A%2E%2C+Wayne%2C+G%2E%2C+and+Danihelka%2C+I%2E+Neural+turing+machines%2E+ArXiv%2C+abs%2F1410%2E5401%2C+2014%2E)
10. Guu, K., Hashimoto, T. B., Oren, Y., and Liang, P. [Generating sentences by editing prototypes](https://md.research.elamur.ai/free/en/papers/guu-2017-generating-sentences-editing-prototypes/). Transactions of the Association for Computational Linguistics, 6:437– 450, 2018.
11. Hashimoto, T. B., Guu, K., Oren, Y., and Liang, P. S. A retrieve-and-edit framework for predicting structured outputs. In Advances in Neural Information Processing Systems, pp. 10052–10062, 2018. [find](https://scholar.google.com/scholar?q=Hashimoto%2C+T%2E+B%2E%2C+Guu%2C+K%2E%2C+Oren%2C+Y%2E%2C+and+Liang%2C+P%2E+S%2E+A+retrieve%2Dand%2Dedit+framework+for+predicting+structured+outputs%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+10052%E2%80%9310062%2C+2018%2E)
12. Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., and Levy, O. [SpanBERT: Improving pre-training by representing and predicting spans](https://md.research.elamur.ai/free/en/papers/joshi-2019-spanbert-improving-pre-training-representing/). arXiv preprint arXiv:1907.10529, 2019.
13. Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., and Lewis, M. Generalization through memorization: Nearest neighbor language models. ArXiv, abs/1911.00172, 2019. [find](https://scholar.google.com/scholar?q=Khandelwal%2C+U%2E%2C+Levy%2C+O%2E%2C+Jurafsky%2C+D%2E%2C+Zettlemoyer%2C+L%2E%2C+and+Lewis%2C+M%2E+Generalization+through+memorization%3A+Nearest+neighbor+language+models%2E+ArXiv%2C+abs%2F1911%2E00172%2C+2019%2E)
14. Kiros, R., Zhu, Y., Salakhutdinov, R. R., Zemel, R., Urtasun, R., Torralba, A., and Fidler, S. Skip-thought vectors. In Advances in neural information processing systems, pp. 3294–3302, 2015. [find](https://scholar.google.com/scholar?q=Kiros%2C+R%2E%2C+Zhu%2C+Y%2E%2C+Salakhutdinov%2C+R%2E+R%2E%2C+Zemel%2C+R%2E%2C+Urtasun%2C+R%2E%2C+Torralba%2C+A%2E%2C+and+Fidler%2C+S%2E+Skip%2Dthought+vectors%2E+In+Advances+in+neural+information+processing+systems%2C+pp%2E+3294%E2%80%933302%2C+2015%2E)
15. Kwiatkowski, T., Palomaki, J., Rhinehart, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Kelcey, M., Devlin, J., et al. [Natural questions: a benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). Transactions of the Association for Computational Linguistics, 2019.
16. Lample, G., Sablayrolles, A., Ranzato, M., Denoyer, L., and Jégou, H. Large memory layers with product keys. In Advances in Neural Information Processing Systems, pp. 8546–8557, 2019. [find](https://scholar.google.com/scholar?q=Lample%2C+G%2E%2C+Sablayrolles%2C+A%2E%2C+Ranzato%2C+M%2E%2C+Denoyer%2C+L%2E%2C+and+Je%CC%81gou%2C+H%2E+Large+memory+layers+with+product+keys%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+8546%E2%80%938557%2C+2019%2E)
17. Lee, K., Salant, S., Kwiatkowski, T., Parikh, A., Das, D., and Berant, J. Learning recurrent span representations for extractive question answering. arXiv preprint arXiv:1611.01436, 2016. — [arXiv:1611.01436](https://arxiv.org/abs/1611.01436) [find](https://scholar.google.com/scholar?q=Lee%2C+K%2E%2C+Salant%2C+S%2E%2C+Kwiatkowski%2C+T%2E%2C+Parikh%2C+A%2E%2C+Das%2C+D%2E%2C+and+Berant%2C+J%2E+Learning+recurrent+span+representations+for+extractive+question+answering%2E+arXiv+preprint+arXiv%3A1611%2E01436%2C+2016%2E+%E2%80%94+%5BarXiv%3A161)
18. Lee, K., Chang, M.-W., and Toutanova, K. [Latent retrieval for weakly supervised open domain question answering](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/). In Proceedings of the Conference of Association for Computational Linguistics, 2019.
19. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. [Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). ArXiv, abs/1910.13461, 2019.
20. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692, 2019.
21. Mikolov, T., Chen, K., Corrado, G., and Dean, J. [Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/). arXiv preprint arXiv:1301.3781, 2013a.
22. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. [Distributed representations of words and phrases and their compositionality](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/). In Advances in neural information processing systems, pp. 3111–3119, 2013b.
23. Miller, A., Fisch, A., Dodge, J., Karimi, A.-H., Bordes, A., and Weston, J. Key-value memory networks for directly reading documents. arXiv preprint arXiv:1606.03126, 2016. [find](https://scholar.google.com/scholar?q=Miller%2C+A%2E%2C+Fisch%2C+A%2E%2C+Dodge%2C+J%2E%2C+Karimi%2C+A%2E%2DH%2E%2C+Bordes%2C+A%2E%2C+and+Weston%2C+J%2E+Key%2Dvalue+memory+networks+for+directly+reading+documents%2E+arXiv+preprint+arXiv%3A1606%2E03126%2C+2016%2E)
24. Min, S., Chen, D., Hajishirzi, H., and Zettlemoyer, L. [A discrete hard em approach for weakly supervised question answering](https://md.research.elamur.ai/free/en/papers/min-2019-discrete-hard-em-approach-weakly/). arXiv preprint arXiv:1909.04849, 2019a.
25. Min, S., Chen, D., Zettlemoyer, L., and Hajishirzi, H. Knowledge guided text retrieval and reading for open domain question answering. arXiv preprint arXiv:1911.[03868, 2019b](https://md.research.elamur.ai/free/en/papers/chen-2020-open-domain-question-answering/).
26. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). In Proc. of NAACL, 2018.
27. Peters, M. E., Neumann, M., IV, R. L. L., Schwartz, R., Joshi, V., Singh, S., and Smith, N. A. Knowledge enhanced contextual word representations, 2019. [find](https://scholar.google.com/scholar?q=Peters%2C+M%2E+E%2E%2C+Neumann%2C+M%2E%2C+IV%2C+R%2E+L%2E+L%2E%2C+Schwartz%2C+R%2E%2C+Joshi%2C+V%2E%2C+Singh%2C+S%2E%2C+and+Smith%2C+N%2E+A%2E+Knowledge+enhanced+contextual+word+representations%2C+2019%2E)
28. Petroni, F., Rocktäschel, T., Lewis, P., Bakhtin, A., Wu, Y., Miller, A. H., and Riedel, S. [Language models as knowledge bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) arXiv preprint arXiv:1909.01066, 2019.
29. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding with unsupervised learning. Technical report, OpenAI, 2018. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Narasimhan%2C+K%2E%2C+Salimans%2C+T%2E%2C+and+Sutskever%2C+I%2E+Improving+language+understanding+with+unsupervised+learning%2E+Technical+report%2C+OpenAI%2C+2018%2E)
30. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multi-task learners. OpenAI Blog, 2019. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Wu%2C+J%2E%2C+Child%2C+R%2E%2C+Luan%2C+D%2E%2C+Amodei%2C+D%2E%2C+and+Sutskever%2C+I%2E+Language+models+are+unsupervised+multi%2Dtask+learners%2E+OpenAI+Blog%2C+2019%2E)
31. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). arXiv preprint arXiv:1910.10683, 2019.
32. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. [Squad: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383– 2392, 2016.
33. Rajpurkar, P., Jia, R., and Liang, P. Know what you don’t know: Unanswerable questions for squad. arXiv preprint arXiv:1806.[03822, 2018](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/).
34. Ram, P. and Gray, A. G. Maximum inner-product search using cone trees. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 931–939, 2012. [find](https://scholar.google.com/scholar?q=Ram%2C+P%2E+and+Gray%2C+A%2E+G%2E+Maximum+inner%2Dproduct+search+using+cone+trees%2E+In+Proceedings+of+the+18th+ACM+SIGKDD+international+conference+on+Knowledge+discovery+and+data+mining%2C+pp%2E+931%E2%80%93939%2C+2012%2E)
35. Roberts, A., Raffel, C., and Shazeer, N. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:TBD, 2020. [find](https://scholar.google.com/scholar?q=Roberts%2C+A%2E%2C+Raffel%2C+C%2E%2C+and+Shazeer%2C+N%2E+How+much+knowledge+can+you+pack+into+the+parameters+of+a+language+model%3F+arXiv+preprint+arXiv%3ATBD%2C+2020%2E)
36. Robertson, S., Zaragoza, H., et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389, 2009. [find](https://scholar.google.com/scholar?q=Robertson%2C+S%2E%2C+Zaragoza%2C+H%2E%2C+et+al%2E+The+probabilistic+relevance+framework%3A+Bm25+and+beyond%2E+Foundations+and+Trends+in+Information+Retrieval%2C+3%284%29%3A333%E2%80%93389%2C+2009%2E)
37. Sang, E. T. K. and De Meulder, F. [Introduction to the conll-2003 shared task: Language-independent named entity recognition](https://md.research.elamur.ai/free/en/papers/sang-2003-introduction-conll-2003-shared-task/). In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pp. 142–147, 2003.
38. Seo, M., Kembhavi, A., Farhadi, A., and Hajishirzi, H. [Bidirectional attention flow for machine comprehension](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/). In International Conference on Learning Representations, 2016.
39. Shen, F., Liu, W., Zhang, S., Yang, Y., and Tao Shen, H. Learning binary codes for maximum inner product search. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4148–4156, 2015. [find](https://scholar.google.com/scholar?q=Shen%2C+F%2E%2C+Liu%2C+W%2E%2C+Zhang%2C+S%2E%2C+Yang%2C+Y%2E%2C+and+Tao+Shen%2C+H%2E+Learning+binary+codes+for+maximum+inner+product+search%2E+In+Proceedings+of+the+IEEE+International+Conference+on+Computer+Vision%2C+pp%2E+4148%E2%80%934156%2C+)
40. Shrivastava, A. and Li, P. Asymmetric lsh (alsh) for sublinear time maximum inner product search (mips). In Advances in Neural Information Processing Systems, pp. 2321–2329, 2014. [find](https://scholar.google.com/scholar?q=Shrivastava%2C+A%2E+and+Li%2C+P%2E+Asymmetric+lsh+%28alsh%29+for+sublinear+time+maximum+inner+product+search+%28mips%29%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+2321%E2%80%932329%2C+2014%2E)
41. Sukhbaatar, S., Weston, J., Fergus, R., et al. End-to-end memory networks. In Advances in neural information processing systems, 2015. [find](https://scholar.google.com/scholar?q=Sukhbaatar%2C+S%2E%2C+Weston%2C+J%2E%2C+Fergus%2C+R%2E%2C+et+al%2E+End%2Dto%2Dend+memory+networks%2E+In+Advances+in+neural+information+processing+systems%2C+2015%2E)
42. Weston, J., Chopra, S., and Bordes, A. Memory networks. arXiv preprint arXiv:1410.3916, 2014. [find](https://scholar.google.com/scholar?q=Weston%2C+J%2E%2C+Chopra%2C+S%2E%2C+and+Bordes%2C+A%2E+Memory+networks%2E+arXiv+preprint+arXiv%3A1410%2E3916%2C+2014%2E)

*Updated 31 October 2026*