---
title: "Large Language Models Struggle to Learn — Kandpal 2022"
description: "This paper asks whether the factual knowledge of large language models depends on how often a fact appears in their pre-training data."
authors: ["Nikhil Kandpal", "Haikang Deng", "Adam Roberts", "Eric Wallace", "Colin Raffel"]
year: 2022
venue: "arXiv:2211.08411"
source: "https://arxiv.org/abs/2211.08411"
doi: "10.48550/arxiv.2211.08411"
arxiv: "2211.08411"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kandpal-2022-large-language-models-struggle-learn/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kandpal-2022-large-language-models-struggle-learn/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/) / Large Language Models Struggle to Learn Long-Tail Knowledge

# Large Language Models Struggle to Learn Long-Tail Knowledge

**Authors:** [Nikhil Kandpal](https://md.research.elamur.ai/free/en/authors/nikhil-kandpal/), Haikang Deng, [Adam Roberts](https://md.research.elamur.ai/free/en/authors/adam-roberts/), [Eric Wallace](https://md.research.elamur.ai/free/en/authors/eric-wallace/), [Colin Raffel](https://md.research.elamur.ai/free/en/authors/colin-raffel/) · **Year:** 2022 · **Venue:** arXiv:2211.08411 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2211.08411) · [PDF](https://arxiv.org/pdf/2211.08411) · [DOI 10.48550/arxiv.2211.08411](https://doi.org/10.48550/arxiv.2211.08411) · [arXiv 2211.08411](https://arxiv.org/abs/2211.08411) · [OpenAlex](https://openalex.org/W4309217888)

**Citations:** 94 citations (openalex-snapshot · 2026-10-14) · [258](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary

This paper asks whether the factual knowledge of large language models depends on how often a fact appears in their pre-training data. The authors ran the DBpedia Spotlight Entity Linker over several large corpora: The Pile, ROOTS (En), C4, OpenWebText and a 2018 Wikipedia dump. For each question in Natural Questions and TriviaQA, they took the main question entity and answer entity and counted the documents where both occur. They then compared these counts with few-shot closed-book QA accuracy for model families such as BLOOM and GPT-Neo.

Accuracy rises sharply with the number of relevant documents, and the pattern holds across datasets, corpora and model sizes. A re-training experiment that removes relevant documents suggests the link is causal. Bigger models do handle rare facts better, but extrapolating the trend implies an impractically large scale-up would be needed. Retrieval augmentation largely removes the dependence on frequency. The work matters because it shows that LLM factual competence is concentrated in well-represented knowledge and points to retrieval as a better route to rare facts.

## Key points

- **Entity linking pipeline.** Linking 2.1TB of pre-training text took about three weeks on a 128-core CPU machine. The code and entity data are public at github.com/nkandpa2/long_tail_knowledge.
- **Human check of the pipeline.** On 300 TriviaQA pairs, 33% of sampled documents fully answered the question and another 27% were partially relevant, for roughly 60% precision.
- **Frequency effect.** BLOOM-176B goes from about 25% to over 55% accuracy as relevant document counts grow from 10¹ to 10⁴.
- **Causal test.** A 4.8B-parameter model re-trained without certain relevant documents lost significant accuracy on the affected questions.
- **Scaling and retrieval.** Accuracy grows log-linearly with parameter count, and competitive long-tail QA would require roughly a quadrillion parameters. Adding three BM25 paragraphs to GPT-Neo on Natural Questions removes most of the low accuracy on rare questions, although BM25 recall itself depends mildly on document counts.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

258 papers of our library cite it (of the 140884 whose bibliographies we read); 20 of them have a page here, 238 not yet.

- [Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4](https://md.research.elamur.ai/free/en/papers/chang-2023-speak-memory-archaeology-books-known/)
- [Ragas: Automated Evaluation of Retrieval Augmented Generation](https://md.research.elamur.ai/free/en/papers/es-2023-ragas-automated-evaluation-retrieval-augmented/)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/gao-2023-retrieval-augmented-generation-large-language/)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/)
- [Survey of Hallucination in Natural Language Generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/)
- [Crossing the Threshold: Idiomatic Machine Translation through Retrieval Augmentation and Loss Weighting](https://md.research.elamur.ai/free/en/papers/liu-2023-crossing-threshold-idiomatic-machine-translation/)
- [Lost in the Middle: How Language Models Use Long Contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/)
- [Sources of Hallucination by Large Language Models on Inference Tasks](https://md.research.elamur.ai/free/en/papers/mckenna-2023-sources-hallucination-large-language-models/)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://md.research.elamur.ai/free/en/papers/min-2023-factscore-fine-grained-atomic-evaluation/)
- [Do Factual Recall Mechanisms Carry over from Text to Speech in Multimodal Language Models?](https://md.research.elamur.ai/free/en/papers/modica-2026-do-factual-recall-mechanisms-carry/)
- [RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval](https://md.research.elamur.ai/free/en/papers/sarthi-2024-raptor-recursive-abstractive-processing-tree/)
- [Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models](https://md.research.elamur.ai/free/en/papers/shao-2024-assisting-writing-wikipedia-like-articles/)
- [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://md.research.elamur.ai/free/en/papers/wang-2023-survey-factuality-large-language-models/)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://md.research.elamur.ai/free/en/papers/xi-2023-rise-potential-large-language/)
- [LongTailQA: Benchmarking LLMs and RAG Models on Disambiguated Long-Tail Entities](https://md.research.elamur.ai/free/en/papers/xion-2026-longtailqa-benchmarking-llms-rag-models/)
- [Hallucination is Inevitable: An Innate Limitation of Large Language Models](https://md.research.elamur.ai/free/en/papers/xu-2024-hallucination-inevitable-innate-limitation-large/)
- [Knowledge Conflicts for LLMs: A Survey](https://md.research.elamur.ai/free/en/papers/xu-2024-knowledge-conflicts-llms-survey/)
- [基于大语言模型的中文实体链接实证研究](https://md.research.elamur.ai/free/en/papers/xu-2025-empirical-study-chinese/)
- [Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models](https://md.research.elamur.ai/free/en/papers/yu-2024-chain-note-enhancing-robustness-retrieval/)
- [Explainability for Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/zhao-2023-explainability-large-language-models-survey/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{kandpal2022large,
  title = {{Large Language Models Struggle to Learn Long-Tail Knowledge}},
  author = {Kandpal, Nikhil and Deng, Haikang and Roberts, Adam and Wallace, Eric and Raffel, Colin},
  year = {2022},
  doi = {10.48550/arxiv.2211.08411},
  eprint = {2211.08411},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2211.08411}
}
```

- [← previous Large Language Models Encode Clinical Knowledge](https://md.research.elamur.ai/free/en/papers/singhal-2022-large-language-models-encode-clinical/)
- [next → Large Pre-Trained Models with Extra-Large Vocabularies: A Contrastive Analysis of Hebrew BERT Models and a New One to Outperform Them All](https://md.research.elamur.ai/free/en/papers/hbo-2022-gueta-large-pre-trained-models-extra/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/)

## References

37 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Akyürek, E., Bolukbasi, T., Liu, F., Xiong, B., Tenney, I., Andreas, J., and Guu, K. Tracing knowledge in language models back to the training data. In Findings of EMNLP, 2022. [find](https://scholar.google.com/scholar?q=Akyu%CC%88rek%2C+E%2E%2C+Bolukbasi%2C+T%2E%2C+Liu%2C+F%2E%2C+Xiong%2C+B%2E%2C+Tenney%2C+I%2E%2C+Andreas%2C+J%2E%2C+and+Guu%2C+K%2E+Tracing+knowledge+in+language+models+back+to+the+training+data%2E+In+Findings+of+EMNLP%2C+2022%2E)
2. Han, X. and Tsvetkov, Y. ORCA: Interpreting prompted language models via locating supporting data evidence in the ocean of pretraining data. arXiv preprint arXiv:2205.12600, 2022. — [arXiv:2205.12600](https://arxiv.org/abs/2205.12600) [find](https://scholar.google.com/scholar?q=Han%2C+X%2E+and+Tsvetkov%2C+Y%2E+ORCA%3A+Interpreting+prompted+language+models+via+locating+supporting+data+evidence+in+the+ocean+of+pretraining+data%2E+arXiv+preprint+arXiv%3A2205%2E12600%2C+2022%2E+%E2%80%94+%5BarXiv%3A2205%2E12600%5D)
3. Black, S., Leo, G., Wang, P., Leahy, C., and Biderman, S. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, 2021. [find](https://scholar.google.com/scholar?q=Black%2C+S%2E%2C+Leo%2C+G%2E%2C+Wang%2C+P%2E%2C+Leahy%2C+C%2E%2C+and+Biderman%2C+S%2E+GPT%2DNeo%3A+Large+Scale+Autoregressive+Language+Modeling+with+Mesh%2DTensorflow%2C+2021%2E)
4. Ippolito, D., Tramèr, F., Nasr, M., Zhang, C., Jagielski, M., Lee, K., Choquette-Choo, C. A., and Carlini, N. Preventing verbatim memorization in language models gives a false sense of privacy. arXiv preprint arXiv:2210.17546, 2022. — [arXiv:2210.17546](https://arxiv.org/abs/2210.17546) [find](https://scholar.google.com/scholar?q=Ippolito%2C+D%2E%2C+Trame%CC%80r%2C+F%2E%2C+Nasr%2C+M%2E%2C+Zhang%2C+C%2E%2C+Jagielski%2C+M%2E%2C+Lee%2C+K%2E%2C+Choquette%2DChoo%2C+C%2E+A%2E%2C+and+Carlini%2C+N%2E+Preventing+verbatim+memorization+in+language+models+gives+a+false+sense+of+privacy%2E+arXiv)
5. Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. GPT-Neox-20B: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745, 2022. [find](https://scholar.google.com/scholar?q=Black%2C+S%2E%2C+Biderman%2C+S%2E%2C+Hallahan%2C+E%2E%2C+Anthony%2C+Q%2E%2C+Gao%2C+L%2E%2C+Golding%2C+L%2E%2C+He%2C+H%2E%2C+Leahy%2C+C%2E%2C+McDonell%2C+K%2E%2C+Phang%2C+J%2E%2C+et+al%2E+GPT%2DNeox%2D20B%3A+An+open%2Dsource+autoregressive+language+model%2E+arXiv+preprint+) [our card](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/)
6. Izacard, G. and Grave, E. Distilling knowledge from reader to retriever for question answering. In ICLR, 2021. [find](https://scholar.google.com/scholar?q=Izacard%2C+G%2E+and+Grave%2C+E%2E+Distilling+knowledge+from+reader+to+retriever+for+question+answering%2E+In+ICLR%2C+2021%2E)
7. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. In NeurIPS, 2020. [find](https://scholar.google.com/scholar?q=Brown%2C+T%2E%2C+Mann%2C+B%2E%2C+Ryder%2C+N%2E%2C+Subbiah%2C+M%2E%2C+Kaplan%2C+J%2E+D%2E%2C+Dhariwal%2C+P%2E%2C+Neelakantan%2C+A%2E%2C+Shyam%2C+P%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+In+NeurIPS%2C+2020%2E)
8. Jagielski, M., Thakkar, O., Tramer, F., Ippolito, D., Lee, K., Carlini, N., Wallace, E., Song, S., Thakurta, A., Papernot, N., et al. Measuring forgetting of memorized training examples. In ICLR, 2023. [find](https://scholar.google.com/scholar?q=Jagielski%2C+M%2E%2C+Thakkar%2C+O%2E%2C+Tramer%2C+F%2E%2C+Ippolito%2C+D%2E%2C+Lee%2C+K%2E%2C+Carlini%2C+N%2E%2C+Wallace%2C+E%2E%2C+Song%2C+S%2E%2C+Thakurta%2C+A%2E%2C+Papernot%2C+N%2E%2C+et+al%2E+Measuring+forgetting+of+memorized+training+examples%2E+In+ICLR%2C+2023)
9. Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., and Song, D. The secret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX Security Symposium, 2019. [find](https://scholar.google.com/scholar?q=Carlini%2C+N%2E%2C+Liu%2C+C%2E%2C+Erlingsson%2C+U%CC%81%2E%2C+Kos%2C+J%2E%2C+and+Song%2C+D%2E+The+secret+sharer%3A+Evaluating+and+testing+unintended+memorization+in+neural+networks%2E+In+USENIX+Security+Symposium%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/carlini-2018-secret-sharer-evaluating-testing-unintended/)
10. Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In ACL, 2017. [find](https://scholar.google.com/scholar?q=Joshi%2C+M%2E%2C+Choi%2C+E%2E%2C+Weld%2C+D%2E+S%2E%2C+and+Zettlemoyer%2C+L%2E+TriviaQA%3A+A+large+scale+distantly+supervised+challenge+dataset+for+reading+comprehension%2E+In+ACL%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)
11. Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., Oprea, A., and Raffel, C. Extracting training data from large language models. In USENIX Security Symposium, 2021. [find](https://scholar.google.com/scholar?q=Carlini%2C+N%2E%2C+Tramer%2C+F%2E%2C+Wallace%2C+E%2E%2C+Jagielski%2C+M%2E%2C+Herbert%2DVoss%2C+A%2E%2C+Lee%2C+K%2E%2C+Roberts%2C+A%2E%2C+Brown%2C+T%2E%2C+Song%2C+D%2E%2C+Erlingsson%2C+U%2E%2C+Oprea%2C+A%2E%2C+and+Raffel%2C+C%2E+Extracting+training+data+from+large+language) [our card](https://md.research.elamur.ai/free/en/papers/carlini-2020-extracting-training-data-large-language/)
12. Kandpal, N., Wallace, E., and Raffel, C. Deduplicating training data mitigates privacy risks in language models. In ICML, 2022. [find](https://scholar.google.com/scholar?q=Kandpal%2C+N%2E%2C+Wallace%2C+E%2E%2C+and+Raffel%2C+C%2E+Deduplicating+training+data+mitigates+privacy+risks+in+language+models%2E+In+ICML%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/kandpal-2022-deduplicating-training-data-mitigates-privacy/)
13. Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-t. Dense passage retrieval for open-domain question answering. In EMNLP, 2020. [find](https://scholar.google.com/scholar?q=Karpukhin%2C+V%2E%2C+Og%CC%86uz%2C+B%2E%2C+Min%2C+S%2E%2C+Lewis%2C+P%2E%2C+Wu%2C+L%2E%2C+Edunov%2C+S%2E%2C+Chen%2C+D%2E%2C+and+Yih%2C+W%2E%2Dt%2E+Dense+passage+retrieval+for+open%2Ddomain+question+answering%2E+In+EMNLP%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/)
14. De Cao, N., Izacard, G., Riedel, S., and Petroni, F. Autoregressive entity retrieval. In ICLR, 2021. [find](https://scholar.google.com/scholar?q=De+Cao%2C+N%2E%2C+Izacard%2C+G%2E%2C+Riedel%2C+S%2E%2C+and+Petroni%2C+F%2E+Autoregressive+entity+retrieval%2E+In+ICLR%2C+2021%2E)
15. Kassner, N., Krojer, B., and Schütze, H. Are pretrained language models symbolic reasoners over knowledge? In CoNLL, 2020. [find](https://scholar.google.com/scholar?q=Kassner%2C+N%2E%2C+Krojer%2C+B%2E%2C+and+Schu%CC%88tze%2C+H%2E+Are+pretrained+language+models+symbolic+reasoners+over+knowledge%3F+In+CoNLL%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/kassner-2020-are-pretrained-language-models-symbolic/)
16. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP, 2016. [find](https://scholar.google.com/scholar?q=Rajpurkar%2C+P%2E%2C+Zhang%2C+J%2E%2C+Lopyrev%2C+K%2E%2C+and+Liang%2C+P%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+EMNLP%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
17. Kwiatkowski, T., Palomaki, J., Rhinehart, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Kelcey, M., Devlin, J., et al. Natural Questions: A benchmark for question answering research. In TACL, 2019. [find](https://scholar.google.com/scholar?q=Kwiatkowski%2C+T%2E%2C+Palomaki%2C+J%2E%2C+Rhinehart%2C+O%2E%2C+Collins%2C+M%2E%2C+Parikh%2C+A%2E%2C+Alberti%2C+C%2E%2C+Epstein%2C+D%2E%2C+Polosukhin%2C+I%2E%2C+Kelcey%2C+M%2E%2C+Devlin%2C+J%2E%2C+et+al%2E+Natural+Questions%3A+A+benchmark+for+question+answering+re) [our card](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
18. Rao, R. M., Liu, J., Verkuil, R., Meier, J., Canny, J., Abbeel, P., Sercu, T., and Rives, A. Msa transformer. In ICML, 2021. [find](https://scholar.google.com/scholar?q=Rao%2C+R%2E+M%2E%2C+Liu%2C+J%2E%2C+Verkuil%2C+R%2E%2C+Meier%2C+J%2E%2C+Canny%2C+J%2E%2C+Abbeel%2C+P%2E%2C+Sercu%2C+T%2E%2C+and+Rives%2C+A%2E+Msa+transformer%2E+In+ICML%2C+2021%2E)
19. Razeghi, Y., Logan IV, R. L., Gardner, M., and Singh, S. Impact of pretraining term frequencies on few-shot reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2022, 2022. [find](https://scholar.google.com/scholar?q=Razeghi%2C+Y%2E%2C+Logan+IV%2C+R%2E+L%2E%2C+Gardner%2C+M%2E%2C+and+Singh%2C+S%2E+Impact+of+pretraining+term+frequencies+on+few%2Dshot+reasoning%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2022%2C+2022%2E)
20. Laurençon, H., Saulnier, L., Wang, T., Akiki, C., del Moral, A. V., Scao, T. L., Werra, L. V., Mou, C., Ponferrada, E. G., Nguyen, H., Frohberg, J., Šaško, M., Lhoest, Q., McMillan-Major, A., et al. The BigScience ROOTS corpus: A 1.6TB composite multilingual dataset. In NeurIPS, 2022. [find](https://scholar.google.com/scholar?q=Laurenc%CC%A7on%2C+H%2E%2C+Saulnier%2C+L%2E%2C+Wang%2C+T%2E%2C+Akiki%2C+C%2E%2C+del+Moral%2C+A%2E+V%2E%2C+Scao%2C+T%2E+L%2E%2C+Werra%2C+L%2E+V%2E%2C+Mou%2C+C%2E%2C+Ponferrada%2C+E%2E+G%2E%2C+Nguyen%2C+H%2E%2C+Frohberg%2C+J%2E%2C+S%CC%8Cas%CC%8Cko%2C+M%2E%2C+Lhoest%2C+Q%2E%2C+McMillan%2DMajor%2C+A%2E%2C+et+al) [our card](https://md.research.elamur.ai/free/en/papers/lauren-2023-ccon-bigscience-roots-corpus-1-6tb/)
21. Roberts, A., Raffel, C., and Shazeer, N. How much knowledge can you pack into the parameters of a language model? In EMNLP, 2020. [find](https://scholar.google.com/scholar?q=Roberts%2C+A%2E%2C+Raffel%2C+C%2E%2C+and+Shazeer%2C+N%2E+How+much+knowledge+can+you+pack+into+the+parameters+of+a+language+model%3F+In+EMNLP%2C+2020%2E)
22. Lee, K., Chang, M.-W., and Toutanova, K. Latent retrieval for weakly supervised open domain question answering. In ACL, 2019. [find](https://scholar.google.com/scholar?q=Lee%2C+K%2E%2C+Chang%2C+M%2E%2DW%2E%2C+and+Toutanova%2C+K%2E+Latent+retrieval+for+weakly+supervised+open+domain+question+answering%2E+In+ACL%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/)
23. Robertson, S. and Zaragoza, H. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in IR, 2009. [find](https://scholar.google.com/scholar?q=Robertson%2C+S%2E+and+Zaragoza%2C+H%2E+The+probabilistic+relevance+framework%3A+BM25+and+beyond%2E+Foundations+and+Trends+in+IR%2C+2009%2E)
24. Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., and Carlini, N. Deduplicating training data makes language models better. In ACL, 2021. [find](https://scholar.google.com/scholar?q=Lee%2C+K%2E%2C+Ippolito%2C+D%2E%2C+Nystrom%2C+A%2E%2C+Zhang%2C+C%2E%2C+Eck%2C+D%2E%2C+Callison%2DBurch%2C+C%2E%2C+and+Carlini%2C+N%2E+Deduplicating+training+data+makes+language+models+better%2E+In+ACL%2C+2021%2E)
25. Scao, T. L., Fan, A., Akiki, C., Pavlick, E.-J., Ili’c, S., Hesslow, D., Castagn’e, R., Luccioni, A. S., Yvon, F., Gallé, M., Tow, J., Rush, A. M., et al. BLOOM: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. [find](https://scholar.google.com/scholar?q=Scao%2C+T%2E+L%2E%2C+Fan%2C+A%2E%2C+Akiki%2C+C%2E%2C+Pavlick%2C+E%2E%2DJ%2E%2C+Ili%E2%80%99c%2C+S%2E%2C+Hesslow%2C+D%2E%2C+Castagn%E2%80%99e%2C+R%2E%2C+Luccioni%2C+A%2E+S%2E%2C+Yvon%2C+F%2E%2C+Galle%CC%81%2C+M%2E%2C+Tow%2C+J%2E%2C+Rush%2C+A%2E+M%2E%2C+et+al%2E+BLOOM%3A+A+176b%2Dparameter+open%2Daccess+multilin)
26. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. In NeurIPS, 2020. [find](https://scholar.google.com/scholar?q=Lewis%2C+P%2E%2C+Perez%2C+E%2E%2C+Piktus%2C+A%2E%2C+Petroni%2C+F%2E%2C+Karpukhin%2C+V%2E%2C+Goyal%2C+N%2E%2C+Ku%CC%88ttler%2C+H%2E%2C+Lewis%2C+M%2E%2C+Yih%2C+W%2E%2Dt%2E%2C+Rockta%CC%88schel%2C+T%2E%2C+et+al%2E+Retrieval%2Daugmented+generation+for+knowledge%2Dintensive+NLP+tasks%2E)
27. Shin, S., Lee, S.-W., Ahn, H., Kim, S., Kim, H., Kim, B., Cho, K., Lee, G., Park, W., Ha, J.-W., et al. On the effect of pretraining corpora on in-context learning by a large-scale language model. In NAACL, 2022. [find](https://scholar.google.com/scholar?q=Shin%2C+S%2E%2C+Lee%2C+S%2E%2DW%2E%2C+Ahn%2C+H%2E%2C+Kim%2C+S%2E%2C+Kim%2C+H%2E%2C+Kim%2C+B%2E%2C+Cho%2C+K%2E%2C+Lee%2C+G%2E%2C+Park%2C+W%2E%2C+Ha%2C+J%2E%2DW%2E%2C+et+al%2E+On+the+effect+of+pretraining+corpora+on+in%2Dcontext+learning+by+a+large%2Dscale+language+model%2E+In+)
28. Liu, L., Lewis, P., Riedel, S., and Stenetorp, P. Challenges in generalization in open domain question answering. In Findings of NAACL, 2022. [find](https://scholar.google.com/scholar?q=Liu%2C+L%2E%2C+Lewis%2C+P%2E%2C+Riedel%2C+S%2E%2C+and+Stenetorp%2C+P%2E+Challenges+in+generalization+in+open+domain+question+answering%2E+In+Findings+of+NAACL%2C+2022%2E)
29. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In NeurIPS, 2017. [find](https://scholar.google.com/scholar?q=Vaswani%2C+A%2E%2C+Shazeer%2C+N%2E%2C+Parmar%2C+N%2E%2C+Uszkoreit%2C+J%2E%2C+Jones%2C+L%2E%2C+Gomez%2C+A%2E+N%2E%2C+Kaiser%2C+L%2E%2C+and+Polosukhin%2C+I%2E+Attention+is+all+you+need%2E+In+NeurIPS%2C+2017%2E)
30. Mallen, A., Asai, A., Zhong, V., Das, R., Hajishirzi, H., and Khashabi, D. When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories. arXiv preprint arXiv:2212.10511, 2022. [find](https://scholar.google.com/scholar?q=Mallen%2C+A%2E%2C+Asai%2C+A%2E%2C+Zhong%2C+V%2E%2C+Das%2C+R%2E%2C+Hajishirzi%2C+H%2E%2C+and+Khashabi%2C+D%2E+When+not+to+trust+language+models%3A+Investigating+effectiveness+and+limitations+of+parametric+and+non%2Dparametric+memories%2E+arX)
31. Wang, B. and Komatsuzaki, A. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. [https://github.com/kingoflolz/mesh-transformer-jax](https://github.com/kingoflolz/mesh-transformer-jax), 2021. [↗](https://github.com/kingoflolz/mesh-transformer-jax)
32. Wang, T., Roberts, A., Hesslow, D., Scao, T. L., Chung, H. W., Beltagy, I., Launay, J., and Raffel, C. What language model architecture and pretraining objective work best for zero-shot generalization? In ICML, 2022. [find](https://scholar.google.com/scholar?q=Wang%2C+T%2E%2C+Roberts%2C+A%2E%2C+Hesslow%2C+D%2E%2C+Scao%2C+T%2E+L%2E%2C+Chung%2C+H%2E+W%2E%2C+Beltagy%2C+I%2E%2C+Launay%2C+J%2E%2C+and+Raffel%2C+C%2E+What+language+model+architecture+and+pretraining+objective+work+best+for+zero%2Dshot+generalization)
33. Mendes, P. N., Jakob, M., Garcı́a-Silva, A., and Bizer, C. DBpedia Spotlight: Shedding light on the web of documents. In International Conference on Semantic Systems, 2011. [find](https://scholar.google.com/scholar?q=Mendes%2C+P%2E+N%2E%2C+Jakob%2C+M%2E%2C+Garc%C4%B1%CC%81a%2DSilva%2C+A%2E%2C+and+Bizer%2C+C%2E+DBpedia+Spotlight%3A+Shedding+light+on+the+web+of+documents%2E+In+International+Conference+on+Semantic+Systems%2C+2011%2E)
34. Wei, J., Garrette, D., Linzen, T., and Pavlick, E. Frequency effects on syntactic rule learning in transformers. In EMNLP, 2021. [find](https://scholar.google.com/scholar?q=Wei%2C+J%2E%2C+Garrette%2C+D%2E%2C+Linzen%2C+T%2E%2C+and+Pavlick%2C+E%2E+Frequency+effects+on+syntactic+rule+learning+in+transformers%2E+In+EMNLP%2C+2021%2E)
35. Petroni, F., Rocktäschel, T., Lewis, P., Bakhtin, A., Wu, Y., Miller, A. H., and Riedel, S. Language models as knowledge bases? In EMNLP, 2019. [find](https://scholar.google.com/scholar?q=Petroni%2C+F%2E%2C+Rockta%CC%88schel%2C+T%2E%2C+Lewis%2C+P%2E%2C+Bakhtin%2C+A%2E%2C+Wu%2C+Y%2E%2C+Miller%2C+A%2E+H%2E%2C+and+Riedel%2C+S%2E+Language+models+as+knowledge+bases%3F+In+EMNLP%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/)
36. Petroni, F., Lewis, P. S. H., Piktus, A., Rocktäschel, T., Wu, Y., Miller, A. H., and Riedel, S. How context affects language models’ factual predictions. In AKBC, 2020. [find](https://scholar.google.com/scholar?q=Petroni%2C+F%2E%2C+Lewis%2C+P%2E+S%2E+H%2E%2C+Piktus%2C+A%2E%2C+Rockta%CC%88schel%2C+T%2E%2C+Wu%2C+Y%2E%2C+Miller%2C+A%2E+H%2E%2C+and+Riedel%2C+S%2E+How+context+affects+language+models%E2%80%99+factual+predictions%2E+In+AKBC%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/petroni-2020-context-affects-language-models-factual/)
37. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P. J., et al. Exploring the limits of transfer learning with a unified text-to-text transformer. In JMLR, 2020. [find](https://scholar.google.com/scholar?q=Raffel%2C+C%2E%2C+Shazeer%2C+N%2E%2C+Roberts%2C+A%2E%2C+Lee%2C+K%2E%2C+Narang%2C+S%2E%2C+Matena%2C+M%2E%2C+Zhou%2C+Y%2E%2C+Li%2C+W%2E%2C+Liu%2C+P%2E+J%2E%2C+et+al%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtext+transformer%2E+In+JMLR%2C+)