--- title: "RAFT: Adapting Language Model to Domain — Zhang 2024" description: "The paper asks how a pretrained LLM should be prepared when it will answer questions over a known, fixed document collection, such as enterprise files…" authors: ["Tianjun Zhang", "Shishir G. Patil", "Naman Jain", "Sheng Shen", "Matei Zaharia", "Ion Stoica", "Joseph E. Gonzalez"] year: 2024 venue: "arXiv:2403.10131" source: "https://arxiv.org/abs/2403.10131" doi: "10.48550/arxiv.2403.10131" arxiv: "2403.10131" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/zhang-2024-raft-adapting-language-model-domain/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2024-raft-adapting-language-model-domain/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / RAFT: Adapting Language Model to Domain Specific RAG # RAFT: Adapting Language Model to Domain Specific RAG **Authors:** [Tianjun Zhang](https://md.research.elamur.ai/free/en/authors/tianjun-zhang/), [Shishir G. Patil](https://md.research.elamur.ai/free/en/authors/shishir-g-patil/), [Naman Jain](https://md.research.elamur.ai/free/en/authors/naman-jain/), [Sheng Shen](https://md.research.elamur.ai/free/en/authors/sheng-shen/), [Matei Zaharia](https://md.research.elamur.ai/free/en/authors/matei-zaharia/), [Ion Stoica](https://md.research.elamur.ai/free/en/authors/ion-stoica/), [Joseph E. Gonzalez](https://md.research.elamur.ai/free/en/authors/joseph-e-gonzalez/) · **Year:** 2024 · **Venue:** arXiv:2403.10131 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2403.10131) · [PDF](https://arxiv.org/pdf/2403.10131) · [DOI 10.48550/arxiv.2403.10131](https://doi.org/10.48550/arxiv.2403.10131) · [arXiv 2403.10131](https://arxiv.org/abs/2403.10131) · [OpenAlex](https://openalex.org/W4392930030) **Citations:** 29 citations (openalex-snapshot · 2026-10-14) · [106](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary The paper asks how a pretrained LLM should be prepared when it will answer questions over a known, fixed document collection, such as enterprise files, medical literature or API documentation, with retrieval at inference time. Prompting with retrieved passages alone ignores the chance to learn the domain beforehand. Standard fine-tuning, meanwhile, either drops the documents at test time or assumes retrieval is perfect. The authors propose RAFT (Retrieval Augmented Fine Tuning), a recipe for building training data. Each example pairs a question with several retrieved-style passages, some relevant ("golden") and some irrelevant ("distractor"). The target answer is chain-of-thought reasoning that explicitly quotes the supporting passage. In a share of examples the golden passage is deliberately left out. A LLaMA-2 7B model trained this way (RAFT-7B) beats domain-specific supervised fine-tuning, with or without RAG, and general-purpose models with RAG on PubMed, HotpotQA and the Gorilla API benchmarks. It also holds up better when the number of retrieved documents changes. This suggests a simple, retriever-agnostic way to specialize models for in-domain RAG. ## Key points - **Training data mix:** a fraction P of the examples contains the golden document(s) plus distractors. The remaining 1 − P contains only distractors, which pushes the model to absorb domain knowledge rather than rely only on context. - **Answer format:** answers use chain-of-thought style and cite evidence between `##begin_quote##` and `##end_quote##` markers. Ablations show this reasoning-with-citations format improves accuracy. Gorilla APIBench already ships with reasoning in its answers. - **Evaluation domains:** - PubMed - HotpotQA - Gorilla's HuggingFace Hub, Torch Hub and TensorFlow Hub sets - Natural Questions and Trivia QA, used in the robustness analysis - **Distractor analysis:** training on golden documents alone performs poorly. The best number of training documents varies by dataset. The default setup uses one golden plus four distractor documents, evaluated on top-k retrieval. - **Code:** released in the Gorilla repository at github.com/ShishirPatil/gorilla. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) · [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/) ## Cited by Cited by 106 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 103 not yet. - [Retrieval-Augmented Generation for Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/gao-2023-retrieval-augmented-generation-large-language/) - [AccurateRAG: A Framework for Building Accurate Retrieval-Augmented Question-Answering Applications](https://md.research.elamur.ai/free/en/papers/nguyen-2026-accuraterag-framework-building-accurate-retrieval/) - [Corrective Retrieval Augmented Generation](https://md.research.elamur.ai/free/en/papers/yan-2024-corrective-retrieval-augmented-generation/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{zhang2024raft, title = {{RAFT: Adapting Language Model to Domain Specific RAG}}, author = {Zhang, Tianjun and Patil, Shishir G. and Jain, Naman and Shen, Sheng and Zaharia, Matei and Stoica, Ion and Gonzalez, Joseph E.}, year = {2024}, doi = {10.48550/arxiv.2403.10131}, eprint = {2403.10131}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2403.10131} } ``` - [← previous Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://md.research.elamur.ai/free/en/papers/yang-2024-qwen2-5-math-technical-report/) - [next → RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models](https://md.research.elamur.ai/free/en/papers/niu-2024-ragtruth-hallucination-corpus-developing-trustworthy/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) ## References 41 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Anthropic. Prompt engineering for claude’s long context window. 2023. [find](https://scholar.google.com/scholar?q=Anthropic%2E+Prompt+engineering+for+claude%E2%80%99s+long+context+window%2E+2023%2E) 2. Asai, A., Wu, Z., Wang, Y., Sil, A., and Hajishirzi, H. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511, 2023. [find](https://scholar.google.com/scholar?q=Asai%2C+A%2E%2C+Wu%2C+Z%2E%2C+Wang%2C+Y%2E%2C+Sil%2C+A%2E%2C+and+Hajishirzi%2C+H%2E+Self%2Drag%3A+Learning+to+retrieve%2C+generate%2C+and+critique+through+self%2Dreflection%2E+arXiv+preprint+arXiv%3A2310%2E11511%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/asai-2023-self-rag-learning-retrieve-generate/) 3. Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Van Den Driessche, G. B., Lespiau, J.-B., Damoc, B., Clark, A., et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pp. 2206–2240. PMLR, 2022. [find](https://scholar.google.com/scholar?q=Borgeaud%2C+S%2E%2C+Mensch%2C+A%2E%2C+Hoffmann%2C+J%2E%2C+Cai%2C+T%2E%2C+Rutherford%2C+E%2E%2C+Millican%2C+K%2E%2C+Van+Den+Driessche%2C+G%2E+B%2E%2C+Lespiau%2C+J%2E%2DB%2E%2C+Damoc%2C+B%2E%2C+Clark%2C+A%2E%2C+et+al%2E+Improving+language+models+by+retrieving+from+trill) 4. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. [find](https://scholar.google.com/scholar?q=Brown%2C+T%2E%2C+Mann%2C+B%2E%2C+Ryder%2C+N%2E%2C+Subbiah%2C+M%2E%2C+Kaplan%2C+J%2E+D%2E%2C+Dhariwal%2C+P%2E%2C+Neelakantan%2C+A%2E%2C+Shyam%2C+P%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+Advances+in+neural+informati) 5. Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., and Song, D. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pp. 267–284, 2019. [find](https://scholar.google.com/scholar?q=Carlini%2C+N%2E%2C+Liu%2C+C%2E%2C+Erlingsson%2C+%C3%9A%2E%2C+Kos%2C+J%2E%2C+and+Song%2C+D%2E+The+secret+sharer%3A+Evaluating+and+testing+unintended+memorization+in+neural+networks%2E+In+28th+USENIX+Security+Symposium+%28USENIX+Security+19%29) [our card](https://md.research.elamur.ai/free/en/papers/carlini-2018-secret-sharer-evaluating-testing-unintended/) 6. Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633–2650, 2021. [find](https://scholar.google.com/scholar?q=Carlini%2C+N%2E%2C+Tramer%2C+F%2E%2C+Wallace%2C+E%2E%2C+Jagielski%2C+M%2E%2C+Herbert%2DVoss%2C+A%2E%2C+Lee%2C+K%2E%2C+Roberts%2C+A%2E%2C+Brown%2C+T%2E%2C+Song%2C+D%2E%2C+Erlingsson%2C+U%2E%2C+et+al%2E+Extracting+training+data+from+large+language+models%2E+In+30th+US) [our card](https://md.research.elamur.ai/free/en/papers/carlini-2020-extracting-training-data-large-language/) 7. Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., and Zhang, C. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2022. [find](https://scholar.google.com/scholar?q=Carlini%2C+N%2E%2C+Ippolito%2C+D%2E%2C+Jagielski%2C+M%2E%2C+Lee%2C+K%2E%2C+Tramer%2C+F%2E%2C+and+Zhang%2C+C%2E+Quantifying+memorization+across+neural+language+models%2E+In+The+Eleventh+International+Conference+on+Learning+Representation) 8. Dernoncourt, F. and Lee, J. Y. Pubmed 200k rct: a dataset for sequential sentence classification in medical abstracts. arXiv preprint arXiv:1710.06071, 2017. [find](https://scholar.google.com/scholar?q=Dernoncourt%2C+F%2E+and+Lee%2C+J%2E+Y%2E+Pubmed+200k+rct%3A+a+dataset+for+sequential+sentence+classification+in+medical+abstracts%2E+arXiv+preprint+arXiv%3A1710%2E06071%2C+2017%2E) 9. Feldman, V. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp. 954–959, 2020. [find](https://scholar.google.com/scholar?q=Feldman%2C+V%2E+Does+learning+require+memorization%3F+a+short+tale+about+a+long+tail%2E+In+Proceedings+of+the+52nd+Annual+ACM+SIGACT+Symposium+on+Theory+of+Computing%2C+pp%2E+954%E2%80%93959%2C+2020%2E) 10. Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929–3938. PMLR, 2020. [find](https://scholar.google.com/scholar?q=Guu%2C+K%2E%2C+Lee%2C+K%2E%2C+Tung%2C+Z%2E%2C+Pasupat%2C+P%2E%2C+and+Chang%2C+M%2E+Retrieval+augmented+language+model+pre%2Dtraining%2E+In+International+conference+on+machine+learning%2C+pp%2E+3929%E2%80%933938%2E+PMLR%2C+2020%2E) 11. Izacard, G., Lewis, P., Lomeli, M., Hosseini, L., Petroni, F., Schick, T., Dwivedi-Yu, J., Joulin, A., Riedel, S., and Grave, E. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 24(251):1–43, 2023. URL [http://jmlr.org/papers/v24/23-0037.html](http://jmlr.org/papers/v24/23-0037.html). [↗](http://jmlr.org/papers/v24/23-0037.html) 12. Jin, Q., Dhingra, B., Liu, Z., Cohen, W. W., and Lu, X. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146, 2019. [find](https://scholar.google.com/scholar?q=Jin%2C+Q%2E%2C+Dhingra%2C+B%2E%2C+Liu%2C+Z%2E%2C+Cohen%2C+W%2E+W%2E%2C+and+Lu%2C+X%2E+Pubmedqa%3A+A+dataset+for+biomedical+research+question+answering%2E+arXiv+preprint+arXiv%3A1909%2E06146%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/jin-2019-pubmedqa-dataset-biomedical-research-question/) 13. Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017. [find](https://scholar.google.com/scholar?q=Joshi%2C+M%2E%2C+Choi%2C+E%2E%2C+Weld%2C+D%2E+S%2E%2C+and+Zettlemoyer%2C+L%2E+Triviaqa%3A+A+large+scale+distantly+supervised+challenge+dataset+for+reading+comprehension%2E+arXiv+preprint+arXiv%3A1705%2E03551%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/) 14. Kandpal, N., Wallace, E., and Raffel, C. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning, pp. 10697–10707. PMLR, 2022. [find](https://scholar.google.com/scholar?q=Kandpal%2C+N%2E%2C+Wallace%2C+E%2E%2C+and+Raffel%2C+C%2E+Deduplicating+training+data+mitigates+privacy+risks+in+language+models%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+10697%E2%80%9310707%2E+PMLR%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/kandpal-2022-deduplicating-training-data-mitigates-privacy/) 15. Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., and Lewis, M. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00172, 2019. [find](https://scholar.google.com/scholar?q=Khandelwal%2C+U%2E%2C+Levy%2C+O%2E%2C+Jurafsky%2C+D%2E%2C+Zettlemoyer%2C+L%2E%2C+and+Lewis%2C+M%2E+Generalization+through+memorization%3A+Nearest+neighbor+language+models%2E+arXiv+preprint+arXiv%3A1911%2E00172%2C+2019%2E) 16. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466, 2019. [find](https://scholar.google.com/scholar?q=Kwiatkowski%2C+T%2E%2C+Palomaki%2C+J%2E%2C+Redfield%2C+O%2E%2C+Collins%2C+M%2E%2C+Parikh%2C+A%2E%2C+Alberti%2C+C%2E%2C+Epstein%2C+D%2E%2C+Polosukhin%2C+I%2E%2C+Devlin%2C+J%2E%2C+Lee%2C+K%2E%2C+et+al%2E+Natural+questions%3A+a+benchmark+for+question+answering+resear) [our card](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/) 17. Lazaridou, A., Gribovskaya, E., Stokowiec, W., and Grigorev, N. Internet-augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05115, 2022. [find](https://scholar.google.com/scholar?q=Lazaridou%2C+A%2E%2C+Gribovskaya%2C+E%2E%2C+Stokowiec%2C+W%2E%2C+and+Grigorev%2C+N%2E+Internet%2Daugmented+language+models+through+few%2Dshot+prompting+for+open%2Ddomain+question+answering%2E+arXiv+preprint+arXiv%3A2203%2E05115%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/lazaridou-2022-internet-augmented-language-models-through/) 18. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474, 2020. [find](https://scholar.google.com/scholar?q=Lewis%2C+P%2E%2C+Perez%2C+E%2E%2C+Piktus%2C+A%2E%2C+Petroni%2C+F%2E%2C+Karpukhin%2C+V%2E%2C+Goyal%2C+N%2E%2C+K%C3%BCttler%2C+H%2E%2C+Lewis%2C+M%2E%2C+Yih%2C+W%2E%2Dt%2E%2C+Rockt%C3%A4schel%2C+T%2E%2C+et+al%2E+Retrieval%2Daugmented+generation+for+knowledge%2Dintensive+nlp+tasks%2E+A) 19. Lin, X. V., Chen, X., Chen, M., Shi, W., Lomeli, M., James, R., Rodriguez, P., Kahn, J., Szilvasy, G., Lewis, M., et al. Ra-dit: Retrieval-augmented dual instruction tuning. arXiv preprint arXiv:2310.01352, 2023a. [find](https://scholar.google.com/scholar?q=Lin%2C+X%2E+V%2E%2C+Chen%2C+X%2E%2C+Chen%2C+M%2E%2C+Shi%2C+W%2E%2C+Lomeli%2C+M%2E%2C+James%2C+R%2E%2C+Rodriguez%2C+P%2E%2C+Kahn%2C+J%2E%2C+Szilvasy%2C+G%2E%2C+Lewis%2C+M%2E%2C+et+al%2E+Ra%2Ddit%3A+Retrieval%2Daugmented+dual+instruction+tuning%2E+arXiv+preprint+arXiv%3A2310%2E) [our card](https://md.research.elamur.ai/free/en/papers/lin-2023-ra-dit-retrieval-augmented-dual/) 20. Lin, X. V., Chen, X., Chen, M., Shi, W., Lomeli, M., James, R., Rodriguez, P., Kahn, J., Szilvasy, G., Lewis, M., et al. Ra-dit: Retrieval-augmented dual instruction tuning. arXiv preprint arXiv:2310.01352, 2023b. [find](https://scholar.google.com/scholar?q=Lin%2C+X%2E+V%2E%2C+Chen%2C+X%2E%2C+Chen%2C+M%2E%2C+Shi%2C+W%2E%2C+Lomeli%2C+M%2E%2C+James%2C+R%2E%2C+Rodriguez%2C+P%2E%2C+Kahn%2C+J%2E%2C+Szilvasy%2C+G%2E%2C+Lewis%2C+M%2E%2C+et+al%2E+Ra%2Ddit%3A+Retrieval%2Daugmented+dual+instruction+tuning%2E+arXiv+preprint+arXiv%3A2310%2E) [our card](https://md.research.elamur.ai/free/en/papers/lin-2023-ra-dit-retrieval-augmented-dual/) 21. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172, 2023. [find](https://scholar.google.com/scholar?q=Liu%2C+N%2E+F%2E%2C+Lin%2C+K%2E%2C+Hewitt%2C+J%2E%2C+Paranjape%2C+A%2E%2C+Bevilacqua%2C+M%2E%2C+Petroni%2C+F%2E%2C+and+Liang%2C+P%2E+Lost+in+the+middle%3A+How+language+models+use+long+contexts%2E+arXiv+preprint+arXiv%3A2307%2E03172%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/) 22. Liu, Z., Kitouni, O., Nolte, N. S., Michaud, E., Tegmark, M., and Williams, M. Towards understanding grokking: An effective theory of representation learning. Advances in Neural Information Processing Systems, 35:34651–34663, 2022. [find](https://scholar.google.com/scholar?q=Liu%2C+Z%2E%2C+Kitouni%2C+O%2E%2C+Nolte%2C+N%2E+S%2E%2C+Michaud%2C+E%2E%2C+Tegmark%2C+M%2E%2C+and+Williams%2C+M%2E+Towards+understanding+grokking%3A+An+effective+theory+of+representation+learning%2E+Advances+in+Neural+Information+Processing) 23. Liu, Z., Ping, W., Roy, R., Xu, P., Shoeybi, M., and Catanzaro, B. Chatqa: Building gpt-4 level conversational qa models. arXiv preprint arXiv:2401.10225, 2024. [find](https://scholar.google.com/scholar?q=Liu%2C+Z%2E%2C+Ping%2C+W%2E%2C+Roy%2C+R%2E%2C+Xu%2C+P%2E%2C+Shoeybi%2C+M%2E%2C+and+Catanzaro%2C+B%2E+Chatqa%3A+Building+gpt%2D4+level+conversational+qa+models%2E+arXiv+preprint+arXiv%3A2401%2E10225%2C+2024%2E) 24. Pan, X., Zhang, M., Ji, S., and Yang, M. Privacy risks of general-purpose language models. In 2020 IEEE Symposium on Security and Privacy (SP), pp. 1314–1331. [find](https://scholar.google.com/scholar?q=Pan%2C+X%2E%2C+Zhang%2C+M%2E%2C+Ji%2C+S%2E%2C+and+Yang%2C+M%2E+Privacy+risks+of+general%2Dpurpose+language+models%2E+In+2020+IEEE+Symposium+on+Security+and+Privacy+%28SP%29%2C+pp%2E+1314%E2%80%931331%2E) 25. Patil, S. G., Zhang, T., Wang, X., and Gonzalez, J. E. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334, 2023. [find](https://scholar.google.com/scholar?q=Patil%2C+S%2E+G%2E%2C+Zhang%2C+T%2E%2C+Wang%2C+X%2E%2C+and+Gonzalez%2C+J%2E+E%2E+Gorilla%3A+Large+language+model+connected+with+massive+apis%2E+arXiv+preprint+arXiv%3A2305%2E15334%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/patil-2023-gorilla-large-language-model-connected/) 26. Power, A., Burda, Y., Edwards, H., Babuschkin, I., and Misra, V. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177, 2022. [find](https://scholar.google.com/scholar?q=Power%2C+A%2E%2C+Burda%2C+Y%2E%2C+Edwards%2C+H%2E%2C+Babuschkin%2C+I%2E%2C+and+Misra%2C+V%2E+Grokking%3A+Generalization+beyond+overfitting+on+small+algorithmic+datasets%2E+arXiv+preprint+arXiv%3A2201%2E02177%2C+2022%2E) 27. Ram, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua, A., Leyton-Brown, K., and Shoham, Y. In-context retrieval-augmented language models. arXiv preprint arXiv:2302.00083, 2023. [find](https://scholar.google.com/scholar?q=Ram%2C+O%2E%2C+Levine%2C+Y%2E%2C+Dalmedigos%2C+I%2E%2C+Muhlgay%2C+D%2E%2C+Shashua%2C+A%2E%2C+Leyton%2DBrown%2C+K%2E%2C+and+Shoham%2C+Y%2E+In%2Dcontext+retrieval%2Daugmented+language+models%2E+arXiv+preprint+arXiv%3A2302%2E00083%2C+2023%2E) 28. Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E. H., Schärli, N., and Zhou, D. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pp. 31210–31227. PMLR, 2023a. [find](https://scholar.google.com/scholar?q=Shi%2C+F%2E%2C+Chen%2C+X%2E%2C+Misra%2C+K%2E%2C+Scales%2C+N%2E%2C+Dohan%2C+D%2E%2C+Chi%2C+E%2E+H%2E%2C+Sch%C3%A4rli%2C+N%2E%2C+and+Zhou%2C+D%2E+Large+language+models+can+be+easily+distracted+by+irrelevant+context%2E+In+International+Conference+on+Machine+) [our card](https://md.research.elamur.ai/free/en/papers/shi-2023-large-language-models-easily-distracted/) 29. Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., and Zettlemoyer, L. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789, 2023b. [find](https://scholar.google.com/scholar?q=Shi%2C+W%2E%2C+Ajith%2C+A%2E%2C+Xia%2C+M%2E%2C+Huang%2C+Y%2E%2C+Liu%2C+D%2E%2C+Blevins%2C+T%2E%2C+Chen%2C+D%2E%2C+and+Zettlemoyer%2C+L%2E+Detecting+pretraining+data+from+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E16789%2C+2023b%2E) 30. Shi, W., Min, S., Lomeli, M., Zhou, C., Li, M., Lin, V., Smith, N. A., Zettlemoyer, L., Yih, S., and Lewis, M. In-context pretraining: Language modeling beyond document boundaries. arXiv preprint arXiv:2310.10638, 2023c. [find](https://scholar.google.com/scholar?q=Shi%2C+W%2E%2C+Min%2C+S%2E%2C+Lomeli%2C+M%2E%2C+Zhou%2C+C%2E%2C+Li%2C+M%2E%2C+Lin%2C+V%2E%2C+Smith%2C+N%2E+A%2E%2C+Zettlemoyer%2C+L%2E%2C+Yih%2C+S%2E%2C+and+Lewis%2C+M%2E+In%2Dcontext+pretraining%3A+Language+modeling+beyond+document+boundaries%2E+arXiv+preprint+arXi) 31. Shi, W., Min, S., Yasunaga, M., Seo, M., James, R., Lewis, M., Zettlemoyer, L., and Yih, W.-t. Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv:2301.12652, 2023d. [find](https://scholar.google.com/scholar?q=Shi%2C+W%2E%2C+Min%2C+S%2E%2C+Yasunaga%2C+M%2E%2C+Seo%2C+M%2E%2C+James%2C+R%2E%2C+Lewis%2C+M%2E%2C+Zettlemoyer%2C+L%2E%2C+and+Yih%2C+W%2E%2Dt%2E+Replug%3A+Retrieval%2Daugmented+black%2Dbox+language+models%2E+arXiv+preprint+arXiv%3A2301%2E12652%2C+2023d%2E) [our card](https://md.research.elamur.ai/free/en/papers/shi-2023-replug-retrieval-augmented-black-box/) 32. Tänzer, M., Ruder, S., and Rei, M. Memorisation versus generalisation in pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7564–7578, 2022. [find](https://scholar.google.com/scholar?q=T%C3%A4nzer%2C+M%2E%2C+Ruder%2C+S%2E%2C+and+Rei%2C+M%2E+Memorisation+versus+generalisation+in+pre%2Dtrained+language+models%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume) 33. Vu, T., Iyyer, M., Wang, X., Constant, N., Wei, J., Wei, J., Tar, C., Sung, Y.-H., Zhou, D., Le, Q., et al. Freshllms: Refreshing large language models with search engine augmentation. arXiv preprint arXiv:2310.03214, 2023. [find](https://scholar.google.com/scholar?q=Vu%2C+T%2E%2C+Iyyer%2C+M%2E%2C+Wang%2C+X%2E%2C+Constant%2C+N%2E%2C+Wei%2C+J%2E%2C+Wei%2C+J%2E%2C+Tar%2C+C%2E%2C+Sung%2C+Y%2E%2DH%2E%2C+Zhou%2C+D%2E%2C+Le%2C+Q%2E%2C+et+al%2E+Freshllms%3A+Refreshing+large+language+models+with+search+engine+augmentation%2E+arXiv+preprint+) [our card](https://md.research.elamur.ai/free/en/papers/vu-2023-freshllms-refreshing-large-language-models/) 34. Wang, B., Ping, W., McAfee, L., Xu, P., Li, B., Shoeybi, M., and Catanzaro, B. Instructretro: Instruction tuning post retrieval-augmented pretraining. arXiv preprint arXiv:2310.07713, 2023. [find](https://scholar.google.com/scholar?q=Wang%2C+B%2E%2C+Ping%2C+W%2E%2C+McAfee%2C+L%2E%2C+Xu%2C+P%2E%2C+Li%2C+B%2E%2C+Shoeybi%2C+M%2E%2C+and+Catanzaro%2C+B%2E+Instructretro%3A+Instruction+tuning+post+retrieval%2Daugmented+pretraining%2E+arXiv+preprint+arXiv%3A2310%2E07713%2C+2023%2E) 35. Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560, 2022. [find](https://scholar.google.com/scholar?q=Wang%2C+Y%2E%2C+Kordi%2C+Y%2E%2C+Mishra%2C+S%2E%2C+Liu%2C+A%2E%2C+Smith%2C+N%2E+A%2E%2C+Khashabi%2C+D%2E%2C+and+Hajishirzi%2C+H%2E+Self%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%2E+arXiv+preprint+arXiv%3A2212%2E10560%2C+2022) 36. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022. [find](https://scholar.google.com/scholar?q=Wei%2C+J%2E%2C+Wang%2C+X%2E%2C+Schuurmans%2C+D%2E%2C+Bosma%2C+M%2E%2C+Xia%2C+F%2E%2C+Chi%2C+E%2E%2C+Le%2C+Q%2E+V%2E%2C+Zhou%2C+D%2E%2C+et+al%2E+Chain%2Dof%2Dthought+prompting+elicits+reasoning+in+large+language+models%2E+Advances+in+Neural+Information+Proces) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/) 37. Weston, J. and Sukhbaatar, S. System 2 attention (is something you might need too). arXiv preprint arXiv:2311.11829, 2023. — [arXiv:2311.11829](https://arxiv.org/abs/2311.11829) [find](https://scholar.google.com/scholar?q=Weston%2C+J%2E+and+Sukhbaatar%2C+S%2E+System+2+attention+%28is+something+you+might+need+too%29%2E+arXiv+preprint+arXiv%3A2311%2E11829%2C+2023%2E+%E2%80%94+%5BarXiv%3A2311%2E11829%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2311%2E11829%29) 38. Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., et al. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309.16039, 2023. [find](https://scholar.google.com/scholar?q=Xiong%2C+W%2E%2C+Liu%2C+J%2E%2C+Molybog%2C+I%2E%2C+Zhang%2C+H%2E%2C+Bhargava%2C+P%2E%2C+Hou%2C+R%2E%2C+Martin%2C+L%2E%2C+Rungta%2C+R%2E%2C+Sankararaman%2C+K%2E+A%2E%2C+Oguz%2C+B%2E%2C+et+al%2E+Effective+long%2Dcontext+scaling+of+foundation+models%2E+arXiv+preprint+arX) [our card](https://md.research.elamur.ai/free/en/papers/xiong-2023-effective-long-context-scaling-foundation/) 39. Xu, P., Ping, W., Wu, X., McAfee, L., Zhu, C., Liu, Z., Subramanian, S., Bakhturina, E., Shoeybi, M., and Catanzaro, B. Retrieval meets long context large language models. arXiv preprint arXiv:2310.03025, 2023. [find](https://scholar.google.com/scholar?q=Xu%2C+P%2E%2C+Ping%2C+W%2E%2C+Wu%2C+X%2E%2C+McAfee%2C+L%2E%2C+Zhu%2C+C%2E%2C+Liu%2C+Z%2E%2C+Subramanian%2C+S%2E%2C+Bakhturina%2C+E%2E%2C+Shoeybi%2C+M%2E%2C+and+Catanzaro%2C+B%2E+Retrieval+meets+long+context+large+language+models%2E+arXiv+preprint+arXiv%3A2310%2E03) [our card](https://md.research.elamur.ai/free/en/papers/xu-2023-retrieval-meets-long-context-large/) 40. Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., and Manning, C. D. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600, 2018. [find](https://scholar.google.com/scholar?q=Yang%2C+Z%2E%2C+Qi%2C+P%2E%2C+Zhang%2C+S%2E%2C+Bengio%2C+Y%2E%2C+Cohen%2C+W%2E+W%2E%2C+Salakhutdinov%2C+R%2E%2C+and+Manning%2C+C%2E+D%2E+Hotpotqa%3A+A+dataset+for+diverse%2C+explainable+multi%2Dhop+question+answering%2E+arXiv+preprint+arXiv%3A1809%2E09600%2C) [our card](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/) 41. Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206, 2023. [find](https://scholar.google.com/scholar?q=Zhou%2C+C%2E%2C+Liu%2C+P%2E%2C+Xu%2C+P%2E%2C+Iyer%2C+S%2E%2C+Sun%2C+J%2E%2C+Mao%2C+Y%2E%2C+Ma%2C+X%2E%2C+Efrat%2C+A%2E%2C+Yu%2C+P%2E%2C+Yu%2C+L%2E%2C+et+al%2E+Lima%3A+Less+is+more+for+alignment%2E+arXiv+preprint+arXiv%3A2305%2E11206%2C+2023%2E)